detect: absent keyword to test absence of sticky buffer

[catenacyber]

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/2224

Describe changes:

• detect: adds absent keyword to match on absent buffer

SV_BRANCH=OISF/suricata-verify#1942

#11301 with use of SIGMATCH_OPTIONAL_OPT

[suricata-qa]

Information: QA ran without warnings.

Pipeline 21274

[victorjulien]

I wondering if this limitation should be lifted. I could easily imagine ppl wanting to express something like
"if there is no user agent, or if it exists and is XYZ, then match"

[catenacyber]

It looks possible, but I had to put in some more changes so that the positive content does not get picked as fast_pattern...

[victorjulien]

The engine analysis output does not reflect the match_on_null state a engine can be in. Would also be good to add detail to the absent match:

{
  "raw": "alert http1 any any -> any any (http.referer; absent; sid:1;)",
  "id": 1,
  "gid": 1,
  "rev": 0,
  "app_proto": "http",
  "requirements": [
    "flow"
  ],
  "type": "app_tx",
  "flags": [
    "src_any",
    "dst_any",
    "sp_any",
    "dp_any",
    "applayer",
    "toserver"
  ],
  "pkt_engines": [],
  "frame_engines": [],
  "engines": [
    {
      "name": "http_referer",
      "direction": "toserver",
      "is_mpm": false,
      "app_proto": "http",
      "progress": 2,
      "matches": [
        {
          "name": "absent"
        }
      ]
    }
  ],
  "lists": {}
}
{
  "raw": "alert http1 any any -> any any (http.referer; absent:or_else; content:!\"ABC\"; sid:3;)",
  "id": 3,
  "gid": 1,
  "rev": 0,
  "app_proto": "http",
  "requirements": [
    "flow"
  ],
  "type": "app_tx",
  "flags": [
    "src_any",
    "dst_any",
    "sp_any",
    "dp_any",
    "applayer",
    "negated_mpm",
    "toserver",
    "prefilter"
  ],
  "pkt_engines": [],
  "frame_engines": [],
  "engines": [
    {
      "name": "http_referer",
      "direction": "toserver",
      "is_mpm": true,
      "app_proto": "http",
      "progress": 2,
      "matches": [
        {
          "name": "absent"
        },
        {
          "name": "content",
          "content": {
            "pattern": "ABC",
            "length": 3,
            "nocase": false,
            "negated": true,
            "starts_with": false,
            "ends_with": false,
            "is_mpm": true,
            "no_double_inspect": false,
            "fast_pattern": false,
            "relative_next": false
          }
        }
      ]
    }
  ],
  "lists": {},
  "mpm": {
    "buffer": "http_referer",
    "pattern": "ABC",
    "length": 3,
    "nocase": false,
    "negated": true,
    "starts_with": false,
    "ends_with": false,
    "is_mpm": true,
    "no_double_inspect": false,
    "fast_pattern": false,
    "relative_next": false
  }
}

[victorjulien]

Plus a SV test for the engine analysis.

[victorjulien]

Does it make sense to allow this for each sticky buffer? I wonder for example, if we can have http w/o a http.method, so this rule may nonsense: alert http1 any any -> any any (http.method; absent; sid:4;). I wonder if a buffer registration should include info about whether it makes sense for it to be "absent".

[catenacyber]

Does it make sense to allow this for each sticky buffer? I wonder for example, if we can have http w/o a http.method, so this rule may nonsense: alert http1 any any -> any any (http.method; absent; sid:4;). I wonder if a buffer registration should include info about whether it makes sense for it to be "absent".

I think it does make sense at first sight... Maybe it can be a later optimization...

[catenacyber]

The engine analysis output does not reflect the match_on_null state a engine can be in.

It does because there is

      "matches": [
        {
          "name": "absent"
        }

Would also be good to add detail to the absent match:

Adding it and the SV test

[catenacyber]

Continued in #11423