detect-flowbits: adding details for flowbits v2

[hadiqaalamdar]

Task #6309

• I have read the contributing guide lines at https://docs.suricata.io/en/latest/devguide/codebase/contributing/contribution-process.html
• I have signed the Open Information Security Foundation contribution agreement at https://suricata.io/about/contribution-agreement/
• I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)

Link to redmine ticket:
Previous PR: #9683

Describe changes:

• added the recommended changes from the last PR.
• The SV tests are failing and I need some help with what edits to make to them.

SV BRANCH = OISF/suricata-verify#1438

[github-actions]

NOTE: This PR may contain new authors:

Hadiqa Alamdar Bukhari <hadiqaalamdar@gmail.com>

[jufajardini]

Please see the inline comments, this was a difficult review, but I hope that the feedback will be enough to help you move forward. If not, don't hesitate to reach out!

I have left feedback on the SV tests on the SV PR itself.

For the commit message, please use the imperative form add, to follow our standards :)

[jufajardini]

If the list size is grater than 0, this means we have a list, and might need to use a for or another loop to access the flowbits names.

[jufajardini]

I'm terribly sorry, I misguided you on how to access the variable name. I misunderstood Shivani's reply, and this led to that.

Seeing the CI failures here I went and further investigated the code. For this work, I believe that VarNameStoreSetupLookup would be our choice, and then we would pass cd->idx and VAR_TYPE_FLOW_BIT to it, to get the variable name.

[jufajardini]

Following the above, I spent some time to understand the code.

Aspects to consider:

• So, cd->idx starts at 1
• cd->or_list_size will only be greater than 0 for rules where we see the flowint or logic (cf https://github.com/OISF/suricata-verify/blob/master/tests/detect-flowbits/test.rules#L3). But the idx is global for flowbits. This means that we could have a list_size 2, and yet the idx for the variables accounted in the or_list be 2 and 3.
• Flowbits will only save a variable name if it sees a rule with flowbits:set, with the said variable name. Otherwise, that variable won't be registered, and therefore can't be retrieved with VarNameStoreSetupLookup.

So, in the cases when cd->or_list_size > 0 we will have to that those into account, and then we can open an array to log the variable names. I think we can use the key ored_variables here. See https://github.com/OISF/suricata/blob/master/src/detect-engine-analyzer.c#L797 for an example of how to open an array to log a list of values.

This info will be important to be able to properly proceed with this task, and also when improving the accompanying SV tests for them.

[hadiqaalamdar]

Am I right to assume that if the cd->or_list_size > 0 and cd->cmd == DETECT_FLOWBITS_CMD_SET then a for loop storing the variable names in an array should begin. I'm not sure about the second condition, should I keep it?

[jufajardini]

No, that's not how this works. cd->or_list_size will only be greater than zero for rules like the ones you see in lines 3 and 4 here https://github.com/OISF/suricata-verify/blob/master/tests/detect-flowbits/test.rules#L3-L4

So you should not see size > 0 when DETECT_FLOWBITS_CMD_SET. The SET command will have only one variable name associated with it. Does this answer your question?

[hadiqaalamdar]

okay, so should I create an ored_variables array and store names in it using a for loop if cd->or_list_size > 0? But I'm not sure on how to access the name if cd->or_list_size == 0. Should it be stored in a separate name variable?:

const char *varname = VarNameStoreSetupLookup(cd->idx, VAR_TYPE_FLOW_BIT);
jb_set_string(js, "name", varname);

[jufajardini]

I think this can be simpler.

• No need to create an array, as you could call VarNameStoreSetupLookup from the jb_set_string or jb_append_string functions directly.

• When or_list_size == 0, then you know the rule only has one flowbits variable, and therefore you can pass cd->idx directly to VarNameStoreSetupLookup as id.

[hadiqaalamdar]

okay, and these would both go in the name variable or should I keep them separate in ored_variables and name?

[jufajardini]

As you'll have either one case or the other, the way I see this is as follows:

• if or_list_size == 0, we log out name
• if or_list_size > 0, we log an array of ored_variables

[hadiqaalamdar]

understood. Thank you so much!

[jufajardini]

I think that or_list_size can have some value for a rule writer, as feedback to indicate that the or flowbit rule that they created works, so we could keep that and maybe change the json key to variables.
Following the same reasoning, idx and or_list can be left out from this output.

[hadiqaalamdar]

Thank you for your detailed response. I'll look into this further and make the suggested changes.

[hadiqaalamdar]

work continued on PR: #9688