Next/605/20220411/v1 #7227

inashivb · 2022-04-11T07:45:34Z

Link to redmine tickets:

suricata-verify-pr: 801

Since: 9551cd0 ("threading: don't pass locked flow between threads") `MoveToWorkQueue()` unconditionally unlocks the flow. This allows simpler locking handling, including of tcp reuse flows. The simpler logic also fixes a scenario where TCP reuse flows got "unlocked" twice, once in `FlowGetFlowFromHash()` and once in `MoveToWorkQueue()`. Bug: OISF#5248. Coverity: 1494354. (cherry picked from commit 57533d3)

If there is a space following a keyword that does not expect a value, the rule fails to load due to improper value evaluation. e.g. Space after "set" command alert http any any -> any any (http.user_agent; dataset:set ,ua-seen,type string,save datasets.csv; sid:1;) gives error [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - dataset action "" is not supported. Fix this by handling values correctly for such cases. (cherry picked from commit 6d2a2a0)

(cherry picked from commit 7366396)

inashivb · 2022-04-11T09:32:46Z

@catenacyber is the fuzz failure the one fixed by #7221?

catenacyber · 2022-04-11T14:30:13Z

is the fuzz failure the one fixed by #7221?

The fuzz failure should be fixed by one of the commits of #7221 indeed

The initdata argument to OutputFlowThreadInit was always NULL, remove it. Internally the ThreadInit functions still get initdata, but this is the data provided when that logging instance was registered. Ticket: OISF#7227

Ticket: OISF#7227

The use of OutputCtx as the data type for initdata was leaking Eve submodule logic into the low level flow logger. Instead use void *, as the flow logging module is not concerned with the type of data here. Also document this initdata parameter. Ticket: OISF#7227

The callback, ThreadExitPrintStats is not used in the flow loggers. Ticket: OISF#7227

A library/plugin user wanting to register a custom flow logger must include "output-flow.h", however that depends on some other includes. One train of thought with respect to include files in libraries, is that they should include all their dependencies on behalf of the user. To make a custom flow logger just a little easier, include "flow.h" and "decode.h". Ticket: OISF#7227

The use of OutputCtx as the data type for initdata was leaking Eve submodule logic into the low level packet logger. Instead use void *, as the packet logging module is not concerned with the type of data here. Also document this initdata parameter. Ticket: OISF#7227

PgsqlTransactionState has a variant named "Init" which is a little too generic to export to C. Fortunately this method doesn't need to be exposed to C, instead remove it as it was only called by rs_pgsql_tx_get_alstate_progress which also doesn't need to be public or expose to C. Ticket: OISF#7227

This allows the header to be used without including other headers this one depends on. Ticket: OISF#7227

Loggers need an ID uniquely identify them for profiling. To help with loggers registered at runtime (library, plugins), provide a LOGGER_USER that can be used. It won't provide per-logger details if they have more than one, but will provide a total for all their registered loggers. Ticket: OISF#7227

Add an example custom logger that hooks into the low level packet and flow logging callbacks. Ticket: OISF#7227

The ThreadExitPrintStats callback was never being used, remove. Ticket: OISF#7227

Rename OutputRegisterPacketLogger to SCOutputRegisterPacketLogger as its part of the public API and document its parameters. Comment on the other functions in the header that they are part of the internal API. Ticket: OISF#7227

Rename OutputRegisterFlowLogger to SCOutputRegisterFlowLogger and document in the header file. Mark other functions in the header file as part of the internal API. Ticket: OISF#7227

Use the extending/output section to introduce the low level logging API. Ticket: OISF#7227

Using OutputCtx results in the low level output-tx packet logger being aware of Suricata's higher level loggers that use OutputCtx, for the low level logger this is purely opaque data that may not be an OutputCtx for custom loggers. Ticket: OISF#7227

Remove the callback to print stats on thread exit. The counter value was never being used and this helps us get rid of this callback altogether as their is only one other usage of it. Ticket: OISF#7227

Remove the ThreadExitPrintStats callback, this is the only logger that was actually using it, and this logger is marked for deprecation. This allows us to remove the callback from the registration signature. Ticket: OISF#7227

Ticket: OISF#7227

After removing the exit stats function from log-tlslog, this callback is no longer used. Ticket: OISF#7227

However, its disabled due to issue https://redmine.openinfosecfoundation.org/issues/7236. Ticket: OISF#7227

Required to properly resolve the types in the header without depending on includes coming before it in C files. Ticket: OISF#7227