[Winlogbeat] XML filtering

[jimmyjones2]

The Windows API allows XML filtering of events - so you can only subscribe to "useful" events rather than the general torrent, again something I've used heavily in the past with nxlog.

<QueryList>
  <Query Id="0">
    <Select Path="Security">*[System[(EventID='4663')]]</Select>
  </Query>
</QueryList>

[andrewkroh]

#1218 adds the ability to do simple filtering based on event_id, provider, and level.

Before adding the ability to specify arbitrary XML queries which can read from multiple logs, we need to change how "bookmarks" are persisted. Currently "bookmarks" are limited to a single event log so if a query read from multiple logs we wouldn't be able to persist all of the read positions. So a change needs to be made to allow writing the read position of all logs involved in a query (probably we will just persist the Windows XML bookmark).

Once that change is made then we can allow XML queries to be used.

[prehor]

We would like to switch to Winlogbeat but without better support for event filtering unfortunately we can not. On the other hand, we do not need the full functionality of XML filtering, which requires substantial modification in Winlogbeat, but we would be satisfied with the XPath Query used for individual events channel:

winlogbeat:
  event_logs:
    - Name: Security
      select:>
        * [System [(EventID=4624 or EventID=4625 or EventID=4648) and
        (Level=0 or Level = 4) and
        Provider [@ Name = 'Microsoft-Windows-Security-Auditing']]]
      supress:>
        * [System [(EventID = 4624)]] and (
        * [EventData [Data [@ Name = "TargetUserSid"] = "S-1-5-7"]] or
        * [EventData [Data [@ Name = "TargetUserSid"] = "S-1-5-18"]] or
        * [EventData [Data [@ Name = "TargetUserSid"] = "S-1-5-19"]] or
        * [EventData [Data [@ Name = "TargetUserSid"] = "S-1-5-20"]]
        )

where event_logs.select could be ORed with the XPath Query created using event_logs.eventid, event_logs.level and event_logs.provider or could be used mutually exclusive.

[adrwh]

Hows this coming along, is using xml filtering an option yet in winlogbeat?

[SpencerLN]

Has there been any additional work on this? It is a crucial feature for most security monitoring use cases.

[mgoritzk]

Winlogbeat: please add the ability to parse nested XML. Currently only the "event_data" is not parsed when it's in XML format.

[andrewkroh]

This issue isn't about parsing nested XML. It's about allowing the user to define a query using the native XML format accepted by Windows for selecting the events to subscribe to.

[andrewkroh]

It should be possible to implement this now given that winlogbeat uses bookmarks in the registry. Each event log reader stores its state based on the name of the channel. In order to be able to persist the state of the XML based query, which can read from multiple channels, we should require that each reader have a unique ID.

• Add a new xml_query config option to the event log reader. (TBD on the name, could be xml_filter?)
• Validate the xml_query is not used with ignore_older, event_id, level, provider, and name. Any of these options should be embedded in the provided XML query.
• Require that an id be specified when using xml_query. This will be used to uniquely store state in the registry. It can be optionally specified when using name to allow more than one reader on the same channel. In the registry, collisions between id and channel name should be prevented (e.g. add a prefix to all id values or store them separately from name).
• winlogbeat test config should validate the XML query (to what extent is TBD, either check that it's valid XML or test loading the query into Windows).

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)