Migrate Winlogbeat to ECS, take 2 #10333

webmat · 2019-01-24T20:24:07Z

Rather than fight with git to push to @ruflin's PR #10169, I'm opening a new one.
See other PR for initial discussions.

In this PR I'll try to take Winlogbeat to the finish line.

Update

Winlogbeat prior to this had over 20 fields directly at the top level of the events (e.g. { "level": "info" }), which is going to be a risk vs introducing ECS fields in the same place.

The Windows event log is also a log transport more than one specific source, of course.

So here's the approach taken, given the two points above:

Nest everything that's windows event log specific under winlog.
Take the opportunity to remove unnecessary name changes, restoring the original Win Event Log naming as much as possible. For example, "Channel" is now "winlog.channel", instead of the field "log_name".
Introduce a very small amount of ECS fields for now. event.created, event.dataset, etc.
Not touch EventData at all. The most interesting data is in there, but is 100% source specific. This should be done in steps, and should be done outside of the Winlogbeat shipper itself.

TODO

Adjust the event stream
Create aliases from the legacy fields to the new fields
Adjust the tests to the new names
Remove dataset.
Document in ecs-migration.yml
Documentation
Changelog
Update dashboard
Figure out how to handle Skipping loading dashboards, No directory C:\vagrant\winlogbeat\kibana/7

ruflin · 2019-01-25T14:31:50Z

winlogbeat/_meta/fields.common.yml

@@ -61,22 +62,27 @@
        The keywords are used to classify an event.

    - name: log_name
-      type: keyword
+      # This does not exist yet


Do you have same example values what it contains?

I don't. I'll try to check out the docs, or better, generate some events

andrewkroh

Copied over my comments from the other PR.

andrewkroh · 2019-01-27T20:06:05Z

winlogbeat/eventlog/eventlog.go

 	}
+	m.Put("event.id", e.EventIdentifier.ID)


This is not the same as ECS event.id (at least in my understanding). This more like an identifier for the log message in the application. For example 1102 is "The audit log was cleared".

record_number is closer to event.id but it's only unique given some additional constraints. If we wanted to populate event.id I'd do a fast hash of @timestamp + computer_name + log_name + record_number.

andrewkroh · 2019-01-27T20:07:18Z

winlogbeat/eventlog/eventlog.go

@@ -139,7 +138,7 @@ func (e Record) ToEvent() beat.Event {
 // MapStr.


A few lines up...

e.TimeCreated.SystemTime is used to populate @timestamp. That's the time the event was originally logged. So it would be nice to populate event.created with the current time.

webmat

There are two commits to consider, in tonight's push.

c1c3e04b is fully functional. It adds a few more fields for this transition, proposes event.code, etc.
bef4b8c4 is me trying to access what's inside event_data, and not succeeding 😆

Please check out the comments I've added in this review and tell me what you think, @ruflin & @andrewkroh.

webmat · 2019-01-30T04:18:14Z

winlogbeat/_meta/fields.common.yml

+      required: false
+
+    # Candidate to add to ECS?
+    - name: event.code


Semantics of Windows event ID are useful, they are a log message code, as Andrew explained in his previous comment.

I think it would make sense to have something like this in ECS, even outside of Windows. For example, MySQL uses error codes as well.

webmat · 2019-01-30T04:24:52Z

winlogbeat/_meta/fields.common.yml

@@ -60,23 +72,33 @@
      description: >
        The keywords are used to classify an event.

+    - name: channel
+
+    - name: provider_name


Just adding the two windows event log fields that were initially mapped like this:

channel was log_name. Example values:

Application

Security

Setup

provider_name was source_name. Example values:

Microsoft-Windows-Security-Auditing

Service Control Manager

Microsoft-Windows-CEIP

I think the Channel idea may be interesting to map to ECS (e.g. is there a correlation with syslog's facility?), but I'm not sure for the provider name. It's interesting data, but I'm not even sure I would map that to event.origin, which has come up every once in a while. Needs more thought (aka not for FF).

webmat · 2019-01-30T04:30:09Z

winlogbeat/eventlog/eventlog.go

 	}

+	m.Put("event.kind", "event")
+	m.Put("event.dataset", fmt.Sprintf("%v.%v", e.API, strings.ToLower(e.Channel)))


Experiment. Not really distinct datasets in the module.fileset of other beats. But here's what the values would look like, in the real world:

Modern windows:

wineventlog.application

wineventlog.security

wineventlog.system

...

Vista and older, iirc:

eventlogging.application

eventlogging.security

eventlogging.system

...

Does the field / json structure in each even heavily differ? If not I'm not sure if we should use dataset for this.

Question can also be asked differently: Could there be 1 visualisation which contains data from multiple dataset events?

"eventlogging" is a really old (before Vista) limited format for these events. Has way less information than "wineventlog". However afaik "wineventlog" was strictly additions vs "eventlogging".

Comparing between application & security, for example, I'm not sure. Probably somewhat possible (just like you can somewhat visualize apache.access & .error together with some caveats), when you have the same source name.

I understand the semantics are different from what we do elsewhere, as the other places the two positions here are really detailing which source this comes from: [source short name].[type of log generated by the source]`.

But unfortunately in Windows event logs, the closest thing resembling a source name is the provider name, which is really verbose (e.g. "Microsoft-Windows-Security-Auditing").

Perhaps we should just take the event.dataset out for now, and add later, after taking some time to consider how to do this right. The information itself is still present under winlog.*

++ on not using event.dataset for this as it's not consitent with other use cases.

Ok I answered this from a very different angle than you were asking, actually.

The winlog.* fields except the content of event_data or user_data should always be almost identical, as it's mostly metadata from the win event log pipeline.

Ok, removing.

The way I read it: There is just one event.dataset if we even populate it. It could be event.dataset: winlogevent or winlog.event or similar. We can still add it later.

It's an interesting idea. But I don't think the value is necessary at the moment (you could still build dashboards by querying other fields). And we might decide there's a better way to populate event.dataset if we were to add "modules" to winlogbeat or fold this whole thing into a "logbeat".

Not quite. It's more like the bullet lists I had pasted above. E.g. recent windows, you'd have a combination of:

wineventlog.application

wineventlog.security

wineventlog.system

...

But the "wineventlog" part never changes for a given host.

webmat · 2019-01-30T04:31:26Z

winlogbeat/eventlog/eventlog.go

+	m.Put("source_name", e.Provider.Name)
+  // Why not keep the Windows naming for those?
+	m.Put("channel", e.Channel)
+	m.Put("provider_name", e.Provider.Name)


Comparing the old Winlogbeat names that were mapped to a different field name vs just keeping the Windows terminology. I ❤️ the Windows name better. Thoughts?

webmat · 2019-01-30T04:36:36Z

winlogbeat/eventlog/eventlog.go

+	m.Put("channel", e.Channel)
+	m.Put("provider_name", e.Provider.Name)
+	m.Put("event.code", e.EventIdentifier.ID)
+	addOptional(m, "event.action", e.Task)


Sample values for Task/event.action:

Logon

Logoff

Special Logon

Credential Validation

However this field was set less than 50% of the time, in my playing around...

webmat · 2019-01-30T04:37:13Z

winlogbeat/eventlog/eventlog.go

@@ -139,7 +138,7 @@ func (e Record) ToEvent() beat.Event {
 // MapStr.


webmat · 2019-01-30T04:38:15Z

winlogbeat/eventlog/eventlog.go

 	addOptional(m, "processor_id", e.Execution.ProcessorID)
 	addOptional(m, "session_id", e.Execution.SessionID)
 	addOptional(m, "kernel_time", e.Execution.KernelTime)
 	addOptional(m, "user_time", e.Execution.UserTime)
 	addOptional(m, "processor_time", e.Execution.ProcessorTime)

+	// debugf("EventData field inspection %v", e.EventData.TargetUserName)
+	// debugf("EventData inspection full %v", e.EventData)
+	// addOptional(m, "user_dbg.name", e.EventData.TargetUserName)


Would like to be able to define this accessor, not sure how, see event.go.

webmat · 2019-01-30T04:39:03Z

winlogbeat/sys/event.go

@@ -133,6 +133,7 @@ func (t *TimeCreated) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error
 // EventData contains the event data. The EventData section is used if the
 // message provider template does not contain a UserData section.
 type EventData struct {
+	// TargetUserName    string `xml:"TargetUserName,attr"`


Not familiar with building these structs backed by a document. That's what I tried.

webmat · 2019-01-30T04:50:54Z

Here's two login events as parsed by the code in this PR, in case this helps understand the data.

login-events.zip

There's no data.json files in Winlogbeat, right? May be useful to add that at some point :-)

andrewkroh · 2019-01-30T14:32:37Z

winlogbeat/eventlog/eventlog.go

+
+  // Additional mapping ideas
+  // user.id
+  // - event_data.TargetUserSid (S-1-5-21-3541430928-2051711210-1391384369-1001)


Depending on anything in event_data is tricky because the value and meaning of these fields change between sources and even event_ids. But it's possible to do if we research the event manifests and add some conditionals.

I do think I agree that that the current values for user.{name,id} are not correct for ECS and correlation purposes. In most cases these values correspond to the user that the logging process is running as and not to the user that the activity is reporting on. In many cases the user.name is SYSTEM and that's not helpful.

The process_id is the PID of the logging process and again this isn't helpful for when you are trying to find events reported about a process. I think we need a place to put this information that's not process.* (or simply drop it).

[cutting it short, meeting time]

ruflin · 2019-01-31T13:08:32Z

@webmat Thanks for the zip. To follow ECS I'm wonder if winlogbeat should populate any fields on the top level or should we start to prefix them with windows or winlog.?

webmat · 2019-01-31T15:28:04Z

@ruflin Yes this is a very good point. I've found it strange to get into this event stream with lots of fields at the top level.

Although I've been hesitant to propose nesting this under a windows-centric name, because why would Windows be nested, and not Linux/Posix?

Also playing into this is the fact that the Posix ones are all nested at various places because there's no single way to do logging in Posix environments. Whereas on Windows, as far as I understand, almost everything uses the windows event log. Akin other logging transports: syslog specific stuff is nested under syslog.* or journald under journald.*.

So in that sense, perhaps we could indeed nest everything under "winlog.*", as the log event transport. And I would really like to take the opportunity to rename "channel" to "channel" (not log_name) and "Provider Name" to "provider_name (not source_name).

@andrewkroh WDYT?

Note that I'll also be talking with one of our resident Windows security expert tomorrow, and discuss some of these things.

The changes we have time to do prior to FF will be limited, but they can set the stage for a more consistent experience in the future.

webmat · 2019-02-01T04:29:38Z

After today's discussion, here's a radical update draft. Note that there's no attempt to set up the aliases yet & so on. I've just:

re-nested everything Windows log specific under winlog.*
Still kept around the already mapped ECS fields
Removed some of the incorrect mappings (e.g. pids were of the event log service, not the event's PID)
Updated my comment for the additional crap we can dig out of event_data

And here's a zip file with the same two events types as yesterday, an ssh login and a desktop login.

Let me know what you think.
login-events.zip

ruflin · 2019-02-01T13:05:01Z

I'm good with the nesting especially if we have migration aliases around for the old fields. Lets see what @andrewkroh thinks.

webmat · 2019-02-01T22:04:16Z

@andrewkroh @ruflin Hey guys, I would say this is feature complete. Please re-read the body of the PR, I've adjusted it to the approach and the thinking I've taken.

Code, fields and tests should be 100% done. If you see a problem in there, it's an actual bug.

Still to do: changelog, ecs-migration.yml (not forgetting the exceptions like type changes, etc), and adjust the documentation.

ruflin

Few minor questions, overall LGTM.

ruflin · 2019-02-04T08:57:11Z

winlogbeat/eventlog/eventlog.go

-	addOptional(m, "kernel_time", e.Execution.KernelTime)
-	addOptional(m, "user_time", e.Execution.UserTime)
-	addOptional(m, "processor_time", e.Execution.ProcessorTime)
+	addOptional(win, "process.pid", e.Execution.ProcessID)


Why not ECS?

Because that's not the "userland" PID, this is always the PID of the Windows service processing log events.

The interesting process.pid is at event_data.ProcessId, and we've decided not to dig in there right from Go / from the deployed agent.

ruflin · 2019-02-04T08:57:15Z

winlogbeat/eventlog/eventlog.go

-		"record_number": strconv.FormatUint(e.RecordID, 10),
+	// Windows Log Specific data
+	win := common.MapStr{
+		"channel":       e.Channel,
 		"event_id":      e.EventIdentifier.ID,


It's a message code (a given kind of message always has the same event id), it's not a unique ID.

So I've actually taken the liberty to map it to event.code, and will propose this gets added to ECS soon :-) So this nested one really should remain there.

You may push back on event.code, however. I'm good if you want to take it out for now.

Whatever you prefer 👍

ruflin · 2019-02-04T08:59:04Z

winlogbeat/eventlog/eventlog.go

 	}

+	m.Put("event.kind", "event")
+	m.Put("event.dataset", fmt.Sprintf("%v.%v", e.API, strings.ToLower(e.Channel)))


Does the field / json structure in each even heavily differ? If not I'm not sure if we should use dataset for this.

Question can also be asked differently: Could there be 1 visualisation which contains data from multiple dataset events?

ruflin · 2019-02-04T08:59:34Z

winlogbeat/eventlog/eventlog.go

+	m.Put("event.code", e.EventIdentifier.ID)
+	addOptional(m, "event.action", e.Task)
+
+	m.Put("host.hostname", e.Computer)


This should work as winlogbeat is always run on the same host (no remote connection) 👍

Winlogbeat can't accept remotely, but the Windows Event Log subsystem can do federation. So it may in fact be processing messages that come from remote hosts. I should double-check if Computer will always be the host at the origin of the event.

(host 1: Winlog) => (host 2: Winlog => Winlogbeat)

Won't this host.hostname value get overwritten by the libbeat publisher pipeline?

Like this stuff

beats/libbeat/publisher/pipeline/processor.go

Lines 130 to 134 in 2ec5e1a

// setup 7: add agent metadata

if !config.SkipAgentMetadata {

needsCopy := global.alwaysCopy || global.processors != nil

processors.add(actions.NewAddFields(createAgentFields(info), needsCopy))

}

.

Yes, you're probably right. I'll remove for now.

But the sad thing is that this value is likely the "true" source of the event. add_host_metadata will set this value to the machine running winlogbeat, and it won't be right, if this windows machine is receiving logs from other machines.

Note that this is a conundrum we've faced everywhere in Beats as well, and don't have a good answer to, yet.

winlogbeat/tests/system/winlogbeat.py

webmat · 2019-02-04T21:55:41Z

@andrewkroh @ruflin Ok this is ready for final review, including documentation, ecs-mig.yml & so on. There shouldn't be anything odd remaining. If that's the case, it's a bug.

dataset and host.hostname and the debugging statement are removed. I also did some minor cleanup in the field definitions.

Also fix small inconsistencies revealed by the tests

webmat · 2019-02-05T04:36:35Z

Ok, I've addressed the latest feedback.

As discussed, the dashboard error was on my size.

Just re-pushed with message_error migrated to error.message, error.code populated, and a dashboard adjusted to the field new names. I imported / re-exported, so the diff isn't clean. But it looks like this:

andrewkroh

LGTM.

It doesn't have to be in this PR, but I do want to have a breaking changes doc page for Winlogbeat that lists these changes (lots of users will need to update their LS configs).

andrewkroh · 2019-02-05T04:38:48Z

dev-tools/ecs-migration.yml

+# Fields moved without adjusting the name
+
+- from: activity_id
+  to: winlog.


It's missing the ending.

Good eye! Fixed :-)

webmat · 2019-02-05T04:53:41Z

Noted for adding a breaking changes page like Packetbeat's.

Ok if this is done post-FF? I'll see if I can get this done tomorrow, but not sure.

ruflin · 2019-02-05T08:59:08Z

I'm hoping for the breaking changes we can use #10405

ruflin

LGTM. Not sure why the dashboard needed adjustments as it should be automatically changed by the migration script. But please don't revert as you already have done it.

For the breaking changes, see my comment above and no rush on this one. Doc changes can go in at any time.

webmat · 2019-02-05T14:07:21Z

Yes, noted separately for the breaking changes docs. Hopefully your script can do most of the work indeed.

ruflin · 2019-02-05T19:18:52Z

winlogbeat/_meta/kibana/7/dashboard/Winlogbeat-overview.json

+      "attributes": {
+        "description": "",
+        "hits": 0,
+        "kibanaSavedObjectMeta": {


@webmat I'm just spotting this now but this a dashboard that is not decoded :-(

)

- Added `event.code` (See elastic/beats#10333) - Added `event.sequence` (See #129, elastic/beats#10760) - Added `event.provider` (See #321) - Note: Beats modules currently put the Syslog "programname" in `process.name` which is sometimes accurate, sometimes not (e.g. "kernel"). event.provider would be a better field for this. - Explain event.module and event.dataset without mentioning Beats

webmat requested a review from a team as a code owner January 24, 2019 20:24

webmat changed the title ~~Migrate Winlogbeat to ECS~~ Migrate Winlogbeat to ECS, take 2 Jan 24, 2019

webmat force-pushed the winlogbeat-ecs branch from e52df7f to ef9dd20 Compare January 25, 2019 13:54

webmat mentioned this pull request Jan 25, 2019

Migrate Winlogbeat to ECS #10169

Closed

ruflin reviewed Jan 25, 2019

View reviewed changes

andrewkroh reviewed Jan 27, 2019

View reviewed changes

ruflin added in progress Pull request is currently in progress. Winlogbeat ecs labels Jan 28, 2019

ruflin mentioned this pull request Jan 29, 2019

Transition Beats to ECS #8655

Closed

webmat force-pushed the winlogbeat-ecs branch from ef9dd20 to bef4b8c Compare January 30, 2019 04:10

webmat commented Jan 30, 2019

View reviewed changes

willemdh mentioned this pull request Jan 30, 2019

Netapp audit event ecs mapping considerations elastic/ecs#312

Closed

andrewkroh reviewed Jan 30, 2019

View reviewed changes

webmat force-pushed the winlogbeat-ecs branch from bef4b8c to 277f37f Compare February 1, 2019 04:25

webmat force-pushed the winlogbeat-ecs branch from e1e098a to 8fddd14 Compare February 1, 2019 19:27

webmat added review and removed in progress Pull request is currently in progress. labels Feb 1, 2019

webmat force-pushed the winlogbeat-ecs branch from 52bcc6d to 09e7f2a Compare February 1, 2019 22:02

ruflin reviewed Feb 4, 2019

View reviewed changes

andrewkroh reviewed Feb 4, 2019

View reviewed changes

winlogbeat/tests/system/winlogbeat.py Outdated Show resolved Hide resolved

webmat force-pushed the winlogbeat-ecs branch 2 times, most recently from cf9af57 to 3901390 Compare February 4, 2019 21:48

Mathieu Martin added 14 commits February 4, 2019 23:23

Move nest message_error, not actually sure it maps to ECS

2103a8a

Adjust the tests to the new field names

a675128

Also fix small inconsistencies revealed by the tests

Minor improvements

77dc2d4

Remove debugging statement, remove host.hostname and dataset from events

3351e81

Make sur all fields defined in same order, for diffing purposes

8bbebdd

Remove field defs used for debugging

1e37c2e

Changelog

8d311f1

Improve how event.code and event.original are defined in fields.yml

b917cbb

Document all field migrations in ecs-migration.yml

7ca07a8

Modify documentation to match field names now emitted by Winlogbeat

bf2a668

Space. The final frontier.

8125638

Set overwrite:true on the re-definition of event.original.

798b0f8

Migrate message_error to error.message. Populate error.code (new)

0e5b0b9

Update dashboard to the new field names

58c08c5

webmat force-pushed the winlogbeat-ecs branch from aabd026 to 58c08c5 Compare February 5, 2019 04:23

Adjust text labels on dashboard to better match the new field names

0c397a6

andrewkroh approved these changes Feb 5, 2019

View reviewed changes

Fix entry in ecs-mig.yml

7d26f79

ruflin approved these changes Feb 5, 2019

View reviewed changes

webmat merged commit b35b1ad into elastic:master Feb 5, 2019

webmat deleted the winlogbeat-ecs branch February 5, 2019 14:10

ruflin reviewed Feb 5, 2019

View reviewed changes

webmat added a commit that referenced this pull request Feb 5, 2019

Fix JSON encoding issue with my Winlogbeat dashboard from #10333 (#10586

0e4df24

)

webmat mentioned this pull request Feb 6, 2019

Missing Event/type ID, event.id is very ambiguous elastic/ecs#306

Closed

webmat mentioned this pull request Apr 23, 2019

Additions to event elastic/ecs#439

Merged

		@@ -139,7 +138,7 @@ func (e Record) ToEvent() beat.Event {
		// MapStr.

	// setup 7: add agent metadata
	if !config.SkipAgentMetadata {
	needsCopy := global.alwaysCopy \|\| global.processors != nil
	processors.add(actions.NewAddFields(createAgentFields(info), needsCopy))
	}

Migrate Winlogbeat to ECS, take 2 #10333

Migrate Winlogbeat to ECS, take 2 #10333

Conversation

webmat commented Jan 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Feb 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat commented Jan 30, 2019 • edited Loading

andrewkroh Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

ruflin commented Jan 31, 2019

webmat commented Jan 31, 2019

webmat commented Feb 1, 2019

ruflin commented Feb 1, 2019

webmat commented Feb 1, 2019

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat commented Feb 4, 2019

webmat commented Feb 5, 2019 • edited Loading

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat commented Feb 5, 2019

ruflin commented Feb 5, 2019

ruflin left a comment

Choose a reason for hiding this comment

webmat commented Feb 5, 2019

Choose a reason for hiding this comment

webmat commented Jan 24, 2019 •

edited

Loading

webmat Feb 4, 2019 •

edited

Loading

webmat Jan 30, 2019 •

edited

Loading

webmat commented Jan 30, 2019 •

edited

Loading

andrewkroh Jan 30, 2019 •

edited

Loading

webmat commented Feb 5, 2019 •

edited

Loading