macOS recursive file monitoring #5575

adriansr · 2017-11-14T05:45:33Z

This patch adds a new file monitoring backend to the integrity module, using the FSEvents facility from macOS.

Coalesced events

FSEvents uses a bitmask to notify for more than one event on the same path. For example created|modified|deleted. Ordering information is lost, so it's impossible to tell from
the event itself if the file was modified, then deleted and then re-created, or first created,
then modified and then deleted. The ordering in which the events are stored in Elasticsearch
seems of little importance as they usually get assigned the same timestamp (granularity of one millisecond).

This patch orders the set of actions in a single event to be meaningful depending if the file existed in the beat database and if it exists anymore at the moment of processing the event.

#5421

andrewkroh

I tested this out on my machine and your approach works well.

However I found that due to the diff'ing Auditbeat does against the last known state that when multiple actions are coalesced (e.g. changed and attributes_modified) that Auditbeat only sends a single event with action=updated. Since the two event are basically clones with different Actions the second event gets discarded. See hasFileChangedSinceLastEvent() for details on what happens when it diffs the events.

You have updated Action to be a bitmask such that it can represent all the changes that have occurred. So instead of pushing one event for each individual Action from fsreader I say we send one event with multiple Actions. I think the only time we need two events is when the file has been deleted and then recreated. Your logic for getting the ordering correct between these two events seems good.

Then in the events that Auditbeat outputs the action field should be changed to be an array so that we can say, for example, that a file was updated and its attributes were modified or a file was updated and deleted.

Since we are changing the action field I think diffEvents needs to be updated for consistency. Currently it only reports a single Action. Say for example that a file is updated and chmod'ed while Auditbeat is stopped. When Auditbeat restarts the scanner should generate an event with action: [updated, attributes_modified].

And I think we should make the change to always use FSEvent on darwin (for recursive and non-recursive). When in non-recursive mode the implementation will need to drop events for files that aren't in the configured dir.

andrewkroh · 2017-11-15T21:55:56Z

auditbeat/module/audit/file/eventreader_fsevents.go

+		// into a single notification. This might be useful to reduce load in
+		// some scenarios so that it deserves its own configuration flag.
+		Latency: 0,
+		Flags:   fsevents.FileEvents | fsevents.WatchRoot,


Should IgnoreSelf be added to prevent Auditbeat from getting into some kind of feedback loop (like if someone decided to monitor /var/lib/auditbeat where the database is maintained)?

andrewkroh · 2017-11-15T21:59:19Z

auditbeat/module/audit/file/eventreader_fsevents.go

+func (r *fsreader) Start(done <-chan struct{}) (<-chan Event, error) {
+	r.stream.Start()
+	go r.consumeEvents(done)
+	logp.Info("%v started fsevents recursive watched", logPrefix)


s/watched/watch/?

andrewkroh · 2017-11-23T13:19:38Z

auditbeat/module/audit/file/action.go

+
+func (action Action) isMultiple() bool {
+	// checks if there is more than one bit set
+	return action != 0 && (action&(action-1)) != 0


There is a bits.OnesCount8 function that is supposed to be optimized.

andrewkroh · 2017-11-23T13:25:14Z

auditbeat/module/audit/file/eventreader_fsevents.go

+	if err != nil {
+		return 0, errors.Wrap(err, fmt.Sprintf("failed to stat"))
+	}
+	meta, err := NewMetadata(path, info)


Given that you need only the inode and file type I would use the os.FileInfo directly to avoid some of the extra work that NewMetadata does (like lookup username and group).

andrewkroh · 2017-11-23T13:27:06Z

auditbeat/module/audit/file/eventreader_fsnotify.go

@@ -19,6 +20,10 @@ type reader struct {

 // NewEventReader creates a new EventProducer backed by fsnotify.
 func NewEventReader(c Config) (EventProducer, error) {
+


Please drop this newline.

andrewkroh · 2017-11-23T13:33:02Z

auditbeat/module/audit/file/event.go

@@ -211,7 +180,7 @@ func (e *Event) String() string {
 	return string(data)
 }

-func buildMapStr(e *Event) common.MapStr {
+func buildMapStr(e *Event, existsBefore, existsNow bool) common.MapStr {


Could existsNow be determined by looking at e? Then you wouldn’t need the third param.

This patch adds a new file monitoring to the integrity module that uses the FSEvents facility from macOS. Uses fsnotify's implementation of fsevents library. Non-recursive behavior is emulated by filtering events.

andrewkroh · 2017-11-27T22:39:04Z

jenkins, test it

andrewkroh · 2017-11-27T22:40:53Z

This looks pretty complete. @adriansr, should the "in progress" be removed?

andrewkroh · 2017-11-27T23:06:17Z

The way Action is persisted is going to need to be updated. Or perhaps there is no use in persisting this value?

beats/auditbeat/module/audit/file/flatbuffers.go

Lines 151 to 164 in d92aa2e

    
           switch e.Action { 
        
           case AttributesModified: 
        
           	schema.EventAddAction(b, schema.ActionAttributesModified) 
        
           case Created: 
        
           	schema.EventAddAction(b, schema.ActionCreated) 
        
           case Deleted: 
        
           	schema.EventAddAction(b, schema.ActionDeleted) 
        
           case Updated: 
        
           	schema.EventAddAction(b, schema.ActionUpdated) 
        
           case Moved: 
        
           	schema.EventAddAction(b, schema.ActionMoved) 
        
           case ConfigChange: 
        
           	schema.EventAddAction(b, schema.ActionConfigChanged) 
        
           }

It could be defined as enum Action : ubyte (bit_flags) { in the schema and the compiler will generate values based on power of two.

adriansr · 2017-11-28T11:52:44Z

Ok I updated the flatbuffers to persist the action as a bitmask, although the field is not used.

adriansr force-pushed the auditbeat/recursive_macos branch from 0df4724 to 00938b9 Compare November 14, 2017 05:59

tsg added Auditbeat review labels Nov 14, 2017

adriansr requested a review from andrewkroh November 14, 2017 12:59

adriansr added the in progress Pull request is currently in progress. label Nov 15, 2017

andrewkroh reviewed Nov 15, 2017

View reviewed changes

adriansr force-pushed the auditbeat/recursive_macos branch from 00938b9 to 0cb6575 Compare November 23, 2017 01:17

andrewkroh reviewed Nov 23, 2017

View reviewed changes

adriansr added 4 commits November 25, 2017 14:54

auditbeat: macOS recursive file monitoring (elastic#5421)

5ee22f6

This patch adds a new file monitoring to the integrity module that uses the FSEvents facility from macOS. Uses fsnotify's implementation of fsevents library. Non-recursive behavior is emulated by filtering events.

Fix fsnotify build

fac8edb

Fix some small issues

58ee5fc

Use os.SameFile instead of comparing inodes

93462ea

adriansr force-pushed the auditbeat/recursive_macos branch from 5706b78 to 93462ea Compare November 25, 2017 13:54

Persist action as a bit mask

70b6c76

andrewkroh merged commit 37bfd85 into elastic:master Nov 28, 2017

ruflin mentioned this pull request Dec 1, 2017

Packaging broken because of uint64 overflow in auditbeat #5783

Closed

monicasarbu removed the in progress Pull request is currently in progress. label Dec 4, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS recursive file monitoring #5575

macOS recursive file monitoring #5575

adriansr commented Nov 14, 2017 •

edited

Loading

andrewkroh left a comment •

edited

Loading

andrewkroh Nov 15, 2017

andrewkroh Nov 15, 2017

andrewkroh Nov 23, 2017 •

edited

Loading

andrewkroh Nov 23, 2017

andrewkroh Nov 23, 2017

andrewkroh Nov 23, 2017

andrewkroh commented Nov 27, 2017

andrewkroh commented Nov 27, 2017

andrewkroh commented Nov 27, 2017 •

edited

Loading

adriansr commented Nov 28, 2017

		@@ -19,6 +20,10 @@ type reader struct {

		// NewEventReader creates a new EventProducer backed by fsnotify.
		func NewEventReader(c Config) (EventProducer, error) {

macOS recursive file monitoring #5575

macOS recursive file monitoring #5575

Conversation

adriansr commented Nov 14, 2017 • edited Loading

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

andrewkroh Nov 15, 2017

Choose a reason for hiding this comment

andrewkroh Nov 15, 2017

Choose a reason for hiding this comment

andrewkroh Nov 23, 2017 • edited Loading

Choose a reason for hiding this comment

andrewkroh Nov 23, 2017

Choose a reason for hiding this comment

andrewkroh Nov 23, 2017

Choose a reason for hiding this comment

andrewkroh Nov 23, 2017

Choose a reason for hiding this comment

andrewkroh commented Nov 27, 2017

andrewkroh commented Nov 27, 2017

andrewkroh commented Nov 27, 2017 • edited Loading

adriansr commented Nov 28, 2017

adriansr commented Nov 14, 2017 •

edited

Loading

andrewkroh left a comment •

edited

Loading

andrewkroh Nov 23, 2017 •

edited

Loading

andrewkroh commented Nov 27, 2017 •

edited

Loading