-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fsnotify-backed recursive watcher for Linux #5833
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,11 +23,11 @@ The operating system features that power this feature are as follows. | |
|
||
* Linux - `inotify` is used, and therefore the kernel must have inotify support. | ||
Inotify was initially merged into the 2.6.13 Linux kernel. | ||
* macOS (Darwin) - `kqueue` is used. It requires one file descriptor for each | ||
file so please check the `ulimit` values used with {beatname_uc}. The FSEvents | ||
API was considered for the implementation, but FSEvents coalesces multiple | ||
notifications into a single event which is inconsistent with the metricset's | ||
behavior on other operating systems. | ||
* macOS (Darwin) - Uses the `FSEvents` API, present since macOS 10.5. This API | ||
coalesces multiple changes to a file into a single event. {beatname_uc} translates | ||
this coalesced changes into a meaningful sequence of actions. However, | ||
in rare situations the reported events may have a different ordering than what | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good explanation |
||
actually happened. | ||
* Windows - `ReadDirectoryChangesW` is used. | ||
|
||
The file metricset should not be used to monitor paths on network file systems. | ||
|
@@ -53,11 +53,11 @@ Linux. | |
file.scan_rate_per_sec: 50 MiB | ||
file.max_file_size: 100 MiB | ||
file.hash_types: [sha1] | ||
file.recursive: false | ||
---- | ||
|
||
*`file.paths`*:: A list of paths (directories or files) to watch. The watches | ||
are non-recursive and globs are not supported. The specified paths should exist | ||
when the metricset is started. | ||
*`file.paths`*:: A list of paths (directories or files) to watch. Globs are | ||
not supported. The specified paths should exist when the metricset is started. | ||
|
||
*`file.scan_at_start`*:: A boolean value that controls if {beatname_uc} scans | ||
over the configured file paths at startup and send events for the files | ||
|
@@ -84,4 +84,10 @@ a suffix to the value. The supported units are `b` (default), `kib`, `kb`, `mib` | |
|
||
*`file.hash_types`*:: A list of hash types to compute when the file changes. | ||
The supported hash types are md5, sha1, sha224, sha256, sha384, sha512, | ||
sha512_224, sha512_256, sha3_224, sha3_256, sha3_384 and sha3_512. The default value is sha1. | ||
sha512_224, sha512_256, sha3_224, sha3_256, sha3_384 and sha3_512. The default | ||
value is sha1. | ||
|
||
*`file.recursive`*:: By default, the watches set to the paths specified in | ||
`file.paths` are not recursive. This means that only changes to the contents | ||
of this directories are watched. If `file.recursive` is set to `true`, the file | ||
metric will watch for changes on this directories and all their subdirectories. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,142 @@ | ||
package monitor | ||
|
||
import ( | ||
"fmt" | ||
"os" | ||
path_pkg "path" | ||
"strings" | ||
) | ||
|
||
// VisitOrder is a two-valued flag used to control how directories are visited. | ||
type VisitOrder int8 | ||
|
||
const ( | ||
// PreOrder has directories visited before their contents. | ||
PreOrder VisitOrder = iota | ||
// PostOrder has directories visited after their contents. | ||
PostOrder | ||
) | ||
|
||
var ( | ||
// PathSeparator can be used to override the operating system separator. | ||
PathSeparator = string(os.PathSeparator) | ||
) | ||
|
||
// FileTree represents a directory in a filesystem-tree structure. | ||
type FileTree map[string]FileTree | ||
|
||
// VisitFunc is the type for a callback to visit the entries on a directory | ||
// and its subdirectories. | ||
type VisitFunc func(path string, isDir bool) error | ||
|
||
// AddFile adds a file to a FileTree. If the path includes subdirectories | ||
// they are created as necessary. | ||
func (tree FileTree) AddFile(path string) error { | ||
return tree.add(path_pkg.Clean(path), nil) | ||
} | ||
|
||
// AddDir adds a directory to a FileTree. If the path includes subdirectories | ||
// they are created as necessary. | ||
func (tree FileTree) AddDir(path string) error { | ||
return tree.add(path_pkg.Clean(path), FileTree{}) | ||
} | ||
|
||
// Remove an entry from a FileTree. | ||
func (tree FileTree) Remove(path string) error { | ||
components := strings.Split(path, PathSeparator) | ||
last := -1 | ||
for pos := len(components) - 1; pos >= 0; pos-- { | ||
if len(components[pos]) != 0 { | ||
last = pos | ||
break | ||
} | ||
} | ||
if last > 0 { | ||
subtree, err := tree.getByComponents(path, components[:last]) | ||
if err != nil { | ||
return err | ||
} | ||
delete(subtree, components[last]) | ||
} | ||
return nil | ||
} | ||
|
||
// Visit calls the callback function for the given path and recursively all its | ||
// contents, if a directory path is passed. | ||
func (tree FileTree) Visit(path string, order VisitOrder, fn VisitFunc) error { | ||
entry, err := tree.At(path) | ||
if err != nil { | ||
return err | ||
} | ||
return entry.visitDirRecursive(path_pkg.Clean(path), order, fn) | ||
} | ||
|
||
// At returns a new FileTree rooted at the given path. | ||
func (tree FileTree) At(path string) (FileTree, error) { | ||
return tree.getByComponents(path, strings.Split(path, PathSeparator)) | ||
} | ||
|
||
func (tree FileTree) add(path string, value FileTree) error { | ||
components := strings.Split(path, PathSeparator) | ||
dir, last := tree, len(components)-1 | ||
for i := 0; i < last; i++ { | ||
if len(components[i]) == 0 { | ||
continue | ||
} | ||
if next, exists := dir[components[i]]; exists { | ||
if next == nil { | ||
return fmt.Errorf("directory expected: '%s' in %s", components[i], path) | ||
} | ||
dir = next | ||
} else { | ||
newDir := FileTree{} | ||
dir[components[i]] = newDir | ||
dir = newDir | ||
} | ||
} | ||
dir[components[last]] = value | ||
return nil | ||
} | ||
|
||
func (tree FileTree) getByComponents(path string, components []string) (FileTree, error) { | ||
dir, exists := tree, false | ||
for _, item := range components { | ||
if len(item) != 0 { | ||
if dir == nil { | ||
// previous component is a file, not a directory | ||
return nil, fmt.Errorf("path component %s is a file: %s", item, path) | ||
} | ||
if dir, exists = dir[item]; !exists { | ||
return nil, fmt.Errorf("path component %s not found in %s", item, path) | ||
} | ||
} | ||
} | ||
return dir, nil | ||
} | ||
|
||
func (tree FileTree) visitDirRecursive(path string, order VisitOrder, fn VisitFunc) error { | ||
if tree == nil { | ||
return fn(path, false) | ||
} | ||
if order == PreOrder { | ||
if err := fn(path, true); err != nil { | ||
return err | ||
} | ||
} | ||
for name, content := range tree { | ||
fullpath := path_pkg.Join(path, name) | ||
if content == nil { | ||
if err := fn(fullpath, false); err != nil { | ||
return err | ||
} | ||
} else { | ||
if err := content.visitDirRecursive(fullpath, order, fn); err != nil { | ||
return err | ||
} | ||
} | ||
} | ||
if order == PostOrder { | ||
return fn(path, true) | ||
} | ||
return nil | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're adding it here then I would also put it into https://github.com/elastic/beats/blob/master/auditbeat/module/audit/file/_meta/docs.asciidoc#configuration-options.