Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Auditbeat] Login metricset #9327

Merged
merged 54 commits into from
Jan 30, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
2774cbc
First working version.
Nov 5, 2018
172273a
Use filepath pattern to catch rotated files.
Nov 6, 2018
1cf7585
Add state.
Nov 19, 2018
81831e2
Test data.
Nov 19, 2018
bd03a7f
Exclude empty fields from event.
Nov 19, 2018
a657a25
TTY lookup.
Nov 20, 2018
7675662
UTMP debug log.
Nov 21, 2018
4f712a7
Add LoginRecord.UID.
Nov 27, 2018
980225b
Move structs to their proper files.
Nov 28, 2018
feb3685
Move FileRecord to utmp.go.
Nov 28, 2018
f7dcb48
Move most logic to utmp.go.
Nov 28, 2018
1051f20
Properly separate logic, add all event logic.
Nov 29, 2018
eb9c40a
Add TODO comments.
Nov 30, 2018
e467633
Use top-level fields where possible.
Dec 2, 2018
126230b
Update test data.
Dec 2, 2018
760d46e
Update fields.yml.
Dec 2, 2018
4cfac44
Minor improvements.
Dec 2, 2018
a54c658
Config.
Dec 3, 2018
2f0bc9d
Hound fixes.
Dec 3, 2018
e05e58f
Add cgo to build tags.
Dec 3, 2018
0b2c555
Change event.summary to message.
Dec 3, 2018
944b9ce
Revert validate:required.
Dec 3, 2018
6e446d5
Change to source.domain.
Dec 5, 2018
7cc8c03
Change constants to loginRecordType.
Dec 5, 2018
6bdb263
Remove fileInfos().
Dec 5, 2018
ea48b62
mage update
Jan 22, 2019
51194c2
Config and docs adjustments.
Jan 22, 2019
6680660
Lowercase config.
Jan 22, 2019
e0c66da
Namespace
Jan 22, 2019
e602899
Addressed some minor review comments.
Jan 22, 2019
7fa836f
Pure Go reader.
Jan 22, 2019
ad2220f
Remove now unneeded seccomp action.
Jan 22, 2019
1458563
Make file re-reading non-recursive.
Jan 22, 2019
a7ca217
Use channels instead of passing arrays to reduce memory footprint.
Jan 23, 2019
0823d7e
Save offset instead of last read record.
Jan 23, 2019
9c9e1b6
Handle failed logins from btmp files as well.
Jan 23, 2019
9774ca0
System test
Jan 23, 2019
04af736
Remove unreachable code.
Jan 23, 2019
3b8ed59
Make Hound happy
Jan 23, 2019
175c992
Change to user.terminal and document event.origin
Jan 23, 2019
3b8002a
Config and docs.
Jan 23, 2019
1a8bbbc
Changelog
Jan 23, 2019
5915f85
Update data.json
Jan 24, 2019
3617549
Test with samle wtmp file.
Jan 24, 2019
3e21182
Remove cgo build constraint.
Jan 25, 2019
03eedee
Improve message string building.
Jan 25, 2019
539feb4
Add UtType.
Jan 25, 2019
0453ce2
Respect machine byte order.
Jan 25, 2019
028663d
Fix debug message.
Jan 25, 2019
c649510
Correct type for all UtType values.
Jan 28, 2019
b1f70fb
Skip login dataset tests when not on a little-endian system.
Jan 28, 2019
fe229c7
mage update
Jan 29, 2019
71d61c2
Remove user.id from system test.
Jan 29, 2019
d0b1969
Fix unit test in integration env.
Jan 30, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add `group.id` (GID) and `group.name` for ECS. {pull}10195[10195]
- System module `process` dataset: Add user information to processes. {pull}9963[9963]
- Add system `package` dataset. {pull}10225[10225]
- Add system module `login` dataset. {pull}9327[9327]

*Filebeat*

Expand Down
22 changes: 22 additions & 0 deletions auditbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6175,6 +6175,28 @@ These are the fields generated by the system module.
*`event.origin`*::
+
--
type: keyword
Origin of the event. This can be a file path (e.g. `/var/log/log.1`), or the name of the system component that supplied the data (e.g. `netlink`).
--
*`user.terminal`*::
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be good to open a PR to elastic/ecs suggesting that this be added. I think the auditd module gives tty info.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does. #10192 will fill process.terminal for the Filebeat auditd module. I wonder if we should use the same field, and whether it should be user.terminal or process.terminal. @andrewkroh @webmat what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah in my PR I've been mapping both auditd.log.tty and auditd.log.terminal to process.terminal, thinking it was inconsistency on the auditd side. This makes me wonder if one is actually user-specific and one process-specific. I'll dig into this some more for my PR.

But I do think user.terminal makes sense, if you have terminal info for users.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found some light reading for the week-end: Auditd field dictionary

+
--
type: keyword
Terminal of the user.
--
[float]
== system.audit fields
Expand Down
7 changes: 7 additions & 0 deletions x-pack/auditbeat/auditbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ auditbeat.modules:
- module: system
datasets:
- host # General host information, e.g. uptime, IPs
- login # User logins, logouts, and system boots.
- package # Installed, updated, and removed packages
- process # Started and stopped processes
- socket # Opened and closed sockets
Expand All @@ -139,6 +140,12 @@ auditbeat.modules:
# detect any changes.
user.detect_password_changes: true

# File patterns of the login record files.
# wtmp: History of successful logins, logouts, and system shutdowns and boots.
# btmp: Failed login attempts.
login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*

#================================ General ======================================

# The name of the shipper that publishes the network data. It can be used to group
Expand Down
5 changes: 5 additions & 0 deletions x-pack/auditbeat/auditbeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ auditbeat.modules:
- module: system
datasets:
- host # General host information, e.g. uptime, IPs
- login # User logins, logouts, and system boots.
- package # Installed, updated, and removed packages
- process # Started and stopped processes
- socket # Opened and closed sockets
Expand All @@ -65,6 +66,10 @@ auditbeat.modules:
# detect any changes.
user.detect_password_changes: true

# File patterns of the login record files.
login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*

#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
Expand Down
11 changes: 11 additions & 0 deletions x-pack/auditbeat/docs/modules/system.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ sample suggested configuration.
- module: system
datasets:
- host
- login
- package
- process
- socket
Expand Down Expand Up @@ -87,6 +88,7 @@ so a longer polling interval can be used.
- module: system
datasets:
- host
- login
- package
- user
period: 1m
Expand All @@ -113,6 +115,7 @@ auditbeat.modules:
- module: system
datasets:
- host # General host information, e.g. uptime, IPs
- login # User logins, logouts, and system boots.
- package # Installed, updated, and removed packages
- process # Started and stopped processes
- socket # Opened and closed sockets
Expand All @@ -127,6 +130,10 @@ auditbeat.modules:
# /etc/passwd and /etc/shadow and store a hash locally to
# detect any changes.
user.detect_password_changes: true
# File patterns of the login record files.
login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*
----

[float]
Expand All @@ -136,6 +143,8 @@ The following datasets are available:

* <<{beatname_lc}-dataset-system-host,host>>

* <<{beatname_lc}-dataset-system-login,login>>

* <<{beatname_lc}-dataset-system-package,package>>

* <<{beatname_lc}-dataset-system-process,process>>
Expand All @@ -146,6 +155,8 @@ The following datasets are available:

include::system/host.asciidoc[]

include::system/login.asciidoc[]

include::system/package.asciidoc[]

include::system/process.asciidoc[]
Expand Down
21 changes: 21 additions & 0 deletions x-pack/auditbeat/docs/modules/system/login.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
////
This file is generated! See scripts/docs_collector.py
////

[id="{beatname_lc}-dataset-system-login"]
=== System login dataset

include::../../../module/system/login/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the dataset, see the
<<exported-fields-system,exported fields>> section.

Here is an example document generated by this dataset:

[source,json]
----
include::../../../module/system/login/_meta/data.json[]
----
1 change: 1 addition & 0 deletions x-pack/auditbeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions x-pack/auditbeat/module/system/_meta/config.yml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
- module: system
datasets:
- host # General host information, e.g. uptime, IPs
{{ if eq .GOOS "linux" -}}
- login # User logins, logouts, and system boots.
{{- end }}
{{ if ne .GOOS "windows" -}}
- package # Installed, updated, and removed packages
{{- end }}
Expand Down Expand Up @@ -38,3 +41,13 @@
# detect any changes.
user.detect_password_changes: true
{{- end }}

{{ if eq .GOOS "linux" -}}
# File patterns of the login record files.
{{- if .Reference }}
# wtmp: History of successful logins, logouts, and system shutdowns and boots.
# btmp: Failed login attempts.
{{- end }}
login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*
{{- end }}
2 changes: 2 additions & 0 deletions x-pack/auditbeat/module/system/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ sample suggested configuration.
- module: system
datasets:
- host
- login
- package
- process
- socket
Expand Down Expand Up @@ -82,6 +83,7 @@ so a longer polling interval can be used.
- module: system
datasets:
- host
- login
- package
- user
period: 1m
Expand Down
24 changes: 21 additions & 3 deletions x-pack/auditbeat/module/system/_meta/fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,25 @@
These are the fields generated by the system module.
release: experimental
fields:
- name: system.audit
type: group

- name: event
type: group
fields:
- name: origin
webmat marked this conversation as resolved.
Show resolved Hide resolved
type: keyword
description: >
fields:
Origin of the event. This can be a file path (e.g. `/var/log/log.1`),
or the name of the system component that supplied the data (e.g. `netlink`).
- name: user
type: group
fields:
- name: terminal
webmat marked this conversation as resolved.
Show resolved Hide resolved
type: keyword
description: >
Terminal of the user.
- name: system.audit
type: group
description: >
fields:
2 changes: 1 addition & 1 deletion x-pack/auditbeat/module/system/fields.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 30 additions & 0 deletions x-pack/auditbeat/module/system/login/_meta/data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"event": {
"action": "user_login",
"dataset": "login",
"module": "system",
"origin": "/var/log/wtmp.1",
"outcome": "success",
"type": "event"
},
"message": "Login by user vagrant (UID: 1000) on pts/1 (PID: 17559) from 10.0.2.2 (IP: 10.0.2.2)",
"process": {
"pid": 17559
},
"service": {
"type": "system"
},
"source": {
"ip": "10.0.2.2"
},
"user": {
"id": 1000,
"name": "vagrant",
"terminal": "pts/1"
}
}
7 changes: 7 additions & 0 deletions x-pack/auditbeat/module/system/login/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[role="xpack"]

experimental[]

This is the `login` dataset of the system module.

It is implemented for Linux only.
20 changes: 20 additions & 0 deletions x-pack/auditbeat/module/system/login/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

// +build linux

package login

// config defines the metricset's configuration options.
type config struct {
WtmpFilePattern string `config:"login.wtmp_file_pattern"`
BtmpFilePattern string `config:"login.btmp_file_pattern"`
}

func defaultConfig() config {
return config{
WtmpFilePattern: "/var/log/wtmp*",
BtmpFilePattern: "/var/log/btmp*",
}
}
Loading