Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECS normalization #62

Merged
merged 11 commits into from
May 15, 2020
Merged

ECS normalization #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b421380
Add normalization stuff
May 14, 2020
e661278
Switch apparmor/selinux ordering because it matters
May 14, 2020
d853608
Fix custom unmarshaler
May 14, 2020
54e16a6
Add support for compound events
May 14, 2020
8ff6108
Fix AUDIT_LOGIN
May 14, 2020
ddba1ce
more default updates
May 14, 2020
c1d127f
Fix process start v creation
May 14, 2020
39525d6
Add write syscall since I added read
May 14, 2020
b6f7bcd
Add syscall descriptions and a missing syscall comment
May 14, 2020
8c22c7a
remove additional select macros from comment list
May 14, 2020
2a64f9c
Update with comments, types changed, README update, and CHANGELOG entry
May 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Change Log

All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).

Expand All @@ -12,6 +13,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).

- Added support for big endian. #48
- Added semantic versioning support via go modules. #61
- Add ECS categorization support for events by record type and syscall. #62

### Removed

Expand All @@ -29,7 +31,6 @@ This project adheres to [Semantic Versioning](http://semver.org/).

### Removed


## [0.3.0]

### Added
Expand Down Expand Up @@ -75,12 +76,12 @@ This project adheres to [Semantic Versioning](http://semver.org/).
PATH records. #20

## [0.0.7]

### Added

- Added WaitForPendingACKs to receive pending ACK messages from the kernel. #14
- The AuditClient will unregister with the kernel if `SetPID` has been called. #19

### Changed

- auparse - Fixed an issue where the proctitle value was being truncated. #15
Expand All @@ -102,11 +103,13 @@ This project adheres to [Semantic Versioning](http://semver.org/).
## [0.0.5]

### Changed

- auparse - Apply hex decoding to CWD field. #10

## [0.0.4]

### Added

- Add a package for building audit rules that can be added to the kernel.
- Add GetRules, DeleteRules, DeleteRule, and AddRule methods to AuditClient.
- auparse - Add conversion of POSIX exit code values to their name.
Expand All @@ -115,12 +118,14 @@ This project adheres to [Semantic Versioning](http://semver.org/).
## [0.0.3]

### Added

- auparse - Convert auid and session values of `4294967295` or `-1` to "unset". #5
- auparse - Added `MarshallText` method to AuditMessageType to enable the value
to be marshaled as a string in JSON. faabfa94ec9479bdc1ad6c0334ff178b8193fce5
- aucoalesce - Enhanced aucoalesce to normalize events. 666ff1c30fe624e9fcd9a108b20fceb82331f5fa

### Changed

- Rename RawAuditMessage fields `MessageType` and `RawData` to `Type` and
`Data` respectively. 8622833714fccd7810669b1265df1c1f918ec0c4
- Make Reassembler concurrency-safe. c57b59c20a684e2a6298a1a5929a79192d76d61b
Expand All @@ -134,6 +139,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
## [0.0.2]

### Added

- Added `libaudit.Reassembler` for reassembling out of order or interleaved
messages and providing notification for lost events based on gaps in sequence
numbers. a60bdd3b1b642cc80a3872d999114ae675456768
Expand All @@ -148,6 +154,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
event. #1

### Changed

- auparse - Changed the behavior of `ParseLogLine()` and `Parse()` to only parse
the message header. To parse the message body, call `Data()` on the returned
`AuditMessage`.
Expand All @@ -159,6 +166,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
## [0.0.1]

### Added

- Added AuditClient for communicating with the Linux Audit Framework in the
Linux kernel.
- Added auparse package for parsing audit logs.
23 changes: 22 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ The second is _auparse_ which parses the log files from the Linux auditd
process or the output of the _audit_ example command. It combines related log
messages that are a part of the same event.


```
$ go install github.com/elastic/go-libaudit/cmd/auparse
$ sudo cat /var/log/audit/audit.log | auparse
Expand Down Expand Up @@ -111,6 +110,12 @@ data:
grantors: pam_env,pam_unix
op: PAM:setcred
terminal: /dev/pts/1
ecs:
event:
category:
- authentication
type:
- info

---
timestamp: 2016-12-07T02:22:14.303Z
Expand Down Expand Up @@ -148,6 +153,12 @@ data:
grantors: pam_keyinit,pam_limits
op: PAM:session_open
terminal: /dev/pts/1
ecs:
event:
category:
- authentication
type:
- info

---
timestamp: 2016-12-07T02:22:14.304Z
Expand Down Expand Up @@ -250,4 +261,14 @@ paths:
ogid: "0"
ouid: "0"
rdev: "00:00"
ecs:
event:
category:
- process
type:
- start
```

## ECS compatibility

This currently provides [Elastic Common Schema (ECS) 1.5](https://www.elastic.co/guide/en/ecs/current/index.html) categorization support for some of the more prominent or meaningful auditd events and syscalls.
39 changes: 37 additions & 2 deletions aucoalesce/coalesce.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ import (
// package does not have a constant defined for this.
const modeBlockDevice = 060000

// ECSEvent contains ECS-specific categorization fields
type ECSEvent struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non blocking
Would event.kind be held somewhere else? or maybe the caller of the library is responsible for that?

What do you think about having something about ECS categorization in the struct type? That might provide clues to the type of fields in the struct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my initial thought is that it could be in the caller--namely because I'm pretty sure we're working only with event kinds--but then again would "anomaly" events actually be alert kinds? If so, or if we just want to be explicit, I can add kind in a follow-up PR.

For the second question, I'm not entirely sure I follow--do you meaning adding constant enums for the categorization values themselves? If so I'd almost prefer to do that in the ECS go generated code from the ECS repo itself (which ideally we'd be using here too--but we still need to add tags and formalize on things like omitempty, pointer usage, etc. before doing that).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with it being the callers responsibility, I just wanted to double check that we weren't missing one of the ECS categorization fields.

I was thinking the name of the type might be better as ECSCategorization or ECSCat or something like that, since it really just holds the categorization info, nothing else about ECS.

Enums would be awesome and I agree it would be good to do that work once in ECS repo.

Category []string `json:"category,omitempty" yaml:"category,omitempty"`
Type []string `json:"type,omitempty" yaml:"type,omitempty"`
Outcome string `json:"outcome,omitempty" yaml:"outcome,omitempty"`
}

type Event struct {
Timestamp time.Time `json:"@timestamp" yaml:"timestamp"`
Sequence uint32 `json:"sequence" yaml:"sequence"`
Expand All @@ -54,6 +61,10 @@ type Event struct {
Data map[string]string `json:"data,omitempty" yaml:"data,omitempty"`
Paths []map[string]string `json:"paths,omitempty" yaml:"paths,omitempty"`

ECS struct {
Event ECSEvent `json:"event" yaml:"event"`
} `json:"ecs" yaml:"ecs"`

Warnings []error `json:"-" yaml:"-"`
}

Expand Down Expand Up @@ -427,10 +438,17 @@ func addFieldsToEventData(msg *auparse.AuditMessage, event *Event) {
func applyNormalization(event *Event) {
setHowDefaults(event)

var syscallNorm *Normalization
if syscall, ok := event.Data["syscall"]; ok {
syscallNorm, ok = syscallNorms[syscall]
if !ok {
syscallNorm = syscallNorms["*"] // get the default to add some basic categorization
}
}

var norm *Normalization
if event.Type == auparse.AUDIT_SYSCALL {
syscall := event.Data["syscall"]
norm = syscallNorms[syscall]
norm = syscallNorm
} else {
norms := recordTypeNorms[event.Type.String()]
switch len(norms) {
Expand All @@ -454,6 +472,23 @@ func applyNormalization(event *Event) {
return
}

event.ECS.Event.Category = norm.ECS.Category.Values
event.ECS.Event.Type = norm.ECS.Type.Values

// we check to see if the non-AUDIT_SYSCALL event has an associated syscall
// from another part of the auditd message log, if it does and we have normalizations
// for that syscall, we merge the ECS categorization and type information so that
// the event has both enrichment for the record type itself and for the syscall it
// captures
hasAdditionalNormalization := syscallNorm != nil && syscallNorm != norm
if hasAdditionalNormalization {
andrewstucki marked this conversation as resolved.
Show resolved Hide resolved
event.ECS.Event.Category = append(event.ECS.Event.Category, syscallNorm.ECS.Category.Values...)
event.ECS.Event.Type = append(event.ECS.Event.Type, syscallNorm.ECS.Type.Values...)
if event.Result == "fail" {
event.ECS.Event.Outcome = "failure"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're only explicit if something was a failure otherwise, unknown/success are implied based off of the context of the category/type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non blocking

because "failure" is one of 3 constant values, it might be nice to use a constant for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally this would also be in an enum in the ECS generated go structures

}
}

event.Summary.Action = norm.Action

switch norm.Object.What {
Expand Down
Loading