Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Source IP "LOCAL" or "Unknown" in WInlogbeat Security Ingest Pipeline #34251

Closed
wants to merge 90 commits into from
Closed

Fix Source IP "LOCAL" or "Unknown" in WInlogbeat Security Ingest Pipeline #34251

wants to merge 90 commits into from

Conversation

MakoWish
Copy link
Contributor

What does this PR do?

Some security events contain a source IP address of "LOCAL" or "Unknown" which are not valid IP addresses. This PR will correct the processing of events containing one of those values.

Why is it important?

This bug causes mapping exceptions and prevents these events from being ingested.

Checklist

  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

-fixes #19627

elasticmachine and others added 30 commits November 16, 2022 09:15
@elasticmachine
docs: update docs
c2b1e78
@elasticmachine
[Release] update test environment
5b957a1
@elasticmachine
fix: make update changes
565d0af
@mergify @andrewkroh
[Filebeat] Add parse_aws_vpc_flow_log processor (#33656) (#33705)
fc2f198 
This is a processor for parsing AWS VPC flow logs. It requires a user specified log format. It can populate the original flow log fields, ECS fields, or both.

Usage:

```yaml
processors:
  - parse_aws_vpc_flow_log:
      format: version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
  - community_id: ~
```

Benchmark:

```
goos: darwin
goarch: arm64
pkg: github.com/elastic/beats/v7/x-pack/filebeat/processors/aws_vpcflow
BenchmarkProcessorRun/original-mode-v5-message-10                2810948              2138 ns/op            2836 B/op         31 allocs/op
BenchmarkProcessorRun/ecs-mode-v5-message-10                     1914754              3107 ns/op            1908 B/op         41 allocs/op
BenchmarkProcessorRun/ecs_and_original-mode-v5-message-10        1693279              3538 ns/op            3076 B/op         41 allocs/op
```

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
(cherry picked from commit 1a86e42)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@mergify @rdner
Fix error spam on AKS (#33697) (#33703)
75e80d7 
This was happening due to the error level logging when the log path
matcher detected a `log.file.path` that does not start with a standard
Docker container log folder `/var/lib/docker/containers` because AKS
dropped support for Docker in September 2022 and switched to containerd.

It looks like this message was not supposed to be on the error level
in the first place since it just means that the matcher didn't
match and it's not an error. But it was mistakenly promoted from the
debug level in #16866 most likely
because the message started with `Error` and looked confusing.

This a partial fix to unblock our customers, but we still need to come
up with the full AKS/containerd support in a follow up change.

(cherry picked from commit 29f0b4c)

Co-authored-by: Denis <denis.rechkunov@elastic.co>
@mergify @andrewkroh
aws.vpcflow - use parse_aws_vpc_flow_log processor (#33699) (#33710)
1f1ccd8 
Update the aws.vpcflow dataset in the AWS module to use the parse_aws_vpc_flow_log processor.
I also updated the module to be aligned with ECS. It was not using valid event.category
and event.type values. It will now produce `event.category: [network]`. And `event.type: [connection]`
or `[connection, allowed]` or `[connection, denied]`.

(cherry picked from commit 703d529)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@mergify @andrewvc
[Heartbeat] Fix zip monitors (#33723) (#33739)
38c0ce1 
* [Heartbeat] Fix zip monitors by changing the check group after summary docs

Fixes #33698

* Add null chuck

* More robust check groups

* Add nil check

* Make linter happy

* Remove accidental config

* Fix heartbeat.yml

* Update x-pack/heartbeat/monitors/browser/synthexec/enrich.go

Co-authored-by: Emilio Alvarez Piñeiro <95703246+emilioalvap@users.noreply.github.com>

* Add changelog

Co-authored-by: Emilio Alvarez Piñeiro <95703246+emilioalvap@users.noreply.github.com>
(cherry picked from commit 453dfcb)

Co-authored-by: Andrew Cholakian <andrewvc@elastic.co>
@mergify @dedemorton
Fix copy/paste error in add_observer_metadata processor (#33729) (#33745
40fb601 
)

Fixes a copy/paste (or possibly renaming) mistake.

(cherry picked from commit ef74a19)

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
@mergify @dedemorton
Fix copy/paste errors (#33721) (#33747)
8b6853c 
Looks like the content here was copy/pasted and not updated correctly.

(cherry picked from commit 4f106cb)

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
@mergify @fearful-symmetry
update elastic-agent-client (#33753) (#33768)
4b6826d 
(cherry picked from commit 01430a4)

Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
@mergify @fearful-symmetry
Fix V2 loggers nil deref (#33770) (#33773)
dc09b2a 
* update elastic-agent-client

* fix nil pointer in V2 log statements

* change log lines

(cherry picked from commit 6fbbf24)

Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
@mergify @ShourieG
[filebeat][httpjson] - Added bug-fix for duplicate data issue #33213 (#…
bfb9df3 
…33664) (#33767)

* added bugfix for duplicate data issue #33213

* updated with PR suggetions

* updated comments

* re-engineered bugfix to update cursors properly

* spelling fix

(cherry picked from commit afb2beb)

Co-authored-by: ShourieG <105607378+ShourieG@users.noreply.github.com>
@mergify @dedemorton
[DOCS] Fix typo in add_nomad_metadata processor (#33730) (#33795)
73fbf54 
* [DOCS] Fix typo in add_nomad_metadata processor

* Fix a couple more typos

(cherry picked from commit a106ad2)

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
@mergify @efd6
x-pack/filebeat/input/http_endpoint: canonicalise tls consistency err…
54f98fd 
…or (#33720) (#33780)

Configs may be run in arbitrary order during tests, so make sure that
the error is invariant with respect to the address and reason. This has
no impact on the logic of the input in production.

(cherry picked from commit 4f9aadc)

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
@mergify @v1v
ci: change slack channel (#33796) (#33810)
ea83f0a 
(cherry picked from commit a1a6bd8)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
@mergify @rdner
Update test versions of NGINX, Apache and MongoDB (#33789) (#33845)
26860af 
The old tests were based on Debian Jessie released in April
2015 (end-of-life on June 30, 2020).

We started having problems with expired GPG keys when installing
dependencies in our Docker files.

Also:
* Add the missing MPM module and the `ServerName` directive for Apache.
* Add retries to Logstash tests because they are flaky
* Do not install mage v1.14.0
* Update elastic-agent-client version to avoid broken agent dependency

(cherry picked from commit 4f639d7)

Co-authored-by: Denis <denis.rechkunov@elastic.co>
@mergify @efd6
filebeat/input/{tcp,udp}: don't panic on nil addresses (#33837) (#33851)
ff00881 
The callback may be called with a nil RemoteAddr in the metadata
parameter, so ensure that this is not nil before attempting to deference
it.

(cherry picked from commit 105fbda)

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
@mergify @elasticmachine
[Release] update test environment (#33693) (#33855)
50b3ffc 
(cherry picked from commit d953786)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@mergify @blakerouse
Register the add_formatted_index as a usable processor. (#33800) (#33846
b9a0c53 
)

* Register the add_formatted_index as a usage processor for the processors list.

* Re-add removed file.

* Add changelog entry.

* Add missing return in String().

* Change implementation to use a boolean.

(cherry picked from commit 4c4eecf)

Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
@apmmachine
[Automation] Update elastic stack version to 8.6.0-ecfefdaf for testi…
9d25efa 
…ng (#33872)

Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co>
@mergify
handle EOF on single line content (#33568) (#33877)
0654524 
* handle EOF on single line content

* changelog

* fallback to encode_eof if no events in aws-s3 input

* lint

* lint

* collect on EOF in line reader

* remove encode eof

* remove iterN

* fix test

* increase test coverage

* linting

* more linting

* increase coverage

(cherry picked from commit 7b45320)

Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
@mergify @v1v
actions: use go-version-file: .go-version (#33879) (#33888)
2a09bdd 
see actions/setup-go#295

(cherry picked from commit 0105608)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
@mergify @kunsonx @rdner
libbeat: add processors "decode_duration", "move_fields" (#31301) (#3…
30233e8 
…3700)

Co-authored-by: hxms <hxms@live.cn>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
(cherry picked from commit dc1a0ca)

Co-authored-by: 冰天雪地 <hxmswl@gmail.com>
Co-authored-by: Denis <denis.rechkunov@elastic.co>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@mergify @kunsonx
feat : update change logs for new processor "decode_duration", "move_…
b6f42c8 
…fields" (#33714) (#33787)

Co-authored-by: hxms <hxms@live.cn>
Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
(cherry picked from commit 26f5d5c)

Co-authored-by: 冰天雪地 <hxmswl@gmail.com>
@mergify @andrewkroh
[libbeat] Allow clock_nanosleep in the default seccomp profiles (#3…
9fd0193 
…3831) (#33869)

* Add clock_nanosleep to seccomp allowlist

Fixes #33792

* Update github.com/elastic/go-seccomp-bpf to v1.3.0

It added support for Linux v6.0 syscall names.

(cherry picked from commit 141ad33)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@mergify @dedemorton
[DOCS] Edit replace processor docs and publish them (#33775) (#33904)
e909d74 
* Edit replace processor docs and publish them

* Fix heading levels

* Apply change from rdner

Co-authored-by: Denis <denis@rdner.de>

Co-authored-by: Denis <denis@rdner.de>
(cherry picked from commit 9d87b62)

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
@mergify @efd6
filebeat/docs: link cel input documentation into inputs page (#33848) (…
1777bba 
…#33909)

(cherry picked from commit 0bf6169)

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
@apmmachine
[Automation] Update elastic stack version to 8.6.0-87c26073 for testi…
be3a9f0 
…ng (#33911)

Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co>
@mergify @rdner
Add support for excluded paths in packaging (#33894) (#33902)
9bbbb6c 
* Add support for excluded paths in packaging

After an update in heartbeat we have have a path like
`usr/share/heartbeat/.node/node/lib/node_modules/@elastic/synthetics/templates/lightweight/heartbeat.yml`
which matches the `configFilePattern` which causes a permission check
which the file does not satisfy. It has nothing to do with the actual
heartbeat configuration file.

* Fix linting issues

* Fix one more linting issue

(cherry picked from commit 3198a68)

Co-authored-by: Denis <denis.rechkunov@elastic.co>
@mergify @rdner
Enforce "yes" when install debian packages in packaging (#33895) (#33899
b1bee18 
)

Older versions of Debian have expired GPG keys. In this case `-y` is
not enough and `apt-get` requires an additional `--force-yes`.

(cherry picked from commit 6c0dc97)

Co-authored-by: Denis <denis.rechkunov@elastic.co>
@MakoWish MakoWish requested review from a team as code owners January 12, 2023 23:04
@MakoWish MakoWish requested review from belimawr, rdner, gsantoro and devamanv and removed request for a team January 12, 2023 23:04
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Automation Label for the Observability productivity team labels Jan 12, 2023
@MakoWish MakoWish requested review from michalpristas and blakerouse and removed request for a team January 12, 2023 23:04
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 12, 2023
@mergify
Copy link
Contributor

mergify bot commented Jan 12, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 8.6 upstream/8.6
git merge upstream/main
git push upstream 8.6

@mergify
Copy link
Contributor

mergify bot commented Jan 12, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @MakoWish? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@MakoWish MakoWish closed this Jan 12, 2023
@MakoWish MakoWish reopened this Jan 12, 2023
@mergify
Copy link
Contributor

mergify bot commented Jan 12, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 8.6 upstream/8.6
git merge upstream/main
git push upstream 8.6

@mergify
Copy link
Contributor

mergify bot commented Jan 12, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @MakoWish? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@MakoWish MakoWish mentioned this pull request Jan 12, 2023
Closed
@MakoWish MakoWish changed the title 8.6 Fix Source IP "LOCAL" or "Unknown" in WInlogbeat Security Ingest Pipeline Jan 12, 2023
@MakoWish MakoWish closed this Jan 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@belimawr belimawr Awaiting requested review from belimawr belimawr is a code owner automatically assigned from elastic/elastic-agent-data-plane

@rdner rdner Awaiting requested review from rdner rdner is a code owner automatically assigned from elastic/elastic-agent-data-plane

@gsantoro gsantoro Awaiting requested review from gsantoro gsantoro is a code owner automatically assigned from elastic/obs-cloudnative-monitoring

@devamanv devamanv Awaiting requested review from devamanv devamanv is a code owner automatically assigned from elastic/obs-cloudnative-monitoring

@michalpristas michalpristas Awaiting requested review from michalpristas michalpristas is a code owner automatically assigned from elastic/elastic-agent-control-plane

@blakerouse blakerouse Awaiting requested review from blakerouse blakerouse is a code owner automatically assigned from elastic/elastic-agent-control-plane

Assignees

@MakoWish MakoWish

Labels
Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Winlogbeat Could not index event - "source.ip: 'LOCAL'" Not an IP String Literal
6 participants
@MakoWish @elasticmachine @apmmachine @amannocci @cmacknz @AndersonQ