Skip to content

Commit

Permalink
[Filebeat][AWS] Fix vpcflow pipeline exception: Cannot invoke "Object…
Browse files Browse the repository at this point in the history
….getClass()" because "receiver" is null (#24167)

The pipeline failed with an obscure error.message:
`Cannot invoke "Object.getClass()" because "receiver" is null`
when the ingested message didn't contain `aws.vpcflow.*` fiels.

Filebeat generates documents that lack those fields when parsing
lines from a .log file (not .json) that doesn't conform to the expected
formats. This happened for empty lines in particular.

(cherry picked from commit f4b7a25)
  • Loading branch information
adriansr committed Feb 22, 2021
1 parent 53e596d commit 0e16c53
Show file tree
Hide file tree
Showing 4 changed files with 23 additions and 5 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix Logstash module handling of logstash.log.log_event.action field. {issue}20709[20709]
- aws/s3access dataset was populating event.duration using the wrong unit. {pull}23920[23920]
- Zoom module pipeline failed to ingest some chat_channel events. {pull}23904[23904]
- Fix aws/vpcflow generating errors for empty logs or unidentified formats. {pull}24167[24167]

*Heartbeat*

Expand Down
8 changes: 3 additions & 5 deletions x-pack/filebeat/module/aws/vpcflow/ingest/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ processors:
ignore_empty_value: true

- set:
if: "ctx.aws.vpcflow.instance_id != '-'"
if: "ctx.aws?.vpcflow?.instance_id != null && ctx.aws.vpcflow.instance_id != '-'"
field: cloud.instance.id
value: "{{aws.vpcflow.instance_id}}"
ignore_empty_value: true
Expand All @@ -131,11 +131,9 @@ processors:
- script:
lang: painless
ignore_failure: true
if: "ctx.aws?.vpcflow?.tcp_flags != null"
source: |
if (ctx?.aws?.vpcflow?.tcp_flags == null)
return;
if (ctx?.aws?.vpcflow?.tcp_flags_array == null) {
if (ctx.aws.vpcflow.tcp_flags_array == null) {
ArrayList al = new ArrayList();
ctx.aws.vpcflow.put("tcp_flags_array", al);
}
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/module/aws/vpcflow/test/bad.log
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Phony unsupported log format.
18 changes: 18 additions & 0 deletions x-pack/filebeat/module/aws/vpcflow/test/bad.log-expected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[
{
"cloud.provider": "aws",
"event.category": "network_traffic",
"event.dataset": "aws.vpcflow",
"event.kind": "event",
"event.module": "aws",
"event.original": "Phony unsupported log format.",
"event.type": "flow",
"fileset.name": "vpcflow",
"input.type": "log",
"log.offset": 0,
"service.type": "aws",
"tags": [
"forwarded"
]
}
]

0 comments on commit 0e16c53

Please sign in to comment.