Rename `source` field in Filebeat #8902

ruflin · 2018-11-02T08:32:39Z

The source field in Filebeat was used to store the file path for logs or the source ip for syslog, udp, tcp input. As source is in ECS an object the fields are now moved to ECS pattern.

For UDP, TCP, syslog input the source field is converted to log.source.ip
For the log input the source field is converted to log.file.path

Done:

Test files updated
Changelog updated
Migration file updated
source removed from fields.yml, two new fields added

ruflin · 2018-11-02T08:33:22Z

filebeat/input/udp/input.go

@@ -74,7 +74,9 @@ func NewInput(
 			},
 			Fields: common.MapStr{
 				"message": string(data),
-				"source":  metadata.RemoteAddr.String(),
+				"source": common.MapStr{


@ph Does this make sense for udp, tcp and syslog?

ruflin · 2018-11-05T07:15:37Z

@ph @andrewkroh @webmat I actually have second thoughts here about the proposed fields. file.path and source.ip can both show up in the log files itself. What if the apache logs are sent through syslog and both have source.ip.

The file.path and source.ip are meta information about a log event. Because of this I propose log.file.path and log.source.ip. Both are not ECS fields but follow the naming patterns.

webmat

LGTM

I agree with nesting this information under log.. Without it, you're right that there's a conflict with this meta information about the source of the event, and the event content itself.

And I also like that you're keeping the full name of log.file.path. It opens up the possibility to fill up more of the file fields under there with log file metadata (e.g. log.file.owner & so on), if someone ever needs this.

webmat

Let me revert that back to "Comment" from "Approve". Of course the tests need to be fixed :-)

But I like the proposed field names as they are.

ph · 2018-11-05T18:26:19Z

@ruflin proposed changes LGTM

webmat · 2018-11-05T18:42:49Z

Since this is one field being split in two locations (file path vs IP), we can't list it in dev-tools/ecs-migration.yml, correct?

ruflin · 2018-11-05T20:56:31Z

@webmat I'll figure out a way to list it

andrewkroh

Thanks for making this change. Left only minor comments.

andrewkroh · 2018-11-06T12:34:43Z

CHANGELOG.asciidoc

@@ -20,6 +20,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
 - Use `initial_scan` action for new paths. {pull}7954[7954]

 *Filebeat*
+- Move source to log.file.path and log.source.ip


I'd use "Rename" rather than "Move".

andrewkroh · 2018-11-06T12:35:29Z

filebeat/_meta/fields.common.yml

+      type: ip
+      required: false
+      description: >
+        Source ip from which the log event was read / sent from.


adriansr

Two minor comments

adriansr · 2018-11-06T13:33:27Z

CHANGELOG.asciidoc

@@ -20,7 +20,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
 - Use `initial_scan` action for new paths. {pull}7954[7954]

 *Filebeat*
-


Nit: You've removed the empty line between the header (Filebeat) and the list of changes

Will fix this in a follow up PR

adriansr · 2018-11-06T13:43:31Z

x-pack/filebeat/module/suricata/eve/test/eve-alerts.log-expected.json

@@ -16,6 +16,7 @@
        "http.request.method": "GET", 
        "http.response.status_code": "200", 
        "input.type": "log", 
+        "log.file.path": "/Users/ruflin/Dev/gopath/src/github.com/elastic/beats/x-pack/filebeat/module/suricata/eve/test/eve-alerts.log", 
        "log.offset": 0, 


As this field is ignored by test_modules.py, can we get rid of this entries?

Good question. Will need to investigate. Will also tackle this in one of the other PR's that is conflicting with this one as soon as it's merged if that is ok with you?

You got lucky, have to rebase now anyways. Will fix these issues.

The source field in Filebeat was used to store the file path for logs or the source ip for syslog, udp, tcp input. As source is in ECS an object the fields are now moved to ECS pattern. * For UDP, TCP, syslog input the source field is converted to log.source.ip * For the log input the source field is converted to log.file.path Done: * Test files updated * Changelog updated * Migration file updated * `source` removed from fields.yml, two new fields added

ruflin · 2018-11-07T10:51:10Z

Ready for an other round of reviews. Failing tests should not be related.

adriansr

LGTM

Related to elastic#8902 but adding the fields instead of replacing

…9435) Related to #8902 but adding the fields instead of replacing

ruflin added in progress Pull request is currently in progress. Packetbeat Filebeat Filebeat Auditbeat ecs labels Nov 2, 2018

ruflin requested review from ph, webmat, andrewkroh and adriansr November 2, 2018 08:32

ruflin commented Nov 2, 2018

View reviewed changes

webmat approved these changes Nov 5, 2018

View reviewed changes

webmat reviewed Nov 5, 2018

View reviewed changes

ruflin removed Auditbeat Packetbeat labels Nov 5, 2018

ruflin changed the title ~~[WIP] Convert source field in Filebeat~~ Convert source field in Filebeat Nov 5, 2018

ruflin added review and removed in progress Pull request is currently in progress. labels Nov 5, 2018

ruflin force-pushed the convert-source-filebeat branch 2 times, most recently from 7885f48 to cb16cbf Compare November 6, 2018 07:25

ruflin mentioned this pull request Nov 6, 2018

Transition Beats to ECS #8655

Closed

ruflin force-pushed the convert-source-filebeat branch from cb16cbf to f7b8647 Compare November 6, 2018 08:46

andrewkroh reviewed Nov 6, 2018

View reviewed changes

ph approved these changes Nov 6, 2018

View reviewed changes

andrewkroh approved these changes Nov 6, 2018

View reviewed changes

adriansr approved these changes Nov 6, 2018

View reviewed changes

ruflin force-pushed the convert-source-filebeat branch from 13db3e9 to a1437a9 Compare November 6, 2018 15:18

ruflin changed the title ~~Convert source field in Filebeat~~ Rename source field in Filebeat Nov 6, 2018

ruflin force-pushed the convert-source-filebeat branch from a1437a9 to f1cdd06 Compare November 6, 2018 15:20

ruflin force-pushed the convert-source-filebeat branch from f1cdd06 to b659e10 Compare November 6, 2018 18:30

adriansr approved these changes Nov 7, 2018

View reviewed changes

ruflin merged commit 0e42349 into elastic:master Nov 7, 2018

ruflin deleted the convert-source-filebeat branch November 7, 2018 11:52

ruflin mentioned this pull request Dec 7, 2018

Introduce log.source.address and log.file.path for 7.x compatiblity #9435

Merged

jsoriano mentioned this pull request Dec 10, 2018

Parse also the port from log sources #9460

Closed

ruflin added a commit to ruflin/beats that referenced this pull request Dec 12, 2018

Introduce log.source.address and log.file.path for 7.x compatiblity

8750ba1

Related to elastic#8902 but adding the fields instead of replacing

ruflin added a commit that referenced this pull request Dec 12, 2018

Introduce log.source.address and log.file.path for 7.x compatiblity (#…

f81831f

…9435) Related to #8902 but adding the fields instead of replacing

axw mentioned this pull request Apr 1, 2019

add_docker_metadata's match_source does not work #11556

Closed

tjanson mentioned this pull request Aug 29, 2019

Beats type as prefix: Change default behavior to disable Graylog2/graylog2-server#6068

Open

michel-laterman mentioned this pull request Mar 1, 2023

Plz check the source fileld (filebeat 8.6.2) #34691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rename `source` field in Filebeat #8902

Rename `source` field in Filebeat #8902

ruflin commented Nov 2, 2018 •

edited

Loading

ruflin Nov 2, 2018

ruflin commented Nov 5, 2018

webmat left a comment

webmat left a comment

ph commented Nov 5, 2018

webmat commented Nov 5, 2018

ruflin commented Nov 5, 2018

andrewkroh left a comment

andrewkroh Nov 6, 2018

ruflin Nov 6, 2018

andrewkroh Nov 6, 2018

ruflin Nov 6, 2018

adriansr left a comment

adriansr Nov 6, 2018

ruflin Nov 6, 2018

adriansr Nov 6, 2018

ruflin Nov 6, 2018

ruflin Nov 6, 2018

ruflin commented Nov 7, 2018

adriansr left a comment

		@@ -20,7 +20,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
		- Use `initial_scan` action for new paths. {pull}7954[7954]

		Filebeat

Rename source field in Filebeat #8902

Rename source field in Filebeat #8902

Conversation

ruflin commented Nov 2, 2018 • edited Loading

Choose a reason for hiding this comment

ruflin commented Nov 5, 2018

webmat left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

ph commented Nov 5, 2018

webmat commented Nov 5, 2018

ruflin commented Nov 5, 2018

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Nov 7, 2018

adriansr left a comment

Choose a reason for hiding this comment

Rename `source` field in Filebeat #8902

Rename `source` field in Filebeat #8902

ruflin commented Nov 2, 2018 •

edited

Loading