Add host metadata processor #5968

ruflin · 2017-12-29T04:30:22Z

This adds a processor to add some information about the host machine to each event very similar to what we have for the cloud metadata processor. The idea is to make it possible to allow better filtering based on additional information about the host if needed. This information partially overlaps with what we have in beat.hostname but adds additional fields. Other fields like the IP address(es) could be also added here.

exekias

+1 to this, would it make sense to send this info always by default instead of adding a processor?

exekias · 2018-01-02T14:11:37Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+
+func (p addHostMetadata) Run(event *beat.Event) (*beat.Event, error) {
+
+	info, err := host.Info()


Do you have any benchmark on this? it would make sense to move it to the constructor and cache the result, we just need to document that a hostname change won't get reflected here

I changed it to be instantiated in the constructor. I'm pretty sure later a request for expiring the info is coming up :-)

andrewkroh · 2018-01-02T16:44:55Z

I was kind of working on something related over the break. It can (or will be able to) fetch some of the information that you want to add with this processor. I am working on this because I wanted to log (#5946) some of this information and also make this data available for xpack monitoring.

https://github.com/andrewkroh/go-sysinfo

tsg · 2018-01-02T19:42:20Z

Big +1 to this. I think a processor makes sense to mirror add_cloud_metadata and co, but we can put it into the default configuration file.

ruflin · 2018-01-03T11:32:20Z

@andrewkroh Very nice. Did not look into your code yet. Did you use a third party lib or implement it yourself?

monicasarbu · 2018-01-05T08:30:27Z

+1 on adding add_host_metadata, that will also add cloud metadata in the future.

ruflin · 2018-02-26T07:11:32Z

This will close #3480

ruflin · 2018-02-27T05:56:05Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+			"hostname": p.info.Hostname,
+			"id":       p.info.UniqueID,
+			"os": common.MapStr{
+				"architecture": p.info.Architecture,


@andrewkroh Seeing that you have Architecture outside of info I have second thoughts on puttig it under os as I agree it more belongs to the host then the os itself.

Also thinking about putting os info on the top level.

ruflin · 2018-02-27T06:04:09Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+package add_host_metadata
+
+import (
+	sysinfo "github.com/elastic/go-sysinfo"


@andrewkroh Did you have to add a - to the lib name? :-D

With Go you don't need to add a package alias, just import plain old "github.com/elastic/go-sysinfo" and use sysinfo.XYZ in the code.

It was mainly my IDE that complained :-)

houndci-bot · 2018-02-27T06:26:33Z

libbeat/processors/add_host_metadata/add_host_metadata_test.go

@@ -0,0 +1,26 @@
+package add_host_metadata


don't use an underscore in package name

andrewkroh · 2018-03-05T01:34:33Z

I'm glad you are trying out go-sysinfo. I'll take a closer look at this next week. I've been sitting on a bunch of updates to go-sysinfo, that I haven't had time to cleanup. I plan on getting the lib into a better place so that this can make 6.3.

ruflin · 2018-03-06T06:13:28Z

@andrewkroh Good timing. I plan to put some work into this next week again. But also plan to start with a very basic version first which we then can expand on. The current fields are probably already enough for a first version.

andrewkroh

This looks like it's just about ready. 👍

andrewkroh · 2018-03-07T23:24:32Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+}
+
+func (p addHostMetadata) String() string {
+	return "add_host_metadata="


How about add_host_metadata=[] to indicate an empty config?

andrewkroh · 2018-03-07T23:29:31Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+		return nil, err
+	}
+	return &addHostMetadata{
+		info: h.Info(),


I think we will want to refresh this data every couple of minutes (image that the user installs an OS update). And given that we are caching this data we might was well cache the common.MapStr containing it.

I'd probably add a time.Time value to track when the value was collected. Then refresh the value from Run after it becomes stale.

I cached the mapstr and set the cache expiry to 5 minutes. We can still make it a config later if needed.

andrewkroh · 2018-03-07T23:32:47Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+			"os": common.MapStr{
+				"platform": p.info.OS.Platform,
+				"version":  p.info.OS.Version,
+				"family":   p.info.OS.Family,


Did you want codename and build?

I added it.

andrewkroh · 2018-03-07T23:35:22Z

libbeat/processors/add_host_metadata/add_host_metadata_test.go

+		Timestamp: time.Now(),
+	}
+	p, err := newHostMetadataProcessor(nil)
+	assert.NoError(t, err)


This should expect to receive sysinfo/types.ErrNotImplemented for any OS other than windows, linux, and darwin.

andrewkroh · 2018-03-07T23:37:45Z

libbeat/processors/add_host_metadata/add_host_metadata.go

+	data := common.MapStr{
+		"host": common.MapStr{
+			"hostname":     p.info.Hostname,
+			"id":           p.info.UniqueID,


info.UniqueID is optional (as signified by ,omitempty in the json tag). You should add handling for this. Older versions of Linux don't have /etc/machine-id.

houndci-bot · 2018-03-13T11:59:47Z

libbeat/processors/add_host_metadata/add_host_metadata.go

@@ -0,0 +1,80 @@
+package add_host_metadata


don't use an underscore in package name

andrewkroh · 2018-03-13T14:29:08Z

I think we no longer need to keep github.com/elastic/procfs in vendor/. I'm pretty sure any changes got pushed upstream. Can you try removing references to github.com/elastic/procfs and replace them with github.com/prometheus/procfs so we don't have a duplicate dep under two names.

ruflin · 2018-03-14T09:59:56Z

github.com/elastic/procfs is still used in the system/socket and system/raid metricset. I suggest to make this change in a follow up PR and not mix it with this one.

ruflin · 2018-03-14T13:29:54Z

Here is a separate PR that changes the dependency. We can merge the other one first and rebase on top of it: #6548

This adds a processor to add some information about the host machine to each event very similar to what we have for the cloud metadata processor. The idea is to make it possible to allow better filtering based on additional information about the host if needed. This information partially overlaps with what we have in `beat.hostname` but adds additional fields. Other fields like the IP address(es) could be also added here.

ruflin · 2018-03-15T12:34:33Z

This PR currently fails because of a crosscompile issue with darwin 32 bit: elastic/go-sysinfo#3

andrewkroh · 2018-03-16T02:33:23Z

I believe elastic/go-sysinfo#4 should fix the cross-compile issue seen here. The sysinfo code was lacking cgo tags to prevent the darwin provider from being included when cross-compiling on Linux (without a C x-compiler).

TimWardOrigami · 2018-03-28T10:44:57Z

Is the host's FQDN included in the output (eg "hostname -f" output)? If not should/can it be?

ruflin · 2018-04-03T12:05:16Z

So far the code use to create the hostname is os.Hostname() which is the same as under beat.hostname. Now that we have a processor for the data we should make it a potential config option for it to be the FQDN if it not already is. I assume the fetch the FQDN we will need to do some lookups. @TimWardOrigami Want to add a feature request for this?

@andrewkroh WDYT on where this belongs? Should we have this in go-sysinfo?

TimWardOrigami · 2018-04-03T12:15:19Z

@ruflin How do I "add a feature request" please?

ruflin · 2018-04-03T12:56:03Z

@TimWardOrigami Just open a new Github issue in this repo, explain quickly on what it is a about and what it is needed for. I will label it then accordingly.

TimWardOrigami · 2018-04-03T14:23:47Z

#6741 Tim Ward From: Nicolas Ruflin <notifications@github.com> Sent: 03 April 2018 13:56 To: elastic/beats <beats@noreply.github.com> Cc: Tim Ward <tim.ward@origamienergy.com>; Mention <mention@noreply.github.com> Subject: Re: [elastic/beats] Add host metadata processor (#5968) @TimWardOrigami<https://github.com/TimWardOrigami> Just open a new Github issue in this repo, explain quickly on what it is a about and what it is needed for. I will label it then accordingly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5968 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AkHS-OuiKQPjRRIHDG_5JmkH3flZSpIRks5tk3FvgaJpZM4RO2cH>. The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i) do not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact the sender immediately; and (iii) delete this email. Origami Energy Limited (company number 8619644); Origami Storage Limited (company number 10436515) and OSSPV001 Limited (company number 10933403), each registered in England and each with a registered office at: Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.

ruflin added :Processors discuss Issue needs further discussion. libbeat labels Dec 29, 2017

exekias reviewed Jan 2, 2018

View reviewed changes

ruflin mentioned this pull request Feb 26, 2018

Feature: add add_host_metadata similar to add_cloud_metadata but for self hosted environments #3480

Closed

ruflin force-pushed the add_host_metadata branch from 42d3931 to 485145d Compare February 26, 2018 09:33

ruflin commented Feb 27, 2018

View reviewed changes

ruflin changed the title ~~[Discuss] Add host metadata processor~~ Add host metadata processor Feb 27, 2018

ruflin force-pushed the add_host_metadata branch from 82df182 to 9da8b6a Compare February 27, 2018 06:26

houndci-bot reviewed Feb 27, 2018

View reviewed changes

elastic deleted a comment from houndci-bot Mar 7, 2018

andrewkroh reviewed Mar 7, 2018

View reviewed changes

ruflin force-pushed the add_host_metadata branch from 9da8b6a to ec4e196 Compare March 13, 2018 11:59

houndci-bot reviewed Mar 13, 2018

View reviewed changes

ruflin force-pushed the add_host_metadata branch from b3a7396 to 7fbe357 Compare March 14, 2018 10:05

ruflin added review and removed discuss Issue needs further discussion. labels Mar 14, 2018

ruflin force-pushed the add_host_metadata branch from 7fbe357 to 08123ea Compare March 14, 2018 13:38

ruflin force-pushed the add_host_metadata branch from 08123ea to 107709c Compare March 14, 2018 13:51

ruflin mentioned this pull request Mar 15, 2018

Add crosscompiling tests elastic/go-sysinfo#3

Closed

add newer gosysinfo libs

52a8fc9

andrewkroh merged commit 1fbeb65 into elastic:master Mar 16, 2018

ruflin deleted the add_host_metadata branch April 3, 2018 11:59


		func (p addHostMetadata) Run(event beat.Event) (beat.Event, error) {

		info, err := host.Info()

Add host metadata processor #5968

Add host metadata processor #5968

Conversation

ruflin commented Dec 29, 2017 • edited Loading

exekias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Jan 2, 2018

tsg commented Jan 2, 2018

ruflin commented Jan 3, 2018

monicasarbu commented Jan 5, 2018

ruflin commented Feb 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Mar 5, 2018

ruflin commented Mar 6, 2018

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Mar 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Mar 13, 2018

ruflin commented Mar 14, 2018

ruflin commented Mar 14, 2018

ruflin commented Mar 15, 2018

andrewkroh commented Mar 16, 2018

TimWardOrigami commented Mar 28, 2018

ruflin commented Apr 3, 2018

TimWardOrigami commented Apr 3, 2018

ruflin commented Apr 3, 2018

TimWardOrigami commented Apr 3, 2018 via email

ruflin commented Dec 29, 2017 •

edited

Loading

andrewkroh Mar 7, 2018 •

edited

Loading