Add support to configure monitoring output #3517

urso · 2017-02-02T13:32:44Z

Add support to configure xpack.monitoring.elasticsearch. If xpack.monitoring.elasticsearch is missing, configuration from output.elasticsearch will be used. If both are present, monitoring.elasticsearch re-uses missing options from outputs.elasticsearch. E.g. this way timeouts and user/password can be configured in monitoring.elasticsearch, while re-using hosts and other connection settings.
send and publisher loop disconnected from event publisher to publish metric snapshots collected via libbeat/monitoring
Introduce common.BeatInfo structure being initialized on startup with common beat meta-data to be used by outputs and monitoring reporter:
- beat (e.g. filebeat, metricbeat, ...)
- version
- name (configure name, otherwise hostname)
- hostname
- uuid

will do doc update in followup PR, to not block review for this PR for too long.

Note:
would be nice to have beat.Info or beat.Meta instead of
publisher.BeatInfo, but due to potential circular imports I have had to put
the common structure into the publisher package (already contains shipper
config/meta).

ruflin · 2017-02-14T08:21:02Z

libbeat/beat/beat.go

-	Name      string              // Beat name.
-	Version   string              // Beat version number. Defaults to the libbeat version when an implementation does not set a version.
-	UUID      uuid.UUID           // ID assigned to a Beat instance.
+	Info      publisher.BeatInfo  // beat metadata.


I can also make use of this in #3527 Perhaps we should put it into a separate package to prevent circular imports.

moved to common.BeatInfo

ruflin · 2017-02-14T08:24:12Z

libbeat/monitoring/monitoring.go

@@ -4,6 +4,7 @@ import "errors"

 type Mode uint8

+//go:generate stringer -type=Mode


We should make sure that make update also executes the generate commands.

oh, it's a leftover from debugging. There are 2 problems with stringer though:

you either need to have it installed or we have to vendor it and use go run

package loading in stringer requires you to have done go install on dependent packages (unless you have build a binary already). Otherwise go generate will fail (See this ticket).

Conclusion on this is that we should not add it to make update, correct?

yeah, I wouldn't add it to make update. I have had the build broken when just doing unit tests without having build the full project before go generate. Maybe introduce separate make go-generate, but this one might fail and developer has to 'fix' it locally. generated files should be committed.

ruflin · 2017-02-14T08:33:23Z

libbeat/monitoring/report/report.go

+	Reporter common.ConfigNamespace `config:",inline"`
+}
+
+type Reporter interface {


Seems like we start to have this Runner interface in lots of different places. We should probably start to use a shared interface.

nice thing about interfaces is, if they are assignable (same definition or sub-type relation), they can be used interchangeable. I'm aware of us having a mix of this interface.

Besides similar definition, giving it a name fitting the packages usage is sometimes nice to have. Plus, the Reporter interface might change. I'm not too much a fan of the 'Start(), Stop()` interfaces, as this clearly mandates/leaks usage of go-routines and very often channels for communication... I think in 'Reporter' we can get rid of 'Start' :)

just realized I don't need Start() ;)

ruflin · 2017-02-14T08:35:51Z

libbeat/publisher/publish.go

@@ -87,6 +87,14 @@ type BeatPublisher struct {
 	numClients uint32
 }

+type BeatInfo struct {


Should we put it under common ?

I'd like it to be beat.Info, not common.BeatInfo. With beats being kind of a framework, it would be nice if the beat (or libbeat) namespace would be the only packages libbeat and the actual beats would meat. Like: beat.Client being the instance used to publish a beat.Event. Only problem is initialization and circular dependencies due to having the main in beat package as well.

Will move it to common for now. Seems to be the place to prevent potential circular dependencies for now.

beat.Info would be nice, but lets keep such a more "radical" move for later.

ruflin · 2017-02-14T15:47:03Z

@urso Seems like you broke the beat generator. Also there you have to pass BeatInfo.

monicasarbu · 2017-02-15T13:24:45Z

libbeat/beat/beat.go

+		Info: common.BeatInfo{
+			Beat:     name,
+			Version:  version,
+			Name:     hostname,


I think here you want Name: name, that it's equal to the name configuration option.

name is the name configured via config file and defaults to hostname. beat.Beat is the beat name like filebeat, packetbeat. We set Name later after having parsed the config file.

monicasarbu · 2017-02-15T13:56:50Z

libbeat/monitoring/report/elasticsearch/elasticsearch.go

+
+		event := outputs.Data{Event: common.MapStr{
+			"timestamp": common.Time(ts),
+			"beat":      r.beatMeta,


Instead of exporting beat.beat, wouldn't be nicer beat.type?

monicasarbu · 2017-02-15T13:59:48Z

libbeat/monitoring/report/elasticsearch/elasticsearch.go

+	}, nil
+}
+
+func parseProxyURL(raw string) (*url.URL, error) {


maybe we should have this in common to be used later by others?

added todo item

monicasarbu · 2017-02-15T14:02:28Z

libbeat/outputs/elasticsearch/url.go

@@ -114,3 +114,18 @@ func makePath(index string, docType string, id string) (string, error) {
 	}
 	return path, nil
 }
+
+func parseProxyURL(raw string) (*url.URL, error) {


multiple times defined?

added todo item. The HTTP connection/configurations should be split and generalized for all beats. e.g. heartbeat and metricbeat also have http clients with similar settings.

- Add support to configure `monitoring.elasticsearch`. If `monitoring.elasticsearch` is missing, configuration from `output.elasticsearch` will be used. If both are present, `monitoring.elasticsearch` re-uses missing options from `outputs.elasticsearch`. E.g. this way timeouts and user/password can be configured in `monitoring.elasticsearch`, while re-using `hosts` and other connection settings. - checks if xpack.monitoring.elasticsearch.hosts not configured if output.elasticsearch.hosts is configured - send and publisher loop disconnected from event publisher to publish metric snapshots collected via `libbeat/monitoring` - Introduce common.BeatInfo structure being initialized on startup with common beat meta-data to be used by outputs and monitoring reporter: - beat (e.g. filebeat, metricbeat, ...) - version - name (configure name, otherwise hostname) - hostname - uuid

urso added the in progress Pull request is currently in progress. label Feb 2, 2017

urso force-pushed the monitoring/report branch 2 times, most recently from 0455316 to 9c25940 Compare February 13, 2017 01:54

urso changed the title ~~[WIP] Add support to configure monitoring output~~ Add support to configure monitoring output Feb 13, 2017

urso added review in progress Pull request is currently in progress. and removed in progress Pull request is currently in progress. review labels Feb 13, 2017

ruflin reviewed Feb 14, 2017

View reviewed changes

urso force-pushed the monitoring/report branch from ee49df7 to 2d90593 Compare February 14, 2017 13:13

monicasarbu reviewed Feb 15, 2017

View reviewed changes

monicasarbu approved these changes Feb 15, 2017

View reviewed changes

urso force-pushed the monitoring/report branch from d68da22 to 57f87b6 Compare February 16, 2017 16:14

ruflin merged commit b2452cf into elastic:master Feb 16, 2017

urso deleted the monitoring/report branch February 19, 2019 18:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to configure monitoring output #3517

Add support to configure monitoring output #3517

urso commented Feb 2, 2017 •

edited by monicasarbu

Loading

ruflin Feb 14, 2017

urso Feb 14, 2017

ruflin Feb 14, 2017

urso Feb 14, 2017

ruflin Feb 14, 2017

urso Feb 14, 2017

ruflin Feb 14, 2017

urso Feb 14, 2017

urso Feb 14, 2017

ruflin Feb 14, 2017

urso Feb 14, 2017

urso Feb 14, 2017

urso Feb 14, 2017

ruflin Feb 14, 2017

ruflin commented Feb 14, 2017

monicasarbu Feb 15, 2017

urso Feb 15, 2017

monicasarbu Feb 15, 2017

urso Feb 15, 2017

monicasarbu Feb 15, 2017

urso Feb 15, 2017

monicasarbu Feb 15, 2017

urso Feb 15, 2017

		@@ -4,6 +4,7 @@ import "errors"

		type Mode uint8

		//go:generate stringer -type=Mode

Add support to configure monitoring output #3517

Add support to configure monitoring output #3517

Conversation

urso commented Feb 2, 2017 • edited by monicasarbu Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Feb 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso commented Feb 2, 2017 •

edited by monicasarbu

Loading