Metricbeat: Support wildcards in jolokia (#6453) #6462

jsoriano · 2018-02-23T18:55:11Z

Partially support wildcards in jolokia. This would cover the case of #6453 where wildcard is used to match one of two possible names. In a more general case this implementation is incomplete because if multiple mbeans match, their field mapping will collide and only one value will be stored.

Update: The option finally chosen and implemented is:

When using wildcards, each mbean generates its own event (as in option 1 below).
Additionally, a new event parameter is added to field mappings, to explicitly separate metrics that have different values.

To solve this I see two options:

Generate an event for each matching mbean, so each event has a field with the name of the mbean. For example, current events are like this:

{
  "port": 8080,
  "max_connections": 200,
  "_namespace": "testnamespace"
}

New events would contain also the mbean, so with a wildcard these events could be generated:

{
  "port": 8080,
  "max_connections": 200,
  "mbean": "Catalina:name=\"http-bio-8080\",type=ThreadPool",
  "_namespace": "testnamespace"
}

{
  "port": 8009,
  "max_connections": 200,
  "mbean": "Catalina:name=\"ajp-bio-8009\",type=ThreadPool",
  "_namespace": "testnamespace"
}

Add a regular expression or similar to the attributes mapping that can be used to fill placeholders in the field name, e.g:

jmx.mappings:
       mbean: 'Catalina:name=*,type=ThreadPool'
       attributes:
                  attr: maxConnections
                  field: "${name}.max_connections"
                  mbeanFields: 'Catalina:name=(?P<name>[^,]),type=ThreadPool'

Would generate events like this one:

{
  "http-bio-8080": {
    "port": 8080,
    "max_connections": 200
  },
  "ajp-bio-8009": {
    "port": 8009,
    "max_connections": 200
  },
  "_namespace": "testnamespace"
}

I think option 1 is more intuitive, but I'm not sure if generating multiple events fits with the general design of the module, opinions?

ruflin · 2018-02-26T00:47:17Z

Could you share an example of two json events on how they would look like with option 1?

jsoriano · 2018-03-06T05:34:02Z

@ruflin I have added an example for option 1 to the description.

jsoriano · 2018-03-06T05:45:30Z

I guess that something like option 2 fits better with EventFetcher interface, so for a Fetch() an only event is generated.

jsoriano · 2018-03-07T01:40:10Z

After talking offline with @exekias about this I will go for option 1 implementing the EventsFetcher interface that allows to generate multiple events in an only call.

houndci-bot · 2018-03-08T01:33:45Z

metricbeat/module/jolokia/jmx/data.go


 	"github.com/joeshaw/multierror"
 	"github.com/pkg/errors"

 	"github.com/elastic/beats/libbeat/common"
 )

+const (
+	MBeanEventKey = "mbean"


exported const MBeanEventKey should have comment (or a comment on this block) or be unexported

jsoriano · 2018-03-08T01:45:26Z

After playing a bit with Jolokia and metricbeat I think that we cannot just generate an event for each mbean as I though. I see two main use cases for this module:

The module is used to obtain metrics from different mbeans in an only event, this was already supported and it makes sense that we continue supporting it. For this use case we will have metrics taken from different specific mbeans into the same event.
The module is used to obtain metrics from multiple unknown sources. For that wildcards are used and each matching mbean generates a different event with its own metrics. Support for wildcard in jolokia module in metricbeat #6453 would fit into this use case even if it only matches with one mbean and generates an only event. In this case an mbean field is added to the event so it can be used for filtering.

ruflin · 2018-03-12T07:20:09Z

For 1 event per mbean I'm wondering on how many metrics we expect per mbean?
The reason I prefer multiple mbeans in one event is that the user can always configure metricbeat to have multiple instances of the metricset and configure 1 mbean per event if he wants to (except for the wildcard).
Should we make it a config option if it should be 1 or multiple events?
For the name conflicts, can saftemapstr help here?

jsoriano · 2018-03-14T05:24:09Z

@ruflin thanks for your comments, find answers inline.

For 1 event per mbean I'm wondering on how many metrics we expect per mbean?

It will depend on the mappings the user configures, for example for the case mentioned in #6453 (comment) it'd be about 20 metrics per matching mbean, but it could be only 2 as in the example I posted.

The reason I prefer multiple mbeans in one event is that the user can always configure metricbeat to have multiple instances of the metricset and configure 1 mbean per event if he wants to (except for the wildcard).

The case of building an only event from multiple known mbeans would still be supported, as well as the case of generating multiple events for known mbeans using multiple instances of the module. Multiple events would only be generated when using wildcards.

Should we make it a config option if it should be 1 or multiple events?

I think this would be more confusing, in general if a wildcard is used, multiple matches of similar things should be expected, so as I see it, multiple events should be generated to avoid conflicts with field names, and to support the case of #6453 (comment) that I think it fits better with the use of wildcards.
If no wildcard is used, then the user is building an only event from one or multiple known mbeans (the case supported now).
There could be a case where the user wants to build an event from mixed known mbeans and a wildcard that is expected to match only once, this seems a tricky corner case, but it could be actually supported by allowing to force the generation of an only event no matter what.
What do you think, should we support this case?

For the name conflicts, can saftemapstr help here?

Well, conflicts here wouldn't come from having dots in the names but by having directly multiple fields with exactly the same name coming from similar mbeans.
On the other hand, names for the fields come from user configuration, so if it has conflicts by dots it could be avoided by defining proper names, we could notice that in the documentation. Or we could use safemapstr to prevent problems with this... Opinions?

When using wildcards with Jolokia, if multiple mbeans match, they will each one have their own metrics. If we only generate an event each match will overwrite the previous ones. Previous behaviour is kept for Jolokia requests without wildcards, this can be used to compose an only event from multiple specific mbeans.

houndci-bot · 2018-03-22T11:37:27Z

metricbeat/module/jolokia/jmx/config.go

+}
+type AttributeMapping map[attributeMappingKey]Attribute
+
+func (m AttributeMapping) Get(mbean, attr string) (Attribute, bool) {


exported method AttributeMapping.Get should have comment or be unexported

houndci-bot · 2018-03-22T11:37:27Z

metricbeat/module/jolokia/jmx/config.go

+type attributeMappingKey struct {
+	mbean, attr string
+}
+type AttributeMapping map[attributeMappingKey]Attribute


exported type AttributeMapping should have comment or be unexported

jsoriano · 2018-03-22T11:48:31Z

As talked offline, and in the line of #6585, I have added explicit grouping of metrics. An optional event parameter can be added to each attribute mapping so metrics with the same event are sent in the same event to Elastic.

…etrics A new optional `event` option is added to attribute mappings, so attributes with the same `event` are sent in the same event to Elastic and attributes with different `event` are sent in different events. Some additional related refactorings done to improve type safety.

jsoriano · 2018-03-23T18:10:42Z

jenkins, test it again please

ruflin · 2018-03-26T08:56:31Z

metricbeat/module/jolokia/jmx/config.go

+
+// Get the mapping options for the attribute of an mbean
+func (m AttributeMapping) Get(mbean, attr string) (Attribute, bool) {
+	a, found := m[attributeMappingKey{mbean, attr}]


Nit: Seems like you could return here directly.

I think it doesn't work when the expression is a value obtained from a map.

ruflin · 2018-03-26T08:58:58Z

@jsoriano Thanks for pushing this forward. Could you update the PR message above with the end solution you went with so people coming to this PR know directly what changed?

Would be great to have a blog post about jolokia/jmx monitoring.

jsoriano · 2018-03-26T18:35:20Z

Updated description

NeQuissimus · 2018-03-29T17:04:46Z

Is this going to make it into 6.3 or left for 7.0?

ruflin · 2018-03-30T10:17:00Z

@NeQuissimus Will be in 6.3

jsoriano added enhancement in progress Pull request is currently in progress. review Metricbeat Metricbeat discuss Issue needs further discussion. labels Feb 23, 2018

ruflin added the module label Feb 26, 2018

jsoriano force-pushed the jolokia-wildcard branch from 9ec8def to 21142da Compare March 8, 2018 01:33

houndci-bot reviewed Mar 8, 2018

View reviewed changes

jsoriano force-pushed the jolokia-wildcard branch from 21142da to a83c3fd Compare March 8, 2018 01:35

jsoriano force-pushed the jolokia-wildcard branch from 90020db to 439dae4 Compare March 8, 2018 02:28

jsoriano force-pushed the jolokia-wildcard branch from 439dae4 to 8768ad7 Compare March 14, 2018 04:48

This was referenced Mar 14, 2018

Support for wildcard in jolokia module in metricbeat #6453

Closed

[Metricbeat] Grouping Windows Perfmon Metrics in Events #6584

Closed

jsoriano added 3 commits March 22, 2018 10:38

Metricbeat: Support wildcards in jolokia (elastic#6453)

e5189a3

Metricbeat: Update documentation for Jolokia wildcards

efcb8ec

jsoriano force-pushed the jolokia-wildcard branch from 8768ad7 to 5c73b4c Compare March 22, 2018 11:37

houndci-bot reviewed Mar 22, 2018

View reviewed changes

jsoriano force-pushed the jolokia-wildcard branch from 5c73b4c to c66a46e Compare March 22, 2018 11:40

jsoriano force-pushed the jolokia-wildcard branch from c66a46e to f3bb1ca Compare March 22, 2018 12:03

jsoriano removed discuss Issue needs further discussion. in progress Pull request is currently in progress. labels Mar 23, 2018

ruflin approved these changes Mar 26, 2018

View reviewed changes

ruflin merged commit 7ac8e2f into elastic:master Mar 26, 2018

jsoriano added the needs_backport PR is waiting to be backported to other branches. label Apr 3, 2018

jsoriano mentioned this pull request Apr 3, 2018

Backport Jolokia wildcards support (#6462) to 6.2 #6738

Closed

jsoriano removed the needs_backport PR is waiting to be backported to other branches. label Apr 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat: Support wildcards in jolokia (#6453) #6462

Metricbeat: Support wildcards in jolokia (#6453) #6462

jsoriano commented Feb 23, 2018 •

edited

Loading

ruflin commented Feb 26, 2018

jsoriano commented Mar 6, 2018

jsoriano commented Mar 6, 2018 •

edited

Loading

jsoriano commented Mar 7, 2018

houndci-bot Mar 8, 2018

jsoriano commented Mar 8, 2018

ruflin commented Mar 12, 2018

jsoriano commented Mar 14, 2018

houndci-bot Mar 22, 2018

houndci-bot Mar 22, 2018

jsoriano commented Mar 22, 2018

jsoriano commented Mar 23, 2018

ruflin Mar 26, 2018

jsoriano Mar 26, 2018

ruflin commented Mar 26, 2018

jsoriano commented Mar 26, 2018

NeQuissimus commented Mar 29, 2018

ruflin commented Mar 30, 2018

Metricbeat: Support wildcards in jolokia (#6453) #6462

Metricbeat: Support wildcards in jolokia (#6453) #6462

Conversation

jsoriano commented Feb 23, 2018 • edited Loading

ruflin commented Feb 26, 2018

jsoriano commented Mar 6, 2018

jsoriano commented Mar 6, 2018 • edited Loading

jsoriano commented Mar 7, 2018

houndci-bot Mar 8, 2018

Choose a reason for hiding this comment

jsoriano commented Mar 8, 2018

ruflin commented Mar 12, 2018

jsoriano commented Mar 14, 2018

houndci-bot Mar 22, 2018

Choose a reason for hiding this comment

houndci-bot Mar 22, 2018

Choose a reason for hiding this comment

jsoriano commented Mar 22, 2018

jsoriano commented Mar 23, 2018

ruflin Mar 26, 2018

Choose a reason for hiding this comment

jsoriano Mar 26, 2018

Choose a reason for hiding this comment

ruflin commented Mar 26, 2018

jsoriano commented Mar 26, 2018

NeQuissimus commented Mar 29, 2018

ruflin commented Mar 30, 2018

jsoriano commented Feb 23, 2018 •

edited

Loading

jsoriano commented Mar 6, 2018 •

edited

Loading