Traverse the Elasticsearch JSON tree to automatically include any numeric values

[regiswilson]

There are a number of metrics that are not exported by the current code. Also, metrics are renamed and slightly differ against the elastic search values. This code traverses the output of /_nodes/_local/stats and pulls every numeric metric out without changes.

Plus, less code is better code. :)

One serious caveat: the metrics names will be different so this is a breaking change.

[johann8384]

Can you attach a list of the new metrics and tags?

[regiswilson]

esmaster.txt
esclient.txt

Here are two files showing example output, one for an elasticsearch client node and an elasticsearch master node.

[johann8384]

I like this collector, let me think about a way to have it alongside the previous one and issue deprecation warnings.

[regiswilson]

We use this in production against 1.7.1 and also tested in dev agaist 2.2.0. It fails against AWS es 1.5.1 but only because Amazon denies requests to /_cluster. It's possible to write some testing for those responses and handle them gracefully at some point.

[regiswilson]

Pardon the jamming, we've open-sourced some more fixes to allow threading for multiple hosts (we monitor up to 10 nodes with one collector, bringing run time from 3s to .5s). We also include byindex stats for bytes and docs (per index) from the master.

Attached is a revised copy of the master's metric outputs. Client/data nodes are unchanged.
esmaster.txt

[johann8384]

Would it be possible to get the plugin names here? I would be concerned that plugin 0 wasn't the same plugin across multiple instances.

elasticsearch.cluster.nodes.plugins.0.site 1455667836 1 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.0.jvm 1455667836 1 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.1.site 1455667836 0 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.1.jvm 1455667836 1 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.2.site 1455667836 0 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.2.jvm 1455667836 1 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.3.site 1455667836 0 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.3.jvm 1455667836 1 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.4.site 1455667836 1 node=esmasterlogs1 cluster=eslogs
elasticsearch.cluster.nodes.plugins.4.jvm 1455667836 0 node=esmasterlogs1 cluster=eslogs

[regiswilson]

This was introduced in
https://github.com/OpenTSDB/tcollector/pull/269/files#diff-319e32e98481f63c8ce941c4ee06c749R127

I think it is a minor cosmetic issue since these are plugin booleans. I think there are several scenarios:

1. Ignore it
2. Drop booleans
3. Add a blacklist facility to not descend certain nodes like "plugins" and "vendor"
4. Try to add the peer non-numeric nodes as tags for the numeric values that we found (may add a lot of even more unnecessary information and pollute the tags)

Option 1 has the least risks and easiest to implement
Option 2 is tempting but not sure what we'd miss (values like "is_master")
Option 3 potentially introduces differences when ES data structures are updated/modified
Option 4 sounds too convoluted and the combined worst of all scenarios

I'll mull over options 2 or 3 to see if this is easy.

[regiswilson]

Here's the json that causes it. Not a lot of good metrics in there. Interesting data but not metrics.

{"plugins":[{"name":"marvel","version":"1.3.1","description":"Elasticsearch Management & Monitoring","url":"/_plugin/marvel/","jvm":true,"site":true},{"name":"watcher","version":"1.0.1","description":"Elasticsearch Watcher","jvm":true,"site":false},{"name":"license","version":"1.0.0","description":"Internal Elasticsearch Licensing Plugin","jvm":true,"site":false},{"name":"watcher","version":"1.0.0","description":"Elasticsearch Watcher","jvm":true,"site":false},{"name":"head","version":"NA","description":"No description found.","url":"/_plugin/head/","jvm":false,"site":true}]}

[regiswilson]

Updated to remove bool metrics.