Add support for adding aliases to index and shard stats

[bobo333]

Optionally will include aliases in a label to index and shard stats. Addresses #412

This adds a cli option es.aliases which when enabled, will add all aliases pointing to a given index, comma-separated, into a label called aliases for index and shard stats.

ex:
3 indices are created, foo_1, foo_2, and foo_3. foo_2 has one alias pointing to it called foo_alias_2_1. foo_3 has 2 aliases pointing to it called foo_alias_3_1 and foo_alias_3_2.

without enabling es.aliases flag:

...
elasticsearch_indices_store_size_bytes_total{cluster="docker-cluster",index="foo_1"} 4518
elasticsearch_indices_store_size_bytes_total{cluster="docker-cluster",index="foo_2"} 4564
elasticsearch_indices_store_size_bytes_total{cluster="docker-cluster",index="foo_3"} 4564
...

with es.aliases flag:

...
elasticsearch_indices_store_size_bytes_total{aliases="",cluster="docker-cluster",index="foo_1"} 4518
elasticsearch_indices_store_size_bytes_total{aliases="foo_alias_2_1",cluster="docker-cluster",index="foo_2"} 4564
elasticsearch_indices_store_size_bytes_total{aliases="foo_alias_3_1,foo_alias_3_2",cluster="docker-cluster",index="foo_3"} 4564
...

[bobo333]

@sysadmind @SuperQ wondering if you had any thoughts or feedback on this

[sysadmind]

Having variable labels is not a good idea. I understand what the intent is, and I think that having aliases exposed is a good idea. I don't think that they belong in a label of the existing metrics however.

I think it would be better to have a new metric elasticsearch_indices_alias with two labels index and alias. The scarping of the aliases would stay the same, but the reporting of the metric would be a loop over the indices and aliases and to expose the above gauge for each index-alias combination with a value of 1. This is a common pattern in prometheus for informational metrics.

How would this help your use case? The alias information would come at query time. You can use your normal query but add group_left(index) to get the alias label into your existing queries.

[bobo333]

@sysadmind thank you very much for the review! I will get started on those changes. Out of curiosity - what is the reason for not wanting aliases in labels of existing metrics? From the end user perspective it seems it would be easier to filter on a label rather than doing group_left, however I imagine you have a good reason for suggesting that route. Thanks again!

[SuperQ]

The main issue is that label values in Prometheus are opaque strings. So you end up with problems trying to do one-to-many relationships. Mapping a list of aliases to a coma separate list is not as useful as mapping to an information metric.

Take your example, it's now much easier to match the literal single alias depend on which one you're looking for:

elasticsearch_indices_store_size_bytes_total{cluster="docker-cluster",index="foo_3"}
...
elasticsearch_indices_aliases{alias="foo_alias_3_1",cluster="docker-cluster",index="foo_3"} 1
elasticsearch_indices_aliases{alias="foo_alias_3_2",cluster="docker-cluster",index="foo_3"} 1

[bobo333]

@SuperQ that makes sense, thank you for explaining!

one last question - would you like the addition of the alias information metrics to be optionally enabled by a cli option, or always enabled? thanks again!

[SuperQ]

I think we could enable it by default, with a flag to disable it if it generates too many metrics for some users.

[bobo333]

@SuperQ @sysadmind this is ready for another look, thanks!

example stats:

...
# HELP elasticsearch_indices_aliases Record aliases associated with an index
# TYPE elasticsearch_indices_aliases gauge
elasticsearch_indices_aliases{alias="foo_alias_2_1",cluster="docker-cluster",index="foo_2"} 1
elasticsearch_indices_aliases{alias="foo_alias_3_1",cluster="docker-cluster",index="foo_3"} 1
elasticsearch_indices_aliases{alias="foo_alias_3_2",cluster="docker-cluster",index="foo_3"} 1
...

[sysadmind]

LGTM I think this is ok now

[SuperQ]

Nice