Filebeat: variables not expanded in rollover_alias

[fpompermaier]

• Version: Filebeat 7.0.1 and 7.1.0
• Operating System: Ubuntu 18.04 LTS

From documentation rollover_alias is by default filebeat-{agent.version} but if I set

setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-{agent.version}"
setup.ilm.pattern: "{now/d}-000001"

the index name is not the expected one (i.e. filebeat-7.1.0-2019.05.21-000001) but the index is created with name filebeat-{agent.version}.

Is it a problem of the documentation or is it a bug?

[exekias]

I'm guessing a $ is missing? perhaps something like this could work?:

setup.ilm.rollover_alias: "filebeat-${agent.version}"

pinging @simitt @urso

[simitt]

Agree, that the $ is missing.

[fpompermaier]

If I do that I get the following error when starting the filebeat:

Exiting: missing field accessing 'setup.ilm.rollover_alias' (source:'filebeat.yml')

[simitt]

I can confirm there is a bug. Beats exit with an error on startup or when trying to run the setup cmd whenever setup.ilm.rollover_alias is configured to include variables to expand.

As a workaround, @fpompermaier could you specify the version in the configuration until this is fixed?

[fpompermaier]

Fortunately commenting out setup.ilm.rollover_alias just works for me (I was going to use filebeat-{agent.version} anyway) so there's no problem right now in my setup.

[simitt]

Update: the configuration would actually be %{[agent.version]}. The issue we saw was due to the $ sign.
Nevertheless, according to the docs adding an agent.version should not be necessary:

The index lifecycle write alias name. The default is filebeat-{agent.version}. Setting this option changes the prefix in the alias name. It doesn’t remove agent.version from the alias name.

I would assume that independent of the config for setup.ilm.rollover_alias the alias always gets prefixed with the beat version. This is not the case (since 7.0.0) for custom config, which is indeed a bug (but a different one than originally expected).

[simitt]

@ph when working on this, you could modify the existing system test to pass in an alias without the version and check that an alias with the version is created.

[ph]

There are actually two distinct problem here, the rolloverAlias doesn't support the fieldref syntax, but to be constant with the documentation, I've decided to automatically appends the version, I did not make the rolloverAlias accepts a fieldref.

[gboddin]

I've been changing the index names depending on the log source for a long time, it became a natural feature to have filebeat handle multiple log sources to different indices.

Beat was never used to create the templates, rather, we have a separate CI process to maintain pipelines and templates ( and now ILM ).

With ILM enabled in beat there's no more way to do it and you have to spawn 1 filebeat per indices you need to populate. Not ideal at all.

This

filebeat.inputs:
- type: log
  paths: ["/logs/internal/[0-9]*.log"]
  json.message_key: message
  json.keys_under_root: true
  json.overwrite_keys: true
  fields:
    hostname: ${HOSTNAME}
    location: ${LOCATION}
    log_index: applog-internal
- type: log
  paths: ["/logs/external/[0-9]*.log"]
  json.message_key: message
  json.keys_under_root: true
  json.overwrite_keys: true
  fields:
    hostname: ${HOSTNAME}
    location: ${LOCATION}
    log_index: applog-external

output.elasticsearch:
  hosts: ["http://es-ingest001:9200", "http://es-ingest002:9200"]

processors:
  - rename:
      fields:
        - from: "@timestamp"
          to: "received"
        - from: "timestamp"
          to: "@timestamp"
      ignore_missing: false
      fail_on_error: true

setup.template.enabled: false
setup.ilm.enabled: true
setup.ilm.rollover_alias: "{fields.log_index}"
setup.ilm.pattern: "{now/d}-000001"

Is not working anymore ( using index_name worked before ).

Did I miss something ?

[gboddin]

I've ended up spawning 1 filebeat per log type for now ...

[fkelbert]

It is indeed a problem in real-world deployments if both

• %{[agent.version]} does not expand in rollover_alias, and
• the version is not being appended (this is different from what is stated in the documentation).

The following configuration results in the index name below:

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "packetbeat-cust10000"
setup.ilm.pattern: "000001"

Resulting index name: packetbeat-cust10000-000001 (Note: no version number, e.g., 7.2.0).

It seems there is currently no way to fix this (Beats 7.2.0). What we would like to have is the following:

packetbeat-7.2.0-cust10000-000001 or packetbeat-cust10000-7.2.0-000001.

[fpompermaier]

I've just tested the 7.5 version and now I can specify the variables in the pattern and in the alias.
However the documentation is still wrong: at https://www.elastic.co/guide/en/beats/filebeat/master/ilm.html#setup-ilm-rollover_alias-option it says that the default is The default is filebeat-%{agent.version} while it should be filebeat-%{[agent.version]}.

Is also the example setup.ilm.pattern right ({now/M{YYYY.MM}}-000001)?
Shouldn't it have a percent in front of the curly bracket?

[urso]

The ilm.pattern is correct. The date pattern is interpreted by Elasticsearch, not Beats.

The docs for setup.ilm.rollover_alias is indeed missing the brackets.

[TomaszKlosinski]

So after this change, can I use kubernetes variables? Like in this example:

    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "%{[kubernetes.namespace]:filebeat}" 
    setup.ilm.pattern: "{now/d}-000001"
    setup.ilm.overwrite: true
    setup.ilm.policy_name: "%{[kubernetes.namespace]:filebeat}" 

[urso]

So after this change, can I use kubernetes variables? Like in this example:

No. ILM and templates are global settings. Beats do not manage templates + policies for multiple indices and have no access to fields in events. In fact templates and ILM policies are setup before the first event is available. This issue is about fixing the documentation, not changing behavior.

[TomaszKlosinski]

@urso , this sucks because it does work with the old-school daily indices:

output.elasticsearch.index: "%{[kubernetes.namespace]:filebeat}-%{+yyyy.MM.dd}"

So, in this case I can either use an index pattern per kubernetes namespace, or I can use ILM. I can't use both. :(