Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Kubernetes metadata fields not expanded in rollover_alias and policy_name #15823

Closed
TomaszKlosinski opened this issue Jan 24, 2020 · 5 comments
Labels
Team:Integrations Label for the Integrations team

Comments

@TomaszKlosinski
Copy link

Describe the enhancement:

As I mentioned in this comment, it's possible to use index per kubernetes namespace with old (daily) index format:

output.elasticsearch.index: "%{[kubernetes.namespace]:filebeat}-%{+yyyy.MM.dd}"

But it's not possible with the new format supported by ILM. Therefore, I can use either one feature or second. I can't use both logs index per kubernetes namespace and ILM. It would be a great enhancement for us, if we could use both.

Currently, my workaround is to use daily indices, but implement separately ILM with API calls - this however has a limit, as rollover action in ILM does not support indices with daily format. I can use ILM only to delete old indices. Alternatively, I could drop ILM and use curator, but I'd prefer to use the newer solution.

Describe a specific use case for the enhancement or feature:

I'd like to be able to use an index and ILM per kubernetes namespace, like this:

    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "%{[kubernetes.namespace]:filebeat}" 
    setup.ilm.pattern: "{now/d}-000001"
    setup.ilm.overwrite: true
    setup.ilm.policy_name: "%{[kubernetes.namespace]:filebeat}" 
@TomaszKlosinski TomaszKlosinski changed the title Filebeat: Kubernetes metadata fields not expanded in rollover_alias Filebeat: Kubernetes metadata fields not expanded in rollover_alias and policy_name Jan 24, 2020
@TomaszKlosinski
Copy link
Author

TomaszKlosinski commented Jan 27, 2020

One update: I've realised that it's not about the index name format I can use e.g.:

output.elasticsearch.index: "%{[kubernetes.namespace]:filebeat}-000001"

There are two problems here. One is that the fileds are not expanded, but the other is that ILM configuration is done only once during initialization of filebeat, so even with the field expansition fix, I still can't have a more dynamic log forwarding based on the kubernetes fields (for each log message coming from different pods/containers). I think the solution would be to first check whether a given ILM configuration exists for a given event (log message) and then send it there, otherwise initialize new ILM configuration and then send it.

@TomaszKlosinski TomaszKlosinski changed the title Filebeat: Kubernetes metadata fields not expanded in rollover_alias and policy_name [Filebeat] Kubernetes metadata fields not expanded in rollover_alias and policy_name Jan 27, 2020
@zorgzerg
Copy link

zorgzerg commented Feb 4, 2020

+1

@andresrc
Copy link
Contributor

Unfortunately, we do not plan to support this directly in the near future, but to solve it in a somewhat different way. Closing for now. Thanks!

@andresrc andresrc added the Team:Integrations Label for the Integrations team label Mar 6, 2020
@plaformsre
Copy link

Hi @andresrc ,

What is the solution then, if solved in a different way? If documented, please point me to the documentation. Thanks.

@sardanam
Copy link

What's the different way here ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Integrations Label for the Integrations team
Projects
None yet
Development

No branches or pull requests

6 participants