[receiver/filelog] Support option to ignore files older than certain age

[ycombinator]

Component(s)

receiver/filelog

Is your feature request related to a problem? Please describe.

For users who keep log files around for long periods of time, it might be useful to tell the filelog receiver to only consider files that are younger than a certain age or, in other words, to ignore files that are older than a certain age.

Describe the solution you'd like

A new, optional, configuration option for the filelog receiver, something like exclude_older_than, which accepts a duration as its value.

Describe alternatives you've considered

No response

Additional context

Filebeat, Elastic's log shipper, supports such an option in it's filestream input, which is conceptually the same as OTel Collector's filelog receiver.

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

@ycombinator, have you looked into whether filelog's ordering_criteria.sort_by.sort_type = mtime works for your use case?

[ycombinator]

Thanks @djaglowski. I did look into that option in combination with ordering_criteria.top_n but the way I understood it, that combination would let me say something to the effect of "only consider the top (latest) N files, sorted by mtime". I couldn't figure out a way to say "only consider all files newer than a certain mtime", which is what I'm proposing in this issue. If I missed some combination of the ordering_criteria.* options that could satisfy the latter use case, please let me know.

[djaglowski]

Thanks for confirming @ycombinator. I see the difference and agree it is a sensible feature. I think it would make sense to add it in the matcher package.

[github-actions]

Pinging code owners for pkg/stanza: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[thomasriera-akkodis]

Hello

This feature would indeed be very useful. I have been trying to set up a similar behaviour using ordering by date and top_n. It works well enough for now but a way to ignore old files would be better.

[ycombinator]

Hi @djaglowski, I'd like to work on this issue if that's okay.

[UPDATE] I've put up a PR to try and resolve this issue: #31916.

[ycombinator]

Implemented via #31916