Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

do not consume compressed files when compression is not enabled #35447

Closed
newly12 opened this issue Sep 27, 2024 · 3 comments
Closed

do not consume compressed files when compression is not enabled #35447

newly12 opened this issue Sep 27, 2024 · 3 comments
Labels
enhancement New feature or request receiver/filelog

Comments

@newly12
Copy link
Contributor

newly12 commented Sep 27, 2024

Component(s)

receiver/filelog

Is your feature request related to a problem? Please describe.

Currently filelog supports consume compressed files(gzip only at this moment), if there is no compression set in receiver config, and file pattern happens to match compressed files, it treats them as "plain" text and sends out the compressed content. Would it make sense to detect file type(mimetype) and automatically(or configurable) to stop processing files if they are not plain text?

Describe the solution you'd like

detect file mimetype and exclude compressed files

Describe alternatives you've considered

No response

Additional context

2024-09-27T10:15:27.643+0800    info    fileconsumer/file.go:256        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/tmp/logs/1.log.gz"}
...
Body: Str(�f1.log�W�n�6��+]����ӻA'h`���5%�$��;��ʘ6T"u4�;��&���iw0�����N)�\��ݏ7�с�T�W�)
@newly12 newly12 added enhancement New feature or request needs triage New item requiring triage labels Sep 27, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@djaglowski
Copy link
Member

This seems like it would add some overhead just to always check for an edge case. Can the user just instead exclude files which end with extensions like .gzip?

@atoulme atoulme removed the needs triage New item requiring triage label Oct 12, 2024
@newly12
Copy link
Contributor Author

newly12 commented Oct 17, 2024

Make sense, I think we can also use the exclude for *.gzip / *.gz files.

@newly12 newly12 closed this as completed Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request receiver/filelog
Projects
None yet
Development

No branches or pull requests

3 participants