Bump fluentd and enable gzip compression by default #1001
Conversation
sumo-drosiek
requested review from
a team,
frankreno,
samjsong,
vsinghal13,
perk-sumo,
andrzej-stencel and
pmalek-sumo
and removed request for
a team
| ## usage of the gzip compression is highly not recommended due to fluentd issue: | ||
| ## rel: https://github.com/fluent/fluentd/issues/3056 | ||
| compress: text | ||
| compress: gzip |
There was a problem hiding this comment.
The issue from the comment doesn't seem closed/fixed: why are we changing this?
There was a problem hiding this comment.
I think I put the wrong issue in the comment 🤦
fluent/fluentd#3110
There was a problem hiding this comment.
Or I created new one afterward, anyway the issue is fixed and compression works properly
There was a problem hiding this comment.
I got into this issue after upgrading to latest Sumologic chart v2.1.1 with compression enabled by default. It keeps logging below warnings -
2021-04-20 21:29:59 +0000 [warn]: #0 [sumologic.endpoint.logs.default] failed to flush the buffer. retry_time=2 next_retry_seconds=2021-04-20 21:30:01 +0000 chunk="5c06a1587371d3cc21653db8ac435965" error_class=Zlib::GzipFile::Error error="not in gzip format"
2021-04-20 21:29:59 +0000 [warn]: #0 suppressed same stacktrace
2021-04-20 21:30:01 +0000 [warn]: #0 [sumologic.endpoint.logs.kubelet] failed to flush the buffer. retry_time=3 next_retry_seconds=2021-04-20 21:30:05 +0000 chunk="5c06a1582a29b9530705397ec337ef87" error_class=Zlib::GzipFile::Error error="not in gzip format"
2021-04-20 21:30:01 +0000 [warn]: #0 suppressed same stacktrace
2021-04-20 21:30:05 +0000 [warn]: #0 [sumologic.endpoint.logs.default] failed to flush the buffer. retry_time=4 next_retry_seconds=2021-04-20 21:30:14 +0000 chunk="5c06a1587371d3cc21653db8ac435965" error_class=Zlib::GzipFile::Error error="not in gzip format"
2021-04-20 21:30:05 +0000 [warn]: #0 suppressed same stacktrace
Also this caused sudden dropping of messages too. So I ended up setting the buffer compression to 'text' and for plugin output, set compression to 'deflate' as per these logs,
2021-04-21 09:05:52 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.1'
2021-04-21 09:05:52 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2021-04-21 09:05:52 +0000 [info]: gem 'fluentd' version '1.12.2'
/usr/local/bundle/gems/fluent-plugin-sumologic_output-1.7.1/lib/fluent/plugin/out_sumologic.rb:23:in initialize': Invalid compression encoding text must be gzip or deflate (RuntimeError) from /usr/local/bundle/gems/fluent-plugin-sumologic_output-1.7.1/lib/fluent/plugin/out_sumologic.rb:209:in new'
from /usr/local/bundle/gems/fluent-plugin-sumologic_output-1.7.1/lib/fluent/plugin/out_sumologic.rb:209:in configure' from /usr/local/bundle/gems/fluentd-1.12.2/lib/fluent/plugin.rb:178:in configure'
There was a problem hiding this comment.
@AshutoshNirkhe Do you think that can be related to this issue?
#1522
There was a problem hiding this comment.
Absolutely @sumo-drosiek I think I came here from that issue probably.
There was a problem hiding this comment.
Ok, we are going to track the progress there
Description
Bump fluentd and enable gzip compression by default
Testing performed