Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix a potential bug that ThreadError may occur on SIGUSR1 #2678

Merged
merged 1 commit into from
Oct 31, 2019

Conversation

ashie
Copy link
Member

@ashie ashie commented Oct 31, 2019

Which issue(s) this PR fixes:

None in GitHub issues, but it will fix the following bug report:

We ClearCode Inc. recently received same bug reports from our some customers.
Their fluentd is v0.12 but I think the same potential bug still exists in the latest version.

What this PR does / why we need it:

This PR removes redundant run calls for some Thread objects.
It may cause the following error on SIGUSR1:

[error]: unexpected error error_class=ThreadError error=#<ThreadError: killed thread>

as reported in

because the new thread may finish before calling "run" method at main
thread.

In addition this PR also replaces { ... } style blocks for them with do ... end style,
because the later style will clarify such mistakes.

Docs Changes:

None.

Release Note:

Same as title

It may cause the following error:

  [error]: unexpected error error_class=ThreadError error=#<ThreadError: killed thread>

as reported in

  https://groups.google.com/d/msg/fluentd/Ten_apYLAX8/PD4jYk8KDwAJ

because the new thread may finish before calling "run" method at main
thread.

In addition replace { ... } style blocks for them with do ... end style,
because the later style will clarify such mistakes.

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@repeatedly repeatedly merged commit 572aa34 into fluent:master Oct 31, 2019
@repeatedly
Copy link
Member

Looks good. Thanks!

@savy1212
Copy link

Which fluentd version this fix is given??

I am facing the same issue in fluent-plugins-3.4.1-1.0.2.x86_64

@daipom
Copy link
Contributor

daipom commented Apr 12, 2023

@savy1212
Copy link

Thanks, I am not good at fluentd.

So that means my fluentd version is 1.2.2 and older than the fixed version right!!

2023-03-08 17:45:01 +0800 [error]: #0 unexpected error error_class=ThreadError error="killed thread"
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:704:in run' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:704:in flush_buffer'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:664:in block in install_main_process_signal_handlers' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:228:in sleep'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:228:in run' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:799:in run_engine'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:549:in block in run_worker' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:724:in main_process'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:544:in run_worker' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/command/fluentd.rb:316:in <top (required)>'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/bin/fluentd:8:in <top (required)>' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in load'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `

'
2023-03-08 17:45:01 +0800 [error]: #0 unexpected error error_class=ThreadError error="killed thread"
2023-03-08 17:45:01 +0800 [error]: #0 suppressed same stacktrace

@ashie ashie deleted the fix-thread-error branch April 12, 2023 02:43
@daipom
Copy link
Contributor

daipom commented Apr 12, 2023

Yes!
There are some Fluentd packages such as td-agent.
The version of the package and the version of Fluentd embedded in it may differ.

@savy1212
Copy link

Thanks again, my td-agent package is td-agent-3.2.0-0.el7.x86_64.
It actually led to a major impact; I think I should go ahead with the upgrade.

@daipom
Copy link
Contributor

daipom commented Apr 12, 2023

If you are updating to the latest version of td-agent v4, the following blog may be helpful.

https://www.fluentd.org/blog/upgrade-td-agent-v3-to-v4

@savy1212
Copy link

savy1212 commented May 4, 2023

We have had this log rotation happening 4 times a day for like 1 year and working fine, now suddenly this bug is triggered, we are not sure what can trigger this bug,

Are there specific scenarios that can trigger this bug?

Also is there a way to just edit the supervisor.rb of fluent 1.2.2 file with the fix mentioned in this bug by removing extra run from the code if we dont want to upgrade, will it work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants