Panic 'fatal error: concurrent map iteration and map write' autodiscover and processor. #8040
Comments
|
Note: |
|
I've run this weekend a stress test with Filebeat and only the The code changed recently with the addition of netinfo in #6878 |
|
I will keep it running with the race detector to see if I catch the offending goroutine. theory so far is the way filebeat is structured with the outlet and the cache data structure inside the processors is problematic. |
|
running 6.3.1 and getting the same error as well. pulled code and rebuild with -race, getting the following race condition on 6.3.1 and 6.4.0 contents of filebeat.yml |
|
Thank you for the trace. Which version did you create the last trace with? The Lines between both versions will be different, but I think I found it. Skimming the trace, the bug seems to be most likely due to add_host_metadata. Updating |
|
Awesome for the trace I will make a patch ASAP
On Sat, Sep 1, 2018 at 3:36 PM Steffen Siering ***@***.***> wrote:
Thank you for the trace.
Which version did you create the last trace with? The Lines between both
versions will be different, but I think I found it.
Skimming the trace, the bug seems to be most likely due to
add_host_metadata. add_host_metadata adds a dictionary to the host field.
The dictionary which is shared (by reference) between a many events. The
host.name field (added with beats 6.3) is some global meta-data which
overwrites and updates the host document. Beats add global meta-data at
the end of any processing. Due to the host document being a shared
instance, we see a race condition when it is serialized in an older event,
while a new event is actively processed.
Updating Run
<https://github.com/elastic/beats/blob/master/libbeat/processors/add_host_metadata/add_host_metadata.go#L71>
in the add_host_metadata processor should fix this race condition:
func (p *addHostMetadata) Run(event *beat.Event) (*beat.Event, error) {
p.loadData()
event.Fields.DeepUpdate(p.data.Clone())
return event, nil
}
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#8040 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAACgAxGqkf57xIE5S9lquHlK9oWnb9Fks5uWuGkgaJpZM4WGYeI>
.
--
ph
|
ph
added a commit
to ph/beats
that referenced
this issue
Sep 4, 2018
Fix a race between updating the host data structure and the serialization of events to json. closes elastic#8040
ph
added a commit
that referenced
this issue
Sep 5, 2018
Fix a race between updating the host data structure and the serialization of events to json. closes #8040
ph
added a commit
to ph/beats
that referenced
this issue
Sep 5, 2018
Fix a race between updating the host data structure and the serialization of events to json. closes elastic#8040 (cherry picked from commit 415715d)
ruflin
pushed a commit
that referenced
this issue
Sep 5, 2018
ph
added a commit
to ph/beats
that referenced
this issue
Sep 5, 2018
Fix a race between updating the host data structure and the serialization of events to json. closes elastic#8040 (cherry picked from commit 415715d)
This was referenced Sep 5, 2018
ph
added a commit
to ph/beats
that referenced
this issue
Sep 5, 2018
Fix a race between updating the host data structure and the serialization of events to json. closes elastic#8040 (cherry picked from commit 415715d)
leweafan
pushed a commit
to leweafan/beats
that referenced
this issue
Apr 28, 2023
…st_metadata processor (elastic#8237) Fix a race between updating the host data structure and the serialization of events to json. closes elastic#8040 (cherry picked from commit b497720)
leweafan
pushed a commit
to leweafan/beats
that referenced
this issue
Apr 28, 2023
…) (elastic#8225) Fix a race between updating the host data structure and the serialization of events to json. closes elastic#8040 (cherry picked from commit b497720)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Created from: https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296
This look like we are not copying a shared data structure and this make the code panic with a concurrent error.
The text was updated successfully, but these errors were encountered: