Skip to content

Commit

Permalink
Fix heartbeat races on event updates (elastic#6950)
Browse files Browse the repository at this point in the history
Heartbeat has a many components, each adding fields to an existing
event. As events are created per 'TaskRun', this is normally fine. But
in case the TaskRunner branches of into multiple Sub-tasks, we will see
races on shared event structures. This is the case with multiple ports
in the http/tcp configuration or the IPAll setting (ping all known IPs
of a given domain).

Namespaces in an event can be potentially populated via globally shared
structures. This can also lead to unwanted updates or races.

This fix clones the event structure before continuing event updates, so
to remove the chance of races.
Steffen Siering authored and ruflin committed Apr 26, 2018
1 parent 44ee36f commit e92ed3e
Showing 2 changed files with 9 additions and 3 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
@@ -67,6 +67,7 @@ https://github.com/elastic/beats/compare/v6.0.0-beta2...master[Check the HEAD di
- Add raw JSON to message field when JSON parsing fails. {issue}6516[6516]

*Heartbeat*
- Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. {issue}6616[6616]

*Metricbeat*

11 changes: 8 additions & 3 deletions heartbeat/monitors/util.go
Original file line number Diff line number Diff line change
@@ -122,6 +122,8 @@ func annotated(
}

if fields != nil {
fields = fields.Clone()

status := look.Status(err)
fields.DeepUpdate(common.MapStr{
"monitor": common.MapStr{
@@ -130,7 +132,7 @@ func annotated(
},
})
if user := settings.Fields; user != nil {
fields.DeepUpdate(user)
fields.DeepUpdate(user.Clone())
}

event.Timestamp = start
@@ -370,10 +372,13 @@ func resolveErr(host string, err error) (common.MapStr, []TaskRunner, error) {
func WithFields(fields common.MapStr, r TaskRunner) TaskRunner {
return MakeCont(func() (common.MapStr, []TaskRunner, error) {
event, cont, err := r.Run()
if event == nil {
if event != nil {
event = event.Clone()
event.DeepUpdate(fields)
} else if err != nil {
event = common.MapStr{}
event.DeepUpdate(fields)
}
event.DeepUpdate(fields)

for i := range cont {
cont[i] = WithFields(fields, cont[i])

0 comments on commit e92ed3e

Please sign in to comment.