Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

native histogram: Fix race between Write and addExemplar #1623

Merged

Conversation

krajorama
Copy link
Member

@krajorama krajorama commented Sep 6, 2024

Follow-up to #1608

Fixes

2024-09-06T08:32:44.9910707Z 08:29:59 ingester-3: WARNING: DATA RACE
2024-09-06T08:32:44.9911367Z 08:29:59 ingester-3: Read at 0x00c001b9f4d8 by goroutine 480:
2024-09-06T08:32:44.9912309Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*histogram).Write()
2024-09-06T08:32:44.9913404Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:849 +0x16ef
2024-09-06T08:32:44.9914502Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*wrappingMetric).Write()
2024-09-06T08:32:44.9915647Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/wrap.go:172 +0x65
2024-09-06T08:32:44.9916698Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*wrappingMetric).Write()
2024-09-06T08:32:44.9917746Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/wrap.go:172 +0x65
2024-09-06T08:32:44.9919099Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*wrappingMetric).Write()
2024-09-06T08:32:44.9920110Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/wrap.go:172 +0x65
2024-09-06T08:32:44.9921088Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.processMetric()
2024-09-06T08:32:44.9922271Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/registry.go:633 +0x112
2024-09-06T08:32:44.9923296Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*Registry).Gather()
2024-09-06T08:32:44.9924408Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/registry.go:502 +0xe8a
2024-09-06T08:32:44.9925524Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*noTransactionGatherer).Gather()
2024-09-06T08:32:44.9926636Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/registry.go:1074 +0x3a
2024-09-06T08:32:44.9927818Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1()
2024-09-06T08:32:44.9929026Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/promhttp/http.go:165 +0x3fa

vs

2024-09-06T08:32:44.9959927Z 08:29:59 ingester-3: Previous write at 0x00c001b9f4d8 by goroutine 370:
2024-09-06T08:32:44.9960836Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*nativeExemplars).addExemplar()
2024-09-06T08:32:44.9962035Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1706 +0x188b
2024-09-06T08:32:44.9963229Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*histogram).updateExemplar()
2024-09-06T08:32:44.9964331Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1140 +0x116
2024-09-06T08:32:44.9965403Z 08:29:59 ingester-3: github.com/prometheus/client_golang/prometheus.(*histogram).ObserveWithExemplar()
2024-09-06T08:32:44.9966601Z 08:29:59 ingester-3: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:770 +0x69
2024-09-06T08:32:44.9967591Z 08:29:59 ingester-3: github.com/grafana/dskit/instrument.ObserveWithExemplar()

Follow-up to 1608

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Copy link
Member

@beorn7 beorn7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you.

(And as usual, handing over to @bwplotka @ArthurSens @kakkoyun for final check and merge.)

@krajorama krajorama merged commit 05fcde9 into prometheus:release-1.20 Sep 7, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants