-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Receive: fix thanos_receive_write_{timeseries,samples}
stats
#7643
Receive: fix thanos_receive_write_{timeseries,samples}
stats
#7643
Conversation
44490ea
to
0aa3bf0
Compare
@MichaHoffmann I think your fix for these metrics does not work quite right, since we need to account not for local/remote writes but for the replication factor, as far as I understand it. Could you have a look at this change? |
89dcf30
to
dd1a8d5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm. Can you update changelog with the fix please?
dd1a8d5
to
0766b50
Compare
@MichaHoffmann I've updated the changelog. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lg
0766b50
to
be97ef7
Compare
Fixed a merge conflict in CHANGELOG |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@cincinnat this also needs a slight doc update, you can do that by running |
918b063
11cfa82
to
8d4c777
Compare
The test failure seems irrelevant to the fix. It seems like a possible race condition in the test to me: if test.bucket != nil {
go func() {
testutil.Ok(t, syncTSDBs(ctx, m, 10*time.Millisecond))
}()
} There is no guarantee |
This reverts commit 66841fb. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io>
Head branch was pushed to by a user without write access
8d4c777
to
c2f47f1
Compare
Just rebased on #7694 to have a green build. Previous build failed due to the flaky test fixed by that PR. |
There are two path data can be written to a receiver: through the HTTP or the gRPC endpoint, and `thanos_receive_write_{timeseries,samples}` only count the number of timeseries/samples received through the HTTP endpoint. So, there is no risk that a sample will be counted twice, once as a remote write and once as a local write. On the other hand, we still need to account for the replication factor, and only count local writes is not enough as there might be no local writes at all (e.g. in RouterOnly mode). Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io>
c2f47f1
to
dbc840a
Compare
…s-io#7643) * Revert "Receive: fix stats (thanos-io#7373)" This reverts commit 66841fb. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> * Receive: fix `thanos_receive_write_{timeseries,samples}` stats There are two path data can be written to a receiver: through the HTTP or the gRPC endpoint, and `thanos_receive_write_{timeseries,samples}` only count the number of timeseries/samples received through the HTTP endpoint. So, there is no risk that a sample will be counted twice, once as a remote write and once as a local write. On the other hand, we still need to account for the replication factor, and only count local writes is not enough as there might be no local writes at all (e.g. in RouterOnly mode). Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> --------- Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io>
There are two path data can be written to a receiver: through the HTTP or the gRPC endpoint, and
thanos_receive_write_{timeseries,samples}
only count the number of timeseries/samples received through the HTTP endpoint.So, there is no risk that a sample will be count twice, once as a remote write and once as a local write. On the other hand, we still need to account for the replication factor, and only count local writes is not enough as there might be no local writes at all (e.g. in RouterOnly mode).
Changes
Verification