Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(otel): Fix Dynamic Sampling context propagation for nested spans #559

Merged
merged 6 commits into from
Feb 1, 2023

Conversation

tonyo
Copy link
Contributor

@tonyo tonyo commented Jan 31, 2023

Thanks to #558, we can now fix the DSC/baggage propagation for non-transaction spans in the OTel propagator.

Fixes #553.

Comment on lines 48 to 53
if sentrySpan != nil {
// Dynamic sampling context is set only on the (root) transaction.
sentryTransaction := sentrySpan.GetTransaction()
if sentryTransaction != nil {
sentryBaggageStr = sentryTransaction.ToBaggage()
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the actual fix.

@tonyo tonyo requested a review from cleptric January 31, 2023 16:37
@tonyo tonyo self-assigned this Jan 31, 2023
@tonyo tonyo added Type: Improvement Topic: OpenTelemetry Issue/PR related to OpenTelemetry integration labels Jan 31, 2023
@codecov
Copy link

codecov bot commented Jan 31, 2023

Codecov Report

Base: 75.48% // Head: 75.62% // Increases project coverage by +0.13% 🎉

Coverage data is based on head (356712c) compared to base (4e5ede9).
Patch coverage: 90.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #559      +/-   ##
==========================================
+ Coverage   75.48%   75.62%   +0.13%     
==========================================
  Files          37       37              
  Lines        3765     3770       +5     
==========================================
+ Hits         2842     2851       +9     
+ Misses        808      805       -3     
+ Partials      115      114       -1     
Impacted Files Coverage Δ
tracing.go 87.77% <75.00%> (+0.30%) ⬆️
otel/propagator.go 92.10% <100.00%> (+0.21%) ⬆️
transport.go 79.21% <0.00%> (+0.84%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

tracing.go Show resolved Hide resolved
@tonyo
Copy link
Contributor Author

tonyo commented Feb 1, 2023

Alright, found another issue: in the propagator's Inject we should actually generate/freeze the baggage.
It's getting more obvious that we should refactor/redo the DSC generation and freezing logic. I think we need a more explicit FinalizeBaggage() (or similar) method on the transaction object.
I don't want to do it in this PR which already got messier than I would like.

@cleptric
Copy link
Member

cleptric commented Feb 1, 2023

What issue, in particular, you're seeing? Freezing the baggage during creation is possible today, what would a more explicit way improve?

Copy link
Member

@cleptric cleptric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor asks/questions, else LGTM!

t.Error(err)
}

if !reflect.DeepEqual(baggageGot, baggageWant) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would always use https://pkg.go.dev/github.com/google/go-cmp/cmp, had quite a few issues with reflect in the past.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, will include things like this in the planned test utils refactoring.

transaction := sentry.StartTransaction(
ctx,
emptyContextWithSentry(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
emptyContextWithSentry(),
emptyCtxWithSentry(),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a helper function that's used in multiple places already, do we really want this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Up to you, no strong feelings.

if withSpan {
span := transaction.StartChild("op")
// We want the child to have the SpanID from transactionContext, so
// we "swap" span IDs from the transaction and the child span.
transaction.SpanID = span.SpanID
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I fully understand this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When withSpan is passed, we want to create a transation and a child span. We also want the child span to have the predefined SpanID (passed via transactionContext). It means that we have to update the SpanID for the transaction. We can generate a new random one, or we can just take the already-generated one from the child span (span), that will be replaced with SpanIDFromHex(transactionContext.spanID) in the next line anyway.

@tonyo
Copy link
Contributor Author

tonyo commented Feb 1, 2023

I see the following issues:

  1. Given a transaction, there's no easy way now to generate the DSC, and then freeze it. We need to do stuff like
    transaction.SetDynamicSamplingContext(DynamicSamplingContextFromTransaction(transaction)), which is cumbersome and error-prone
  2. Span.ToBaggage() returns the serialized DSC, but doesn't generate it. Honestly, I don't see a good case of using it alone.

Freezing the baggage during creation is possible today, what would a more explicit way improve?

Ideally we should have convenient ways of explicitly generating the DSC/baggage in a proper format (as baggage values, or as a string). I don't have a super clear picture yet how it will look like, though.

@cleptric
Copy link
Member

cleptric commented Feb 1, 2023

The DSC APIs are something I want to avoid to expose to publicly, as it has lots of nitty-gritty details. Taking some extra steps for Otel is fine IMO.

Span.ToBaggage() returns the serialized DSC, but doesn't generate it. Honestly, I don't see a good case of using it alone.

You would use this function to get the value of the header string for outgoing HTTP requests.
Here is a real-world example where I'm using it: https://github.com/getsentry/gib-potato/blob/e7eda439ad93517a27f7c3201d838b769c766587/potal/httpclient.go#L37

@tonyo
Copy link
Contributor Author

tonyo commented Feb 1, 2023

Here is a real-world example where I'm using it: https://github.com/getsentry/gib-potato/blob/e7eda439ad93517a27f7c3201d838b769c766587/potal/httpclient.go#L37

Ok, fair, but there's an assumption that the DSC was already populated/frozen at some point before. When you start a transaction manually, and then call transaction.ToBaggage() to send a request before the transaction is finished, the DSC will be empty.

@cleptric
Copy link
Member

cleptric commented Feb 1, 2023

You are right, we actually need to create a DSC when calling ToBaggage if there is none. This is the case if we're the head SDK. Great catch!

@tonyo
Copy link
Contributor Author

tonyo commented Feb 1, 2023

Cool, will think about how to improve this (as a separate task)

@tonyo tonyo merged commit c9145b9 into master Feb 1, 2023
@tonyo tonyo deleted the tonyo/fix-baggage-propagation-otel-inject branch February 1, 2023 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Topic: OpenTelemetry Issue/PR related to OpenTelemetry integration Type: Improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[OTel] Dynamic Sampling Context propagation does not work for nested spans
2 participants