tracing: fix lazytag export for Jaegar/otel

[aadityasondhi]

Previously, lazytags would not render in the otel export to
Jaeger. This was because otel doesn't support an on-demand
nature of the lazytags in its interface.

This change manually adds lazytags as otel tags before an otel
span is finished, allowing them to render in Jaeger after they
are exported.

Resolves: #85166

Release note: None

Release justification: bug fixes

[cockroach-teamcity]

This change is 

[andreimatei]

Good stuff?

Can you please add a test? See if TestOtelTracer can be tweaked to do it.

Did you happen to figure out why the rendering of the lock time in the diagnostics bundle was off?

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @aadityasondhi and @andreimatei)

---

pkg/util/tracing/span_inner.go line 107 at r1 (raw file):

	if s.otelSpan != nil {
		// It is ok to access s.crdb after calling s.crdb.finish() since it will be cleaned up by the parent of this func

Scratch this comment. Replace it with // Serialize the lazy tags.

If it stays, it needs a period, 80-col wrapping (ask me about an golang plugin if you don't have it) and it needs to be clear who the "it" is in "it will be cleaned up".

---

pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

		s.crdb.mu.Lock()
		defer s.crdb.mu.Unlock()
		for _, kv := range s.crdb.mu.lazyTags {

let's make this a method on s.crdb that returns []atribute.KeyValue. I think you've copied this code from somewhere, right? It'd probably be good to share.

[aadityasondhi]

Added tests.

Re: lock wait time rendering, did some digging and it turns out when it is off for the diagnostics bundle, it is also off in the otel export to Jaeger. Interestingly, the same information is rendered correctly in the logs attached to these traces. I am still trying to narrow down where/how the wait time gets corrupted for lazy tags. My initial plan was to try and useTestContentionEventTracer to consistently reproduce this, but haven't had any luck yet. Would you have any ideas on what to try or where to look?

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)

---

pkg/util/tracing/span_inner.go line 107 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

Scratch this comment. Replace it with // Serialize the lazy tags.

If it stays, it needs a period, 80-col wrapping (ask me about an golang plugin if you don't have it) and it needs to be clear who the "it" is in "it will be cleaned up".

Replaced with the suggested comment

---

pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

let's make this a method on s.crdb that returns []atribute.KeyValue. I think you've copied this code from somewhere, right? It'd probably be good to share.

Done. Even though the switch structure is similar to the code in getRecordingNoChildrenLocked, it does slightly different things. Here we want to generate a flattened []atribute.KeyValue while there we are generating tag groups, and directly adding tags to the span.

[andreimatei]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @aadityasondhi and @andreimatei)

---

pkg/util/tracing/crdbspan.go line 973 at r2 (raw file):

// It flattens lazy tags with children and attaches the parent's
// key as a prefix on each child.
func (s *crdbSpan) getLazyTagsKvLocked() []attribute.KeyValue {

KV in the function name - we generally capitalize all acronyms

---

pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

Previously, aadityasondhi (Aaditya Sondhi) wrote…

Done. Even though the switch structure is similar to the code in getRecordingNoChildrenLocked, it does slightly different things. Here we want to generate a flattened []atribute.KeyValue while there we are generating tag groups, and directly adding tags to the span.

Consider making the method return []tracingpb.TagGroup, and then processing the tag groups here to turn them into what otel wants.
The part I'd like shared with getRecordingNoChildrenLocked is recognizing Stringers vs LaxyTags.

[aadityasondhi]

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @andreimatei)

---

pkg/util/tracing/crdbspan.go line 973 at r2 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

KV in the function name - we generally capitalize all acronyms

Renamed the function as a part of the latest change.

---

pkg/util/tracing/span_inner.go line 110 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

Consider making the method return []tracingpb.TagGroup, and then processing the tag groups here to turn them into what otel wants.
The part I'd like shared with getRecordingNoChildrenLocked is recognizing Stringers vs LaxyTags.

Done.

[aadityasondhi]

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)