Add data quality metric to measure traces without a root #3812
Conversation
mapno
requested review from
joe-elliott,
annanay25,
mdisibio,
yvrhdn,
zalegrala,
electron0zero,
ie-pham and
stoewer
as code owners
| @@ -27,3 +28,7 @@ func WarnOutsideIngestionSlack(tenant string) { | |||
| func WarnDisconnectedTrace(tenant string, phase string) { | |||
| metric.WithLabelValues(tenant, reasonDisconnectedTrace+phase).Inc() | |||
| } | |||
|
|
|||
| func WarnRootlessTrace(tenant string, phase string) { | |||
There was a problem hiding this comment.
doesn't disconnected_trace_compactor_combine already cover this?
does the generator publish disconnected_trace_flushed_to_wal and _wal_to_complete. that would tell us a lot about the quality of traces produced by the generator
There was a problem hiding this comment.
Yes, rootless traces would be a subset of disconnected traces. Maybe it makes more sense as an extension of that warning?
There was a problem hiding this comment.
I'm good on this PR and will approve, but I am struggling to think what it would mean to a Tempo operator to have the majority of their disconnected traces rootless or rooted. Would they investigate data quality issues differently?
That doesn't mean it doesn't have a use and we might be quite happy we added this at some point in the future.
There was a problem hiding this comment.
I see it as a particularly bad type of disconnected trace—usually higher in the trace == more important. I think it just adds more information to the operator to investigate.
What this PR does:
Adds new reason
rootless_traceto data quality metric warnings.Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]