[receiver/datadog] add datadog span and trace id

[povilasv]

Description:

Add datadog trace id and span id so we cancorrelate traces received by datadogreceiver with logs.

Datadog libraries automatically provide datadog span ids and trace ids with the logs for various languages:

• https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/python/
  -https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/java?tab=log4j2

Example log for python:

2023-06-05 06:01:30,459 INFO [__main__] [test-enhanced.py:44] [dd.service= dd.env= dd.version= dd.trace_id=6249785623524942554 dd.span_id=2
28114450199004348] - Starting a new Game!

Link to tracking Issue: #23057
Testing:

• local testing with python app
• updated unit tests

Documentation:

[boostchicken]

I would be nice to have these strings in an enum like semconv. Better yet, making a proposal to get them in semconv. The actual spec shouldn't have DD specific stuff, but I don't think it would be a problem to add in the code

[povilasv]

Added some enums -> https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/23058/files#diff-da29a5fa6d38a3d168aff52f1f706fa53830e898975a082dfa6f3b5dccb88218R26-R37

What do you think? :)

[MovieStoreGuy]

Can I ask what are you trying to visualise this data in? Is there any reason why you can't use the trace ID and span ID that is already sent?

I don't believe adding span field to an attribute is a good idea, it is wasteful since it is duplicating data.

[povilasv]

Can I ask what are you trying to visualise this data in? Is there any reason why you can't use the trace ID and span ID that is already sent?

I don't believe adding span field to an attribute is a good idea, it is wasteful since it is duplicating data.

@MovieStoreGuy thanks for the review!

I sort of agree, but I see this is a temporary way of correlating logs and traces:

Since datadog library give log in this format:

2023-06-05 06:01:30,459 INFO [main] [test-enhanced.py:44] [dd.service= dd.env= dd.version= dd.trace_id=6249785623524942554 dd.span_id=2
28114450199004348] - Starting a new Game!

And atm I'm not sure how to parse this trace id and span id to use for correlation in opentelemetry?

I know otlp logs have dedicated field for SpanId and TraceID. Should we instead use some processor to transform logs in this format?

But I'm not sure where the best place to contribute parsing from datadog trace id to otel trace id would be?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[jpkrohling]

Since datadog library give log in this format:

Aren't those trace IDs and span IDs the same as already set a few lines above?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[jpkrohling]

ping @povilasv

[povilasv]

Hey @jpkrohling .Ah sorry forgot to reply, so coming back to the problem. Datadog instrumentation gives trace ids in this format:

2023-06-05 06:01:30,459 INFO [__main__] [test-enhanced.py:44] [dd.service= dd.env= dd.version= dd.trace_id=6249785623524942554 dd.span_id=2
28114450199004348] - Starting a new Game!

This receiver encodes DD TraceID's using:

newSpan.SetTraceID(uInt64ToTraceID(0, span.TraceID))


func uInt64ToTraceID(high, low uint64) pcommon.TraceID {
	traceID := [16]byte{}
	binary.BigEndian.PutUint64(traceID[:8], high)
	binary.BigEndian.PutUint64(traceID[8:], low)
	return traceID
}

Basically binary encoding.

So question - how do we correlate the binray encoded trace ID with raw DD trace id from the log?

I don't think in log storage systems it's common to have such decoding functions. But I might be wrong here.

So it would be ideal we would have same exact representation, so we can easily switch from log's trace id to span.

[jpkrohling]

Understood, and it makes sense to me then!

[jpkrohling]

@boostchicken , @MovieStoreGuy , can you take another look at this one?

[boostchicken]

@boostchicken , @MovieStoreGuy , can you take another look at this one?

Sure thing

[MovieStoreGuy]

I understand why this is needed but ideally I would like to see this work be done by a connector in a more generic sense

[povilasv]

Hey @boostchicken, would appreciate another look :)

[jpkrohling]

I'm merging this as it's only about adding two new attributes. If @boostchicken has concerns, we can revert or work on this on a follow-up PR.

[boostchicken]

No major concerns fine as is, my feedback was more a matter of opinion than best practice :)

[clintonb]

I recognize I'm late to the party, but what was the goal of this change? It makes sense to add the dd.trace_id and dd.span_id attributes to logs. What's the point of adding them to traces and spans? What is using these datadog.* attributes?