OpenTelemetry: InvalidTraceId due to date way in the past

[namnd]

There seems to be an error when converting the agent's tracing data to Xray format
Sample trace

Logging output

buildkite-agent-collector-1  | ScopeSpans #0
buildkite-agent-collector-1  | ScopeSpans SchemaURL: 
buildkite-agent-collector-1  | InstrumentationScope buildkite-agent 
buildkite-agent-collector-1  | Span #0
buildkite-agent-collector-1  |     Trace ID       : 470965deabdd7b05e92bab26e2c9c2c0
buildkite-agent-collector-1  |     Parent ID      : ba4b42ec09e4a565
buildkite-agent-collector-1  |     ID             : f228bdc87426ef9f
buildkite-agent-collector-1  |     Name           : agent environment hook
buildkite-agent-collector-1  |     Kind           : Internal
buildkite-agent-collector-1  |     Start time     : 2023-10-24 00:39:55.217447464 +0000 UTC
buildkite-agent-collector-1  |     End time       : 2023-10-24 00:39:55.668894007 +0000 UTC
buildkite-agent-collector-1  |     Status code    : Unset
buildkite-agent-collector-1  |     Status message : 
buildkite-agent-collector-1  | Attributes:
buildkite-agent-collector-1  |      -> analytics.event: Str(true)
buildkite-agent-collector-1  |      -> hook.type: Str(agent)
buildkite-agent-collector-1  |      -> hook.name: Str(environment)
buildkite-agent-collector-1  |      -> hook.command: Str(/etc/buildkite-agent/hooks/environment)

Xray exporter format

{
  "name": "agent environment hook",
  "id": "f228bdc87426ef9f",
  "start_time": 1698107995.2174475,
  "service": {
    "version": "3.56.0"
  },
  "trace_id": "1-470965de-abdd7b05e92bab26e2c9c2c0",
  "end_time": 1698107995.668894,
  "fault": false,
  "error": false,
  "throttle": false,
  "aws": {
    "xray": {
      "auto_instrumentation": false
    }
  },
  "metadata": {
    "default": {
      "analytics.event": "true",
      "hook.command": "/etc/buildkite-agent/hooks/environment",
      "hook.name": "environment",
      "hook.type": "agent"
    }
  },
  "parent_id": "ba4b42ec09e4a565",
  "type": "subsegment"
}

Error response from xray api

{
      ErrorCode: "InvalidTraceId",
       Id: "f228bdc87426ef9f",
       Message: "Invalid segment. ErrorCode: InvalidTraceId"
}

The 8 digit hexadecimal value representing the time of the original request 470965de seem to be way in the past (7 October 2007)

[triarius]

Hi @namnd, we're using open telemetry to generate these traces. There seem to be some issue with xray compatibility with otel: open-telemetry/opentelemetry-collector-contrib#24389

It may be possible to work around this by configuring the tracer provider to generate timestamped trace ids. We would welcome such a contribution.

[namnd]

Thanks @triarius . This issue has now been resolved aws-observability/aws-otel-collector#492 (comment)