Jaeger all-in-one complaints about missing parent spans

[bobrik]

Describe the bug

I have an issue when jaeger-all-in-one incorrectly complains about missing parent spans:

Here's an example:

{
    "data": [
        {
            "traceID": "ef5dd64a1aa62bae",
            "spans": [
                {
                    "traceID": "ef5dd64a1aa62bae",
                    "spanID": "3e911c0058d13ebb",
                    "flags": 1,
                    "operationName": "trace_me",
                    "references": [
                        {
                            "refType": "CHILD_OF",
                            "traceID": "ef5dd64a1aa62bae",
                            "spanID": "ce8562a5c9781762"
                        }
                    ],
                    "startTime": 1610144303993197,
                    "duration": 30,
                    "tags": [
                        {
                            "key": "internal.span.format",
                            "type": "string",
                            "value": "proto"
                        }
                    ],
                    "logs": [],
                    "processID": "p1",
                    "warnings": [
                        "invalid parent span IDs=ce8562a5c9781762; skipping clock skew adjustment"
                    ]
                },
                {
                    "traceID": "ef5dd64a1aa62bae",
                    "spanID": "ce8562a5c9781762",
                    "flags": 1,
                    "operationName": "top_trace",
                    "references": [],
                    "startTime": 1610144303993035,
                    "duration": 4798745,
                    "tags": [
                        {
                            "key": "sampler.type",
                            "type": "string",
                            "value": "const"
                        },
                        {
                            "key": "sampler.param",
                            "type": "bool",
                            "value": true
                        },
                        {
                            "key": "internal.span.format",
                            "type": "string",
                            "value": "proto"
                        }
                    ],
                    "logs": [],
                    "processID": "p1",
                    "warnings": null
                }
            ],
            "processes": {
                "p1": {
                    "serviceName": "salt",
                    "tags": [
                        {
                            "key": "hostname",
                            "type": "string",
                            "value": "rockpro64"
                        },
                        {
                            "key": "ip",
                            "type": "string",
                            "value": "192.168.1.168"
                        },
                        {
                            "key": "jaeger.version",
                            "type": "string",
                            "value": "Python-4.4.0"
                        }
                    ]
                }
            },
            "warnings": null
        }
    ],
    "total": 0,
    "limit": 0,
    "offset": 0,
    "errors": null
}

The error says "invalid parent span IDs=ce8562a5c9781762", but ce8562a5c9781762 is right there in the JSON.

To Reproduce

Steps to reproduce the behavior:

1. Run jaeger all-in-one
2. Start producing a trace
3. Open the trace before it's full produced
4. Open the trace after it's fully produced
5. Observe lingering warnings about missing parents

Expected behavior

No complaints about a missing parent when parent is not in fact missing.

Screenshots

The screenshot corresponds to the first JSON snippet in this issue:

Version (please complete the following information):

• OS: Linux 5.11-rc2
• Jaeger version: 1.21.1
• Deployment: Bare metal arm64

Additional context

The reason this happens seems to be because ClockSkew adjuster runs on real in-memory spans from memory storage engine, as opposed to copies of those spans. So if you happen to request the trace while it's not fully built and the parent is not yet flushed, you'll get a warning for every load. Those warnings are never cleared.

It's possible to get multiple warnings too:

                {
                    "traceID": "eae14db35410faa8",
                    "spanID": "ac4b22f86a3ff0e1",
                    "flags": 1,
                    "operationName": "trace_me",
                    "references": [
                        {
                            "refType": "CHILD_OF",
                            "traceID": "eae14db35410faa8",
                            "spanID": "1da4278e72b02ea1"
                        }
                    ],
                    "startTime": 1610145703000068,
                    "duration": 30,
                    "tags": [
                        {
                            "key": "internal.span.format",
                            "type": "string",
                            "value": "proto"
                        }
                    ],
                    "logs": [],
                    "processID": "p1",
                    "warnings": [
                        "invalid parent span IDs=1da4278e72b02ea1; skipping clock skew adjustment",
                        "invalid parent span IDs=1da4278e72b02ea1; skipping clock skew adjustment"
                    ]
                }

[yurishkuro]

good catch, the memory storage should return cloned data so that the caller is free to do whatever with it, similar to how remote storage backends work