Celery contrib: read custom parent span id from the task headers

[namoshizun]

What does this pull request do?

In a complex Celery workflow, tasks might be invoked in a chain like this:

<A> ---> <B> ---> <C> ... --> <Z>

In such setting, because the current Celery instrument only propagates the first caller's traceparent, all downstream tasks share the same traceparent from A. This might be undesirable if every task should be regarded as a subtask of its caller.

For instance, "submit_task_for_computation" initiates "do_heavy_lifting" and waits for completion so that it can immediately rollback a database transaction if the job failed. But instead of correlate do_heavy_lifting with submit_task_for_computation, current instrument produces the following trace:

Whereas we actually hope for this:

This PR is to allow a caller task to set the receiver task's parent span by passing a parent_span_id in the Celery headers.

I have prepared this snippet as a demo.

[cla-checker-service]

💚 CLA has been signed

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-29T18:04:21.125+0000

• Duration: 20 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4773
Skipped	3209
Total	7982

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /test linters : Run the Python linters only.

• /test full : Run the full matrix of tests.

• /test benchmark : Run the APM Agent Python benchmarks tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[basepi]

Looks great! We're fixing a CI failure caused by upstream changes in another library, and then we should be able to get the tests run here and merge this.

Can you add a note to the CHANGELOG?

[basepi]

/test

[basepi]

run elasticsearch-ci/docs