Broken span hierarchy when using opentelemetry java agent

[buttasam]

Environment Details

• Helidon Version: 3.1.1
• Helidon MP
• JDK version: 17.0.5
• OS: macOS 13.1
• Docker version (if applicable):

---

Problem Description

Hi team,
we're fully migrating our codebase from opentracing to opentelemetry and decided to use open telemetry java agent. This agent should be a pretty standard and convenient way how to handle tracing because we want to get rid of some third-party dependencies related to opentracing such as opentracing-contrib/java-kafka-client and we're looking for a replacement.

I created two very basic microservices based on Helidon, where the first one (helidon-mp1) calls the second one (helidon-mp2), to see whether the agent works.
The agent seems to be working, but traces look different from traces provided by Helidon without the agent.

I have attached this project to this issue. The project contains README.md where can be found steps to reproduce.

The main difference is that traces with java agent have a different hierarchy of spans, see these examples:

1. Span security:atn should be a child of security as you can see on the first image (without agent) but with agent, the spans are on the same level.

Expected (without java agent):

Actual (with java agent):

2. When microservice (helidon-mp1) creates custom span (custom-mp1-span-1) which creates another nested span (custom-mp1-span-2) the hierarchy seems broken similarly as well.

Expected (without java agent):

Actual (with java agent):

Additional version info:

• Opentelemetry java agent: 1.22.1
• Jaeger: 1.41

Steps to reproduce

Attached zip file with REAMDE

[trentjeff]

Note that I asked Samuel to file an issue on this - minimally we may need a documentation issue around this, but it may be more than that since the parenting of the span seems off in the above example.

[tjquinno]

Possibly another manifestation of the same issue (no agent involved) from the public #helidon-users slack channel

I have attached the user-provided reproducer. See the slack thread for details.

helidon-template-tracing.zip

[bogedal]

@dalexandrov @tjquinno
Any timeline for this issue?
It currently blocks me from embracing Helidon 3.

[dalexandrov]

About to be fixed. PR will be submitted soon. Doing some final tests.