[APM - OpenTelemetry] Span Events and Logs support: enhancement requests

[cyrille-leclerc]

Kibana version

8.0-rc1

APM Server version (if applicable)

8.0-rc1

Elasticsearch version (if applicable)

8.0-rc1

Steps to Reproduce

• Use a SpringBoot App that produces span events and logs through logback.
• Instrument with the Otel Java Agent 1.10
• Example: https://github.com/cyrille-leclerc/my-shopping-cart/tree/open-telemetry

Expected Behavior

Enhancement Requests on the Trace Logs Tab

Decreasing priority order priority

• Display attributes of span events
  • Attributes are displayed labels.* key value pairs in the log details flyout - this means it would be a short-term solution to look at integrating the log details flyout experience into the embeddable
    • [APM Service Logs] Show flyout in LogsEmbeddable #111325
  • If the log message contains some labels.* or numeric_labels.* fields, then we should display them in the log “message column”. If the “log.message” is quite long, then we can assume that the fields are less useful to be visualized on the default view. (Design issue: https://github.com/elastic/observability-design/issues/160)
• Link to show log message details (the full json document), probably redirecting to Kibana Discover or opening a flyout.
  • [APM] Add "View in Logs" button by the Logstream component in Trace detail #85915
• Add log.level column
• Timestamp column should be collapsible
• Service.name column should be collapsible into a color code similarly to what we have in the timeline view (related [APM] Discuss: Configure columns in the Logstream component used in the Trace detail view #85947)
• Replace the computed "Message" column by the "log.message" column
• ❓ should we enable/disable showing span events ❓

Actual Behavior

• Current Trace Timeline tab
  
• Current Trace Metadata tab

• Current Trace Logs tab
  

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[elasticmachine]

Pinging @elastic/unified-observability (Team:Unified observability)

[formgeist]

@cyrille-leclerc Thanks for creating this issue. I noticed there's a mix of APM and Logs embeddable enhancements, so I've added the @elastic/unified-observability team to the ticket as well as we'll probably need to extend the capabilities of this component as well.

I have some follow up questions;

Display attributes of span events

I'm not sure I understand what the expected experience will be? What do these span events look like and are they supposed to go in the Logs stream or the Trace waterfall/timeline?

Link to show log message details (the full json document), probably redirecting to Kibana Discover or opening a flyout.

There's an open issue already for adding the "View details" functionality to the Logs component - I assume this is similar? #111325 I think the only thing missing is the ability to View document in Discover.

Timestamp column should be collapsible

This is probably best achieved with a control option to show/hide the timestamp column in the embeddable. Collapsing columns are not supported in the Logs component at the moment.

Service.name column should be collapsible into a color code similarly to what we have in the timeline view

Yes, we have an open issue already for adding the service.name column and with an enhanced legend related to the Waterfall/Timeline #85947

Replace the computed "Message" column by the "log.message" column

I'm not familiar with the difference between the two - can you explain? I would expect message to be log.message.

[cyrille-leclerc]

Display attributes of span events
I'm not sure I understand what the expected experience will be? What do these span events look like and are they supposed to go in the Logs stream or the Trace waterfall/timeline?

The span event API looks like:

span.addEvent("order-creation", Attributes.of(
    "customer_id", 123456,
    "order_price", 654,
    "payment_method", "visa
    "shipping_method", "express",
    "shipping_country", "FR"));

Visualization today is

order-creation

Proposed style of visualization

(e) order-creation: customer_id=123456, order_price=654, payment_method=visa, shipping_method=express, shipping_country=FR

Where (e) could be an icon indicating it's a span event rather than an unstructured log message

See example of Jaeger visualization of span events

Replace the computed "Message" column by the "log.message" column

I'm not familiar with the difference between the two - can you explain? I would expect message to be log.message.

Messagewith upper case M is the combination of multiple fields, on the screenshot it is [${log.level}] ${log.message}]

Link to show log message details (the full json document), probably redirecting to Kibana Discover or opening a flyout.

There's an open issue already for adding the "View details" functionality to the Logs component - I assume this is similar? #111325 I think the only thing missing is the ability to View document in Discover.

LGTM

[formgeist]

@cyrille-leclerc Thanks for added details. I've added this issue to our design board for planning 👍

[jasonrhodes]

@elastic/infra-monitoring-ui we will want to look at this from the Logs UI logic side, working with @elastic/unified-observability to figure out how this will be used more generally

[axw]

Where (e) could be an icon indicating it's a span event rather than an unstructured log message

There's currently nothing in the documents to distinguish span events; "span event" isn't something that exists in our model -- they're just log records.

"... rather than an unstructured log message" implies that everything that's not a span event is unstructured. That's not the case, there are plenty of structured logging libraries out there. Do they come under the same category as span events? How do we distinguish them?

[formgeist]

After conversation with @Mpdreamz @axw and @cyrille-leclerc about how to best visualize these span events and their attributes, we decided on exploring the following concept;

If the log message contains some labels.* or numeric_labels.* fields, then we should display them in the log “message column”. If the “log.message” is quite long, then we can assume that the fields are less useful to be visualized on the default view.

We will prioritize exploring how to enhance the existing log message rows in the Logs UI to support this.

[jasonrhodes]

It sounds like some of this is possible today with the existing Log Stream component in use. I'm going to un-assign this from our board while APM figures out what exactly it would like to do here, and if there are any features needed with the log stream, we will open a ticket to address them. Please reach out for anything you need!

[rodrigc]

@jasonrhodes I filed https://support.elastic.co/cases/5008X00002Ly2fJQAR with Elastic, and they opened
Enhancement Request ER-18465 on this topic.

If you have access to track that ER inside Elastic, please let me know.

[felixbarny]

Enhancements should probably be made in the Logs Explorer embeddable rather than the Logs UI embeddable. I think that is already better at displaying structured events.

cc @ruflin