Support loading traces with large number of spans.

[harshsinh]

Requirement - what kind of business use case are you trying to solve?

We are trying to port our tracing infrastructure to Jaeger, but some of our workflows end up with traces that have upwards 40K spans. Trying to load these spans in the Jaeger UI doesn't work.

Problem - what in Jaeger blocks you from solving the requirement?

The fact that once, you open a trace all of the spans are loaded and are expanded. In the case of traces with a very large number of spans, Jaeger UI fails to load the traces.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Instead of loading all of the spans within a trace at once, load them only when they are clicked on, i.e. don't load or show any of the children, when one clicks on the span, then only fetch and show the children of the clicked span.
This can be done selectively for large spans only as well since that would lead to no behavior change in case of smaller traces, whereas great improvement in the case of massive ones.
(We have a similar solution in case of our current in-house tracing solution.)

cc: @annanay25

[annanay25]

Thanks for opening the issue @harshsinh.

To maintainers: A similar issue has been opened before but this is particularly interesting because of the proposed solution -

Instead of loading all of the spans within a trace at once, load them only when they are clicked on, i.e. don't load or show any of the children, when one clicks on the span, then only fetch and show the children of the clicked span.

Personally I liked this idea of supporting an API endpoint FindSpansWithParent(parentSpanID), but using this issue as a placeholder for further discussions/opinions.

[objectiser]

Its more efficient to just retrieve the whole trace and handle everything else locally within the UI. So if we wanted to handle large traces, it might be better if the backend can return a response that includes (a) the top x levels of the trace instance hierarchy, (b) the total number of spans, (c) ....

And then offer additional endpoint to retrieve further sub-trees - so not just one level at a time, but a manageable group?

[yurishkuro]

@harshsinh We have previously tested the UI with traces containing up to 80k spans. What do you mean exactly by "Trying to load these spans in the Jaeger UI doesn't work"?

[harshsinh]

@yurishkuro When we tried opening these large spans, the UI could never load them, and the browser (I tried both Firefox and Chrome, although I did have a bunch of other tabs open) couldn't open the trace page, instead, after a few seconds, the browser would suggest stopping the page for it's slowing the browser.

[yurishkuro]

It's possible some changes have been introduced that increased the processing time (like adding O(n^2) somewhere). It's worth running a profiler to see where the slowdown is coming from.