[Discuss] Progress, partial results and background search capabilities in the expression language and interpreter

[stacey-gammon]

We've thought about how to implement progress, partial results and background search in the search apis but we don't yet know how we would incorporate them into the expression language. The biggest question is:

How can we cause the same input to a dashboard container cause every child embeddable to change the way they query from making a live query to re-using a background query?

There are many layers between dashboards/embeddable containers and the search api so we have to consider how this will work for all different types of Embeddables.

There are many considerations. We may be able to work around some of them, but we've discussed:

• How to do this through the expression language (if we need to).
• How to do this through the embeddable layer.
• Security (is data saved in kibana, are search ids saved in kibana, how to link this per user)
• How to retrieve the results
• How to render the results so they use the saved query version, and don't "live query"
• Must solve: rendering both versions of embeddables, which means the same expression will also need to do two different things, or we need two different expressions.
• We also need to detect this "mode" so if the user wants to filter or change the time it flips back into "live query" mode.
  • Or maybe not if we do reporting... how important is this? Even scheduled reports is not an implemented feature though.

Many ideas have been tossed around, some which only focus on one of the above considerations:

Two versions of every expression string

We expose (or plan to expose) this expression string to the user so they can modify it. We don't want to encapsulate the logic of "live query" vs "background query" inside the language. This option would quickly grow out of scope. However, for the sake of completeness, this flow would be something like (only focusing on a sample Expression Embeddable to simplify things):

• render expression embeddable. input is { expression: "esaggs query='foo' | debug" }
• dashboard (somehow 👋) detects a query went past the timeout, asks user if they wish to send to background. If user says yes, dashboard loops through embeddable children, says something like child.sendToBackground(). Child modifies it's input state (which will get saved on the dashboard) to be something like explicitInput: { expression: "bgEsaggs searchId=xyz" }. Next time dashboard is loaded all the panels should be in a version that allows them to use a bg search.

Lots of issues with this version:

• Need two versions of every embeddable and expression - one that says use "live query" one that says re-use "bg query". The expression especially is shown to the user, we don't want to encapsulate this logic in there.
• Doesn't solve partial execution of interpreter.
• Still very handwavvy.

Saving partial state plus the ability to dynamically change input context.

The state of each partially completed expression would be saved after each function completes. When a "send to background" signal is received, the interpreter would need tell each function to pause(abort?) and each function would return the context, this is where it would have an opportunity to swap out query/filter/time with search id. This partial state of the expression would have to be stored in a saved object.

Notes:

• Con: Expression fns dictate input and output shape, I think changing this would be a huge and risky effort.
• Pro: Expression string can remain unchanged.
• Still a lot of unknowns.
• Doesn't solve the issue with interpreter completing execution of the expression when partial results are returned. Unless we could combine this with changing the expression fns to return observables instead of promises??

Re-using shareable embeddables

I talk about this option here. There are still a lot of open questions with that. This project is also higher priority than creating shareable embeddables so it may not be worth it to couple the two together, even if in the end it'd be less work. Still all the open questions about the expression too.

Search batch Service

Discussed in the link above, but summary is to have a search batcher service that exposes a search fn that looks like the normal search fn, but searches are first intercepted, so they can be stored, de-duped (look for a request that matches that has already been sent out, use that one...), and expose group search functionality, like aborting or sending to background.

Other ways to solve the initial problem, so we don't need to solve this one?

If we thought of other ways to solve the issue with long queries, we could possibly do something like incorporate pdf reports and not implement

cc

@alexh97 @lukasolson @lukeelmers - Trying to brain dump options we've thought about already and options we are still considering.

Plans for POC include:

• Expression embeddable
• All in new platform (if possible)
• async fake demo strategy
• register a new expression fn that just calls fake demo strategy
• Move debug expression fn into NP

POC does not include:

• partial results in the expression. We think this can be solved by having expresison fns return observables. Still a big project, but not as important as ability to close browser and come back to the pending dashboard.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[stacey-gammon]

Currently plan is to proceed with #53335. Showing partial progress is not as important but is tracked here: #53336