ESQL: async search responses have CCS metadata while searches are running #117265

quux00 · 2024-11-21T15:31:36Z

ES|QL async search responses now include CCS metadata while the query is still running.
The CCS metadata will be present only if a remote cluster is queried and the user requested
it with the include_ccs_metadata: true setting on the original request to POST /_query/async.
The setting cannot be modified in the query to GET /_query/async/:id.

The core change is that the EsqlExecutionInfo object is set on the EsqlQueryTask, which is used
for async ES|QL queries, so that calls to GET /_query/async/:id have access to the same
EsqlExecutionInfo object that is being updated as the planning and query progress.

Example response showing in progress state of the query:

GET _query/async/FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA

{
  "id": "FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA==",
  "is_running": true,
  "columns": [],
  "values": [],
  "_clusters": {
    "total": 3,
    "successful": 1,
    "running": 2,
    "skipped": 0,
    "partial": 0,
    "failed": 0,
    "details": {
      "(local)": {
        "status": "running",
        "indices": "web_traffic",
        "_shards": {
          "total": 2,
          "skipped": 0
        }
      },
      "remote1": {
        "status": "running",
        "indices": "web_traffic"
      },
      "remote2": {
        "status": "successful",
        "indices": "web_traffic",
        "took": 180,
        "_shards": {
          "total": 2,
          "successful": 2,
          "skipped": 0,
          "failed": 0
        }
      }
    }
  }
}

…working for sync search; not yet for async

…sportEsqlQueryAction, so metadata is now shown in async response while search is running

…se we have a race condition)

…ter query, not after can-match. Coordinator only queries set shard counts before the compute phase in order to allow asserts to pass in ComputeListener that need to be enforced for data node based queries.

…usAndShardCounts, since skipped==successful Only setting took time in XContent when status != RUNNING, to avoid showing intermediate took times that get set as each node finishes. Adjusted tests to match new behavior - now passing: • EsqlQueryResponseTests.java • ComputeListenerTests.java

… pause plugin yet

…clean it up and ensure it is right

…ntermediate state async result has metadata

…include it in the initial response

…-have-progress-metadata

quux00 · 2024-11-26T15:35:12Z

buildkite test this

…-have-progress-metadata

elasticsearchmachine · 2024-11-26T15:47:22Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2024-11-26T15:47:23Z

Hi @quux00, I've created a changelog YAML for you.

.../plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/TransportEsqlQueryAction.java

smalyshev · 2024-11-26T22:17:02Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeListener.java

                }
            }
            delegate.onResponse(result);
        }, e -> delegate.onFailure(failureCollector.getFailure())));
    }

+    private static void setFinalStatusAndShardCounts(String clusterAlias, EsqlExecutionInfo executionInfo) {
+        executionInfo.swapCluster(clusterAlias, (k, v) -> {
+            if (v.getStatus() != EsqlExecutionInfo.Cluster.Status.SKIPPED) {


What about PARTIAL status here?

Good call out. Currently in the the code, there is no place where PARTIAL is ever set. Yours will be the first, so I put in a TODO to handle that once we start adding support for partial. I don't have any context or reason to add it right now.

OK I'll need to remember this and if this patch merges first (likely) to update mine for this case.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlExecutionInfo.java

…-have-progress-metadata

pawankartik-elastic · 2024-11-27T15:14:23Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeListener.java

@@ -112,6 +112,7 @@ private ComputeListener(
            if (runningOnRemoteCluster()) {
                // for remote executions - this ComputeResponse is created on the remote cluster/node and will be serialized and
                // received by the acquireCompute method callback on the coordinating cluster
+                setFinalStatusAndShardCounts(clusterAlias, executionInfo);


Line 105 states that clusterAlias and executionInfo should both be null or non-null. If I assume that they both are null, won't setFinalStatusAndShardCounts() be dereferencing null? Or am I missing something?

The two if-guards of runningOnRemoteCluster() and coordinatingClusterIsSearchedInCCS() ensure that they are not null. clusterAlias and executionInfo are both null when handling data node operations rather than coordinator level operations and CCS metadata is not tracked/updated during data node operations, only coordinator operations.

…arch is running ("took so far"). Addressed a few other minor aspects based on PR feedback.

…-have-progress-metadata

pawankartik-elastic

No more comments from my side; I've the required context.

elasticsearchmachine · 2024-11-27T21:16:11Z

💚 Backport successful

Status	Branch	Result
✅	8.x

…ning (elastic#117265) ES|QL async search responses now include CCS metadata while the query is still running. The CCS metadata will be present only if a remote cluster is queried and the user requested it with the `include_ccs_metadata: true` setting on the original request to `POST /_query/async`. The setting cannot be modified in the query to `GET /_query/async/:id`. The core change is that the EsqlExecutionInfo object is set on the EsqlQueryTask, which is used for async ES|QL queries, so that calls to `GET /_query/async/:id` have access to the same EsqlExecutionInfo object that is being updated as the planning and query progress. Secondly, the overall `took` time is now always present on ES|QL responses, even for async-searches while the query is still running. The took time shows a "took-so-far" value and will change upon refresh until the query has finished. This is present regardless of the `include_ccs_metadata` setting. Example response showing in progress state of the query: ``` GET _query/async/FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA ``` ```json { "id": "FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA==", "is_running": true, "took": 2032, "columns": [], "values": [], "_clusters": { "total": 3, "successful": 1, "running": 2, "skipped": 0, "partial": 0, "failed": 0, "details": { "(local)": { "status": "running", "indices": "web_traffic", "_shards": { "total": 2, "skipped": 0 } }, "remote1": { "status": "running", "indices": "web_traffic" }, "remote2": { "status": "successful", "indices": "web_traffic", "took": 180, "_shards": { "total": 2, "successful": 2, "skipped": 0, "failed": 0 } } } } } ```

…ning (#117265) (#117664) ES|QL async search responses now include CCS metadata while the query is still running. The CCS metadata will be present only if a remote cluster is queried and the user requested it with the `include_ccs_metadata: true` setting on the original request to `POST /_query/async`. The setting cannot be modified in the query to `GET /_query/async/:id`. The core change is that the EsqlExecutionInfo object is set on the EsqlQueryTask, which is used for async ES|QL queries, so that calls to `GET /_query/async/:id` have access to the same EsqlExecutionInfo object that is being updated as the planning and query progress. Secondly, the overall `took` time is now always present on ES|QL responses, even for async-searches while the query is still running. The took time shows a "took-so-far" value and will change upon refresh until the query has finished. This is present regardless of the `include_ccs_metadata` setting. Example response showing in progress state of the query: ``` GET _query/async/FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA ``` ```json { "id": "FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA==", "is_running": true, "took": 2032, "columns": [], "values": [], "_clusters": { "total": 3, "successful": 1, "running": 2, "skipped": 0, "partial": 0, "failed": 0, "details": { "(local)": { "status": "running", "indices": "web_traffic", "_shards": { "total": 2, "skipped": 0 } }, "remote1": { "status": "running", "indices": "web_traffic" }, "remote2": { "status": "successful", "indices": "web_traffic", "took": 180, "_shards": { "total": 2, "successful": 2, "skipped": 0, "failed": 0 } } } } } ```

…ning (elastic#117265) ES|QL async search responses now include CCS metadata while the query is still running. The CCS metadata will be present only if a remote cluster is queried and the user requested it with the `include_ccs_metadata: true` setting on the original request to `POST /_query/async`. The setting cannot be modified in the query to `GET /_query/async/:id`. The core change is that the EsqlExecutionInfo object is set on the EsqlQueryTask, which is used for async ES|QL queries, so that calls to `GET /_query/async/:id` have access to the same EsqlExecutionInfo object that is being updated as the planning and query progress. Secondly, the overall `took` time is now always present on ES|QL responses, even for async-searches while the query is still running. The took time shows a "took-so-far" value and will change upon refresh until the query has finished. This is present regardless of the `include_ccs_metadata` setting. Example response showing in progress state of the query: ``` GET _query/async/FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA ``` ```json { "id": "FlhaeTBxUU0yU2xhVzM2TlRLY3F1eXcceWlSWWZlRDhUVTJEUGFfZUROaDdtUTo0MDQwNA==", "is_running": true, "took": 2032, "columns": [], "values": [], "_clusters": { "total": 3, "successful": 1, "running": 2, "skipped": 0, "partial": 0, "failed": 0, "details": { "(local)": { "status": "running", "indices": "web_traffic", "_shards": { "total": 2, "skipped": 0 } }, "remote1": { "status": "running", "indices": "web_traffic" }, "remote2": { "status": "successful", "indices": "web_traffic", "took": 180, "_shards": { "total": 2, "successful": 2, "skipped": 0, "failed": 0 } } } } } ```

quux00 added >enhancement auto-backport Automatically create backport pull requests when merged v9.0.0 v8.18.0 labels Nov 21, 2024

quux00 force-pushed the esql/async-responses-have-progress-metadata branch 3 times, most recently from 065f5a5 to 47756d7 Compare November 25, 2024 17:45

quux00 added 11 commits November 25, 2024 16:30

Init commit: create EsqlExecutionInfo in doExecuteForked - should be …

ebcff8b

…working for sync search; not yet for async

Async search now has EsqlExecutionInfo - gets set on the task in Tran…

6d6f9a5

…sportEsqlQueryAction, so metadata is now shown in async response while search is running

Only showing took time in async-response if is_running=false (otherwi…

14f5600

…se we have a race condition)

Successful and failed shard counts are now set at the end of the clus…

df6c71d

…ter query, not after can-match. Coordinator only queries set shard counts before the compute phase in order to allow asserts to pass in ComputeListener that need to be enforced for data node based queries.

Started CrossClusterAsyncQueryIT but not yet async

9afa84f

mish mash on non-working Async tests

214622f

Init working version of CrossClusterAsyncQueryIT, but doesn't use the…

da7be3e

… pause plugin yet

I _think_ I have a working version of the Pause plugin - now need to …

15fed8f

…clean it up and ensure it is right

Working version of CrossClusterAsyncQueryIT using the PausePlugin - i…

c1af4a9

…ntermediate state async result has metadata

Added EsqlExecutionInfo to the async query EsqlQueryTask earlier and …

d2151b1

…include it in the initial response

quux00 force-pushed the esql/async-responses-have-progress-metadata branch from 47756d7 to d2151b1 Compare November 25, 2024 21:31

quux00 added 3 commits November 26, 2024 08:57

Merge remote-tracking branch 'elastic/main' into esql/async-responses…

8ce3354

…-have-progress-metadata

Minor cleanup for PR review

b466b42

Merge remote-tracking branch 'elastic/main' into esql/async-responses…

f6ef797

…-have-progress-metadata

quux00 added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels Nov 26, 2024

quux00 marked this pull request as ready for review November 26, 2024 15:46

Merge remote-tracking branch 'elastic/main' into esql/async-responses…

cbcd0f7

…-have-progress-metadata

Update docs/changelog/117265.yaml

516207a

quux00 requested review from smalyshev and pawankartik-elastic November 26, 2024 15:47

smalyshev reviewed Nov 26, 2024

View reviewed changes

Merge remote-tracking branch 'elastic/main' into esql/async-responses…

9e315fc

…-have-progress-metadata

pawankartik-elastic reviewed Nov 27, 2024

View reviewed changes

quux00 added 2 commits November 27, 2024 11:15

Overall took time is now present in intermediate responses while se…

398c011

…arch is running ("took so far"). Addressed a few other minor aspects based on PR feedback.

Merge remote-tracking branch 'elastic/main' into esql/async-responses…

2668492

…-have-progress-metadata

smalyshev approved these changes Nov 27, 2024

View reviewed changes

pawankartik-elastic approved these changes Nov 27, 2024

View reviewed changes

quux00 merged commit e33e1a0 into elastic:main Nov 27, 2024
16 checks passed

quux00 mentioned this pull request Nov 27, 2024

[8.x] ESQL: async search responses have CCS metadata while searches are running (#117265) #117664

Merged

quux00 mentioned this pull request Nov 28, 2024

Adjust EsqlQueryResponseTests to pass with the new took time in ES|QL async responses #117673

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESQL: async search responses have CCS metadata while searches are running #117265

ESQL: async search responses have CCS metadata while searches are running #117265

quux00 commented Nov 21, 2024 •

edited

Loading

quux00 commented Nov 26, 2024

elasticsearchmachine commented Nov 26, 2024

elasticsearchmachine commented Nov 26, 2024

smalyshev Nov 26, 2024

quux00 Nov 27, 2024

smalyshev Nov 27, 2024

pawankartik-elastic Nov 27, 2024

quux00 Nov 27, 2024

pawankartik-elastic left a comment

elasticsearchmachine commented Nov 27, 2024

ESQL: async search responses have CCS metadata while searches are running #117265

ESQL: async search responses have CCS metadata while searches are running #117265

Conversation

quux00 commented Nov 21, 2024 • edited Loading

quux00 commented Nov 26, 2024

elasticsearchmachine commented Nov 26, 2024

elasticsearchmachine commented Nov 26, 2024

smalyshev Nov 26, 2024

Choose a reason for hiding this comment

quux00 Nov 27, 2024

Choose a reason for hiding this comment

smalyshev Nov 27, 2024

Choose a reason for hiding this comment

pawankartik-elastic Nov 27, 2024

Choose a reason for hiding this comment

quux00 Nov 27, 2024

Choose a reason for hiding this comment

pawankartik-elastic left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Nov 27, 2024

💚 Backport successful

quux00 commented Nov 21, 2024 •

edited

Loading