Request index not exist handling #169

penghuo · 2023-11-19T02:11:44Z

Description

In Flint.createIndex, create query_execution_request index if not exist
Check query_execution_request, before write to **query_execution_request **index, if index not exist, throw IllegalStateException, emit metrics QueryExecutionRequestIndexNotFound.
When write
- In translog state update, **query_execution_request **index, service not available / write block, throw IllegalStateException. Emit metrics QueryExecutionRequestIndexWriteFail
- heartbeat update, **query_execution_request **index, service not available / write block, catch exception, Emit metrics QueryExecutionRequestIndexWriteFail. After 5 times, Monitor system detect the job lastUpdateTime is stale. if index is green, Monitor system retry EMR-S job. if index is RED / write block. Monitor system should cut ticket.
When read
- In translog state init***, query_execution_request*** index, read block, throw IllegalStateException, Emit metrics QueryExecutionRequestIndexReadFail.
- heartbeat update, **query_execution_request **index, service not available / write block, catch exception, Emit metrics QueryExecutionRequestIndexReadFail. After 5 times, Monitor system detect the job lastUpdateTime is stale. if index is green, Monitor system retry EMR-S job. if index is read block, Monitor system should cut ticket.

Detail info in #166

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

flint-spark-integration/src/main/scala/org/opensearch/flint/spark/FlintSpark.scala

penghuo · 2023-11-19T19:38:53Z

flint-core/src/main/scala/org/opensearch/flint/core/storage/OpenSearchUpdater.java

@@ -28,6 +33,7 @@ public void upsert(String id, String doc) {
        // also, failure to close the client causes the job to be stuck in the running state as the client resource
        // is not released.
        try (RestHighLevelClient client = flintClient.createClient()) {
+            assertIndexExist(client, indexName);


@kaituo any other test cases we need to cover when write to query_execution_request index?

We may also need to check request indexing mapping as we did in the result index mapping. This can be done once during REPL lifetime.

Track in #172

flint-core/src/main/scala/org/opensearch/flint/core/storage/FlintOpenSearchClient.java

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo · 2023-11-21T00:36:56Z

spark-sql-application/src/main/scala/org/apache/spark/sql/FlintREPL.scala

@@ -46,7 +42,7 @@ object FlintREPL extends Logging with FlintJobExecutor {
  private val HEARTBEAT_INTERVAL_MILLIS = 60000L
  private val DEFAULT_INACTIVITY_LIMIT_MILLIS = 10 * 60 * 1000
  private val MAPPING_CHECK_TIMEOUT = Duration(1, MINUTES)
-  private val DEFAULT_QUERY_EXECUTION_TIMEOUT = Duration(10, MINUTES)
+  private val DEFAULT_QUERY_EXECUTION_TIMEOUT = Duration(30, MINUTES)


Increase query execution timeout to 30mins.

dai-chen · 2023-11-21T16:35:23Z

flint-core/src/main/scala/org/opensearch/flint/core/metadata/log/FlintMetadataLogEntry.scala

+      |    "lastUpdateTime": {
+      |      "type": "date",
+      |      "format": "strict_date_time||epoch_millis"
+      |    },


Previously this was Long type? And should we add jobStartTime here too?

aligned with https://github.com/opensearch-project/sql/blob/main/spark/src/main/resources/query_execution_request_mapping.yml

dai-chen · 2023-11-21T18:05:00Z

flint-core/src/main/scala/org/opensearch/flint/core/storage/OpenSearchUpdater.java

@@ -44,6 +48,7 @@ public void upsert(String id, String doc) {

    public void update(String id, String doc) {
        try (RestHighLevelClient client = flintClient.createClient()) {
+            assertIndexExist(client, indexName);


So this check is not required for Flint metadata log entry update?

I found I didn't set refresh policy in create/update doc in FlintOpenSearchMetadataLog. Could you help check if we should do .setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL) there as well?

checked in https://github.com/opensearch-project/opensearch-spark/pull/169/files#diff-22f1073d1ad43e334a608c35e6f0c05a31afe821194f8fa8e7dc20bb4ec4ebdfR63

kaituo · 2023-11-21T18:31:33Z

flint-core/src/main/scala/org/opensearch/flint/core/storage/FlintOpenSearchClient.java

+        } else {
+          String errorMsg = "Metadata log index not found " + metaLogIndexName;
+          LOG.warning(errorMsg);
+          throw new IllegalStateException(errorMsg);


where do you emit metrics QueryExecutionRequestIndexNotFound ?

will add after metrics sink ready.

track in here. #117

kaituo · 2023-11-21T18:43:13Z

flint-core/src/main/scala/org/opensearch/flint/core/storage/OpenSearchUpdater.java

@@ -28,6 +33,7 @@ public void upsert(String id, String doc) {
        // also, failure to close the client causes the job to be stuck in the running state as the client resource
        // is not released.
        try (RestHighLevelClient client = flintClient.createClient()) {
+            assertIndexExist(client, indexName);


We may also need to check request indexing mapping as we did in the result index mapping. This can be done once during REPL lifetime.

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo commented Nov 19, 2023

View reviewed changes

penghuo marked this pull request as ready for review November 20, 2023 16:01

penghuo requested review from dai-chen, rupal-bq, vmmusings, anirudha, kaituo and YANG-DB as code owners November 20, 2023 16:01

penghuo self-assigned this Nov 20, 2023

penghuo added 0.1.1 enhancement New feature or request labels Nov 20, 2023

penghuo marked this pull request as draft November 20, 2023 18:00

handle index not found and not avaiable exception

6055e1a

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo force-pushed the handleNotIndexFound branch from 4d77ba4 to 6055e1a Compare November 21, 2023 00:32

penghuo commented Nov 21, 2023

View reviewed changes

penghuo marked this pull request as ready for review November 21, 2023 01:39

dai-chen reviewed Nov 21, 2023

View reviewed changes

kaituo reviewed Nov 21, 2023

View reviewed changes

add refresh=wait_until when create metalog

a0a251d

Signed-off-by: Peng Huo <penghuo@gmail.com>

kaituo approved these changes Nov 21, 2023

View reviewed changes

penghuo merged commit e201f09 into opensearch-project:main Nov 21, 2023
4 checks passed

dai-chen mentioned this pull request Nov 27, 2023

[Feature] OpenSearch and Apache Spark Integration #3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Request index not exist handling #169

Request index not exist handling #169

penghuo commented Nov 19, 2023 •

edited

Loading

penghuo Nov 19, 2023

kaituo Nov 21, 2023

penghuo Nov 21, 2023

penghuo Nov 21, 2023

dai-chen Nov 21, 2023

penghuo Nov 21, 2023

dai-chen Nov 21, 2023

dai-chen Nov 21, 2023

penghuo Nov 21, 2023

kaituo Nov 21, 2023

penghuo Nov 21, 2023

penghuo Nov 21, 2023

kaituo Nov 21, 2023

Request index not exist handling #169

Request index not exist handling #169

Conversation

penghuo commented Nov 19, 2023 • edited Loading

Description

Issues Resolved

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

penghuo commented Nov 19, 2023 •

edited

Loading