[Remote Store] Adding Segment download stats to remotestore stats API #8440

shourya035 · 2023-07-05T09:08:30Z

Description

Adding segment download stats to the _remotestore/stats API
Refactored API output to accommodate both segment and translog stats

Sample API outputs from running in local dev setup

RemoteStore Stats with all nodes up : https://gist.github.com/shourya035/69d1e217c8b5a854f4b652bb290c1e5e
RemoteStore stats with replica being promoted to primary : https://gist.github.com/shourya035/af3a26202d061b4f147ac84f7b805802
RemoteStore Stats with replica disabled : https://gist.github.com/shourya035/2302f53e8774d28488160f6d365670e0

Related Issues

Resolves #8395

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-05T09:30:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19117/
CommitID: 97b733a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-05T11:09:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19125/
CommitID: e667270
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-05T14:12:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19141/
CommitID: 743c09a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T13:55:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19259/
CommitID: 94f0f58
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T14:01:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19258/
CommitID: 94f0f58
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T14:10:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19261/
CommitID: ac3fbc0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T16:33:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19279/
CommitID: bb20d64
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T16:39:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19281/
CommitID: b42912f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T16:41:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19280/
CommitID: bb20d64
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

...ain/java/org/opensearch/action/admin/cluster/remotestore/stats/RemoteStoreStatsResponse.java

...a/org/opensearch/action/admin/cluster/remotestore/stats/TransportRemoteStoreStatsAction.java

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

ashking94

Is it possible to move the segments download to a different java class. That would not bloat the IndexShard class more.

server/src/main/java/org/opensearch/index/remote/RemoteRefreshSegmentPressureService.java

server/src/main/java/org/opensearch/index/remote/RemoteRefreshSegmentPressureSettings.java

server/src/main/java/org/opensearch/index/remote/RemoteRefreshSegmentTracker.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

ashking94 · 2023-07-10T16:34:18Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

+        if (!indexSettings.isRemoteStoreEnabled()) {
+            return;


Why is this check required? The concerned code only runs for remote store enabled indexes.

As discussed, the same copySegmentFiles method is being used during snapshot restores from remote store also. This is to handle any indices during snapshot restore which has an overriden index setting to disable RemoteStore. Will add a comment here regarding this.

ashking94 · 2023-07-10T16:35:10Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

+        if (!indexSettings.isRemoteStoreEnabled()) {
+            return;
+        }
+        downloadStatsTracker.incrementTotalDownloadsStarted();


This is used to track overall downloads and not file level. Lets fix this.

downloadStatsTracker.incrementTotalDownloadsStarted(); should be called only once per invocation of syncSegmentsFromRemoteSegmentStore.

server/src/main/java/org/opensearch/index/shard/IndexShard.java

ashking94 · 2023-07-10T16:42:58Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

+            return;
+        }
+        long currentTimeInNs = System.nanoTime();
+        downloadStatsTracker.updateLastDownloadTimestampMs(System.currentTimeMillis());


Do we need this file level or per sync level?

ashking94

Is it possible to move the segments download to a different java class. That would not bloat the IndexShard class more.

ashking94 · 2023-07-10T16:48:30Z

Also, please fix the DCO check - https://github.com/opensearch-project/OpenSearch/pull/8440/checks?check_run_id=14908813252

psychbot · 2023-07-10T16:49:21Z

...er/src/main/java/org/opensearch/action/admin/cluster/remotestore/stats/RemoteStoreStats.java

+            builder.endObject();
+            builder.endObject();
+        } else {
+            builder.startObject(SubFields.DOWNLOAD);


So for primary shards we will show download field as {}.
Can we validate and put some of the existing API which does the same. We just need to make sure that user is well aware that this stats will be empty in case of primary shard and this should not lead to any issues in future.

This is inline with the other APIs also, wherein null or 0 values are being shown as a simple {}. The shard routing section of the _cluster/state is an instance

...er/src/main/java/org/opensearch/action/admin/cluster/remotestore/stats/RemoteStoreStats.java

psychbot · 2023-07-10T16:53:49Z

...er/src/main/java/org/opensearch/action/admin/cluster/remotestore/stats/RemoteStoreStats.java

+                .field(SubFields.FAILED, remoteSegmentShardStats.downloadBytesFailed);
+            builder.endObject();
+            builder.startObject(DownloadStatsFields.DOWNLOAD_SIZE_IN_BYTES)
+                .field(SubFields.LAST_SUCCESSFUL, remoteSegmentShardStats.lastSuccessfulSegmentDownloadBytes)


Is there a specific reason to have last successfully downloaded segment file size? How does this benefit user from stats perspective?

psychbot · 2023-07-10T16:57:37Z

...er/src/main/java/org/opensearch/action/admin/cluster/remotestore/stats/RemoteStoreStats.java

+                    remoteSegmentShardStats.localRefreshNumber - remoteSegmentShardStats.remoteRefreshNumber
+                )
+                .field(UploadStatsFields.BYTES_LAG, remoteSegmentShardStats.bytesLag)
+                .field(UploadStatsFields.BACKPRESSURE_REJECTION_COUNT, remoteSegmentShardStats.rejectionCount)


Question : Will there be a scenario where we will have back pressure during download?
Is this something that we need or are we good, thinking from future perspective where n numbers of segments are getting restored or recovered which might lead to some delay or congestion.

As of now we are not putting any backpressure on downloads. IMO the indices.recovery.max_bytes_per_sec setting is already there to enforce a check on downloads.

...ain/java/org/opensearch/action/admin/cluster/remotestore/stats/RemoteStoreStatsResponse.java

server/src/main/java/org/opensearch/index/remote/RemoteRefreshSegmentTracker.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

github-actions · 2023-07-11T11:02:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19836/
CommitID: 6d49924
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-11T11:19:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19838/
CommitID: d826579
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-11T12:45:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19856/
CommitID: 2213186
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-11T13:24:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/19866/
CommitID: 14d8b2e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-11T13:44:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/19870/
CommitID: f282d54
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-11T18:40:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase

URL: https://build.ci.opensearch.org/job/gradle-check/19941/
CommitID: 331cb81
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-11T18:53:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.classMethod
      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testScrollWithConcurrentIndexAndSearch
      1 org.opensearch.remotestore.CreateRemoteIndexClusterDefaultDocRep.testRemoteStoreTranslogDisabledByUser

URL: https://build.ci.opensearch.org/job/gradle-check/19942/
CommitID: 331cb81
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-11T18:53:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19943/
CommitID: 8de6d00
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-11T20:10:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testPitCreatedOnReplica
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testPressureServiceStats

URL: https://build.ci.opensearch.org/job/gradle-check/19964/
CommitID: 9daeb74
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-13T11:44:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testPitCreatedOnReplica

URL: https://build.ci.opensearch.org/job/gradle-check/20107/
CommitID: 2f4545a
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

shourya035 marked this pull request as ready for review July 6, 2023 17:05

shourya035 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and VachaShah as code owners July 6, 2023 17:05