Added support for search replica to return segrep stats #16678

vinaykpud · 2024-11-18T20:43:49Z

Description

In this PR we are adding,

Support for search replica to return its segrep stats
This doesn't change any of the existing implementation stats calculation for the regular replicas
For search replicas we are just calculating the stats using the difference between the computed checkpoint in the shard and latest pulled checkpoint from the remote store. So this stat will help to know if any problematic replications in search replica which is taking longer.

Related Issues

Resolves #15534
Related #15306

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… replica in shards Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-11-18T22:53:56Z

❌ Gradle check result for 39576c3: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-19T04:14:18Z

✅ Gradle check result for 39576c3: SUCCESS

codecov · 2024-11-19T04:14:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.14%. Comparing base (5ba909a) to head (70cb875).
Report is 6 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16678      +/-   ##
============================================
+ Coverage     72.11%   72.14%   +0.03%     
- Complexity    65237    65271      +34     
============================================
  Files          5318     5318              
  Lines        304003   304051      +48     
  Branches      43992    43990       -2     
============================================
+ Hits         219228   219372     +144     
+ Misses        66874    66702     -172     
- Partials      17901    17977      +76

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-11-22T20:51:15Z

❌ Gradle check result for 803c15b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-22T21:55:27Z

❌ Gradle check result for 803c15b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-11-22T23:07:46Z

✅ Gradle check result for c6e2c80: SUCCESS

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-11-25T23:30:48Z

❕ Gradle check result for d3a8a89: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-12-10T23:29:52Z

❌ Gradle check result for 6944ce6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-12-11T01:08:23Z

❌ Gradle check result for 6944ce6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-12-11T02:18:39Z

❌ Gradle check result for 6944ce6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-12-11T06:18:41Z

❌ Gradle check result for 046e26c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-12-12T00:36:42Z

✅ Gradle check result for e09dde8: SUCCESS

.../internalClusterTest/java/org/opensearch/indices/replication/SearchReplicaReplicationIT.java

server/src/main/java/org/opensearch/index/SegmentReplicationShardStats.java

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-12-12T22:03:12Z

❌ Gradle check result for 424ad09: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

stale

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions · 2024-12-12T23:35:55Z

❕ Gradle check result for 70cb875: UNSTABLE

TEST FAILURES:

      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

* Added implementation for the stats calculation for search and regular replica in shards Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Updated changelog Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Added unit tests for TransportSegmentReplicationStatsAction Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed java style after running precommit locally Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * refined the test cases Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed style issues Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Made changes in the bytes to download calculation based on comments Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * added addReplicaStats method to SegmentReplicationPerGroupStats Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed style issues Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed issue with immutable set Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed PR comments and moved the integration tests to separate module Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed failing integ tests Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed failing integ test Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed some comments for PR Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed failing tests Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> --------- Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit b67cdf4) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#16678) * Added implementation for the stats calculation for search and regular replica in shards Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Updated changelog Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Added unit tests for TransportSegmentReplicationStatsAction Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed java style after running precommit locally Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * refined the test cases Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed style issues Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Made changes in the bytes to download calculation based on comments Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * added addReplicaStats method to SegmentReplicationPerGroupStats Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed style issues Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed issue with immutable set Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed PR comments and moved the integration tests to separate module Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed failing integ tests Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * Fixed failing integ test Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed some comments for PR Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed failing tests Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> --------- Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> remove license Signed-off-by: Mingshi Liu <mingshl@amazon.com>

) * Added implementation for the stats calculation for search and regular replica in shards * Updated changelog * Added unit tests for TransportSegmentReplicationStatsAction * fixed java style after running precommit locally * refined the test cases * fixed style issues * Made changes in the bytes to download calculation based on comments * added addReplicaStats method to SegmentReplicationPerGroupStats * fixed style issues * Fixed issue with immutable set * Fixed PR comments and moved the integration tests to separate module * Fixed failing integ tests * Fixed failing integ test * fixed some comments for PR * fixed failing tests --------- (cherry picked from commit b67cdf4) Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Added implementation for the stats calculation for search and regular…

0acfe37

… replica in shards Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

github-actions bot added the v2.19.0 Issues and PRs related to version 2.19.0 label Nov 18, 2024

vinaykpud force-pushed the stats-search-replica branch from 765852d to 93e48ff Compare November 18, 2024 20:46

Updated changelog

39576c3

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

vinaykpud force-pushed the stats-search-replica branch from 93e48ff to 39576c3 Compare November 18, 2024 20:50

opensearch-ci-bot mentioned this pull request Nov 11, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreMultipartIT #15819

Open

vinaykpud closed this Nov 19, 2024

vinaykpud reopened this Nov 19, 2024

Added unit tests for TransportSegmentReplicationStatsAction

803c15b

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

vinaykpud closed this Nov 22, 2024

vinaykpud reopened this Nov 22, 2024

fixed java style after running precommit locally

c6e2c80

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

sandeshkr419 assigned vinaykpud Nov 25, 2024

vinaykpud added 2 commits November 25, 2024 14:15

refined the test cases

fcdcdda

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

fixed style issues

d3a8a89

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

vinaykpud marked this pull request as ready for review November 26, 2024 00:28

vinaykpud requested review from anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock and dbwiddis as code owners November 26, 2024 00:28

opensearch-ci-bot mentioned this pull request Dec 10, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteRestoreSnapshotIT #14324

Open

Fixed PR comments and moved the integration tests to separate module

6944ce6

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

vinaykpud closed this Dec 11, 2024

vinaykpud reopened this Dec 11, 2024

vinaykpud closed this Dec 11, 2024

vinaykpud reopened this Dec 11, 2024

Fixed failing integ tests

046e26c

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

Fixed failing integ test

e09dde8

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

mch2 reviewed Dec 12, 2024

View reviewed changes

.../internalClusterTest/java/org/opensearch/indices/replication/SearchReplicaReplicationIT.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/SegmentReplicationShardStats.java Outdated Show resolved Hide resolved

vinaykpud added 2 commits December 12, 2024 12:44

fixed some comments for PR

4352114

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

Merge branch 'main' into stats-search-replica

424ad09

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

mch2 previously approved these changes Dec 12, 2024

View reviewed changes

mch2 added the backport 2.x Backport to 2.x branch label Dec 12, 2024

fixed failing tests

70cb875

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

mch2 approved these changes Dec 12, 2024

View reviewed changes

mch2 merged commit b67cdf4 into opensearch-project:main Dec 12, 2024
38 checks passed

opensearch-trigger-bot bot mentioned this pull request Dec 12, 2024

[Backport 2.x] Added support for search replica to return segrep stats #16842

Merged

opensearch-ci-bot mentioned this pull request Dec 17, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SegmentReplicationAllocationIT #14327

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for search replica to return segrep stats #16678

Added support for search replica to return segrep stats #16678

vinaykpud commented Nov 18, 2024

github-actions bot commented Nov 18, 2024

github-actions bot commented Nov 19, 2024

codecov bot commented Nov 19, 2024 •

edited

Loading

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 25, 2024

github-actions bot commented Dec 10, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

Added support for search replica to return segrep stats #16678

Added support for search replica to return segrep stats #16678

Conversation

vinaykpud commented Nov 18, 2024

Description

Related Issues

Check List

github-actions bot commented Nov 18, 2024

github-actions bot commented Nov 19, 2024

codecov bot commented Nov 19, 2024 • edited Loading

Codecov Report

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 22, 2024

github-actions bot commented Nov 25, 2024

github-actions bot commented Dec 10, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

codecov bot commented Nov 19, 2024 •

edited

Loading