[Remote Store] Moving stale commit deletion to async flow #8201

psychbot · 2023-06-21T20:38:54Z

Description

Changes to move the stale commit deletion logic to async flow.

Related Issues

Resolves #4315

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-06-21T21:16:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/18116/
CommitID: aa3c392

codecov · 2023-06-21T21:18:42Z

Codecov Report

Merging #8201 (5c446f3) into main (b594f52) will decrease coverage by 0.02%.
The diff coverage is 86.95%.

@@             Coverage Diff              @@
##               main    #8201      +/-   ##
============================================
- Coverage     70.86%   70.84%   -0.02%     
+ Complexity    56825    56768      -57     
============================================
  Files          4752     4752              
  Lines        268599   268610      +11     
  Branches      39336    39336              
============================================
- Hits         190335   190292      -43     
- Misses        62235    62243       +8     
- Partials      16029    16075      +46

Impacted Files	Coverage Δ
...in/java/org/opensearch/index/shard/IndexShard.java	`69.21% <0.00%> (-0.33%)`	⬇️
...java/org/opensearch/index/shard/StoreRecovery.java	`57.25% <0.00%> (-0.42%)`	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	`85.56% <100.00%> (-4.39%)`	⬇️
...earch/index/store/RemoteSegmentStoreDirectory.java	`91.91% <100.00%> (+1.86%)`	⬆️
...ndex/store/RemoteSegmentStoreDirectoryFactory.java	`95.65% <100.00%> (+0.19%)`	⬆️
server/src/main/java/org/opensearch/node/Node.java	`84.95% <100.00%> (ø)`

... and 435 files with indirect coverage changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

...er/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreBaseIntegTestCase.java

ashking94

have left couple of comments, do take a look

github-actions · 2023-06-22T08:27:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/18158/
CommitID: 6b94980

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

github-actions · 2023-06-22T11:02:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18160/
CommitID: 560c17b
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-28T19:51:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18600/
CommitID: 051a512
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/test/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryTests.java

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

github-actions · 2023-06-29T17:29:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18734/
CommitID: 63e08cf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-29T18:55:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18735/
CommitID: 45b8233
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java

ashking94

Left some minor comments, please take a look.

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

github-actions · 2023-06-30T10:13:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogCleanup
      1 org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogCleanup
      1 org.opensearch.remotestore.RemoteStoreIT.testRemoteSegmentCleanup

URL: https://build.ci.opensearch.org/job/gradle-check/18800/
CommitID: d541935
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

github-actions · 2023-06-30T12:02:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18803/
CommitID: 5c446f3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94

LGTM

sachinpkale · 2023-07-03T07:34:26Z

Build failure due to flaky tests.

github-actions · 2023-07-03T08:07:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18919/
CommitID: 5c446f3
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-07-03T08:53:39Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8201-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1d2274045b42e129b8c06748f8185d2431aa5336
# Push it to GitHub
git push --set-upstream origin backport/backport-8201-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8201-to-2.x.

…ow (opensearch-project#8201) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

…ow (#8201) (#8442) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

…-project#8201) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com> Signed-off-by: sahil buddharaju <sahilbud@amazon.com>

…-project#8201) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

…-project#8201) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

ashking94 reviewed Jun 22, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java Outdated Show resolved Hide resolved

ashking94 reviewed Jun 22, 2023

View reviewed changes

...er/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreBaseIntegTestCase.java Outdated Show resolved Hide resolved

ashking94 requested changes Jun 22, 2023

View reviewed changes

ashking94 reviewed Jun 22, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java Outdated Show resolved Hide resolved

sachinpkale reviewed Jun 29, 2023

View reviewed changes

server/src/test/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryTests.java Show resolved Hide resolved

Adding assertBusy statement in unit tests

45b8233

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

psychbot force-pushed the delete-stale-commits-async-flow branch from 63e08cf to 45b8233 Compare June 29, 2023 17:25

ashking94 reviewed Jun 30, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java Outdated Show resolved Hide resolved

ashking94 reviewed Jun 30, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java Outdated Show resolved Hide resolved

ashking94 reviewed Jun 30, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java Show resolved Hide resolved

ashking94 requested changes Jun 30, 2023

View reviewed changes

ankitkala reviewed Jun 30, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java Show resolved Hide resolved

ankitkala reviewed Jun 30, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java Show resolved Hide resolved

Removing catch exception from IT

5c446f3

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

psychbot force-pushed the delete-stale-commits-async-flow branch from d541935 to 5c446f3 Compare June 30, 2023 11:27

psychbot mentioned this pull request Jun 30, 2023

[Remote Store] Garbage Collection for Remote Backed Storage #8378

Open

ashking94 approved these changes Jul 3, 2023

View reviewed changes

sachinpkale approved these changes Jul 3, 2023

View reviewed changes

sachinpkale merged commit 1d22740 into opensearch-project:main Jul 3, 2023

sachinpkale added the backport 2.x Backport to 2.x branch label Jul 3, 2023

psychbot added a commit to psychbot/OpenSearch that referenced this pull request Jul 5, 2023

[Backport 2.x][Remote Store] Moving stale commit deletion to async fl…

d6859b8

…ow (opensearch-project#8201) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

psychbot mentioned this pull request Jul 5, 2023

[Backport 2.x][Remote Store] Moving stale commit deletion to async fl… #8442

Merged

6 tasks

sachinpkale pushed a commit that referenced this pull request Jul 5, 2023

[Backport 2.x][Remote Store] Moving stale commit deletion to async fl…

9b9718f

…ow (#8201) (#8442) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023

[Remote Store] Moving stale commit deletion to async flow (opensearch…

4631c58

…-project#8201) --------- Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Moving stale commit deletion to async flow #8201

[Remote Store] Moving stale commit deletion to async flow #8201

psychbot commented Jun 21, 2023 •

edited

Loading

github-actions bot commented Jun 21, 2023

codecov bot commented Jun 21, 2023 •

edited

Loading

ashking94 left a comment

github-actions bot commented Jun 22, 2023

github-actions bot commented Jun 22, 2023

github-actions bot commented Jun 28, 2023

github-actions bot commented Jun 29, 2023

github-actions bot commented Jun 29, 2023

ashking94 left a comment

github-actions bot commented Jun 30, 2023

github-actions bot commented Jun 30, 2023

ashking94 left a comment

sachinpkale commented Jul 3, 2023

github-actions bot commented Jul 3, 2023

opensearch-trigger-bot bot commented Jul 3, 2023

[Remote Store] Moving stale commit deletion to async flow #8201

[Remote Store] Moving stale commit deletion to async flow #8201

Conversation

psychbot commented Jun 21, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jun 21, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jun 21, 2023 • edited Loading

Codecov Report

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 29, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 30, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 left a comment

Choose a reason for hiding this comment

sachinpkale commented Jul 3, 2023

github-actions bot commented Jul 3, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jul 3, 2023

psychbot commented Jun 21, 2023 •

edited

Loading

codecov bot commented Jun 21, 2023 •

edited

Loading