[Remote Store] Add support to restore only unassigned shards of an index

[sachinpkale]

Description

• Currently, remote store restore API restores all the shards of a given index irrespective of the shard state.
• This approach is not optimized for scenarios where only few of the shards are unassigned. Restore will still download data for all the shards in order to restore them.
• We also need to close the index before calling remote store restore API. This impacts the index availability during the restore.
• Ideally, we should be able to restore only those shards that are unassigned and that too without closing the index.
• In restore API execution, while marking shards to be restored, we filter shards based on existing ShardRoutingState.
• We pick only shards that are in UNASSIGNED state as shards to be restored.
• User do not need to provide shard info in the API request. API signature would remain same which accepts index/indices to be restored. API implementation internally picks up only unassigned shards to restore.
• We also provided a query parameter restore_all_shards which, as the name suggests, will restore all the shards of the given index. As some of the shards can be active, to avoid an inconsistent state, when restore_all_shards=true is passed, index needs to be in closed state.

Related Issues

Resolves #8836

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20561/
• CommitID: 446187d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20566/
• CommitID: b2485fc
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20568/
• CommitID: 5be4c1f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20569/
• CommitID: 9d87df5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20583/
• CommitID: 38d11b0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20585/
• CommitID: ccdf757
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20601/
• CommitID: 8b6f741
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20639/
• CommitID: 4f4c650
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

• URL: https://build.ci.opensearch.org/job/gradle-check/20655/
• CommitID: cb66244
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8792 (9ad837f) into main (4319f2b) will decrease coverage by 0.07%.
Report is 1 commits behind head on main.
The diff coverage is 38.88%.

@@             Coverage Diff              @@
##               main    #8792      +/-   ##
============================================
- Coverage     71.06%   70.99%   -0.07%     
+ Complexity    57253    57245       -8     
============================================
  Files          4758     4758              
  Lines        269840   269862      +22     
  Branches      39480    39484       +4     
============================================
- Hits         191751   191596     -155     
- Misses        61925    62191     +266     
+ Partials      16164    16075      -89     

Files Changed	Coverage Δ
...on/admin/cluster/RestRestoreRemoteStoreAction.java	30.00% <0.00%> (-3.34%)	⬇️
.../java/org/opensearch/snapshots/RestoreService.java	54.79% <0.00%> (+0.51%)	⬆️
...remotestore/restore/RestoreRemoteStoreRequest.java	70.68% <100.00%> (+0.68%)	⬆️
.../opensearch/cluster/routing/IndexRoutingTable.java	76.22% <100.00%> (+0.19%)	⬆️
...a/org/opensearch/cluster/routing/RoutingTable.java	94.37% <100.00%> (ø)

... and 475 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20977/
• CommitID: a30f2cf
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDropPrimaryDuringReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/20979/
• CommitID: 0c2dca3
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/20980/
• CommitID: 5b3df83

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/20997/
• CommitID: 9ad837f

[sachinpkale]

Integ tests are covering new changes. Bypassing the codecov/patch failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8792-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c25c175c94efa4d90a1d98e3c7eac3e4aad6e41f
# Push it to GitHub
git push --set-upstream origin backport/backport-8792-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8792-to-2.x.