Add remote shards allocator for searchable snapshots #4870

kotwanikunal · 2022-10-21T17:32:20Z

Signed-off-by: Kunal Kotwani kkotwani@amazon.com

Description

Adds a remote allocator for searchable snapshots

Issues Resolved

[Searchable Snapshots] Add support for RemoteShardsBalancer #4777

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-10-21T17:36:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5209/
CommitID: 52660d0

server/src/main/java/org/opensearch/common/util/FeatureFlags.java

github-actions · 2022-10-21T17:44:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5211/
CommitID: ac4589e

github-actions · 2022-10-21T18:15:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5216/
CommitID: 57ac40e

server/src/main/java/org/opensearch/cluster/routing/RoutingPool.java

github-actions · 2022-10-21T18:57:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5220/
CommitID: 25ed1be

github-actions · 2022-10-21T20:43:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5232/
CommitID: c0d9171

kotwanikunal · 2022-10-21T20:45:37Z

Unrelated failures:

> [2022-10-21T23:41:37,721][INFO ][o.o.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=index/80_geo_point/Single point test] after test
  2> REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v2.4.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=index/80_geo_point/Single point test}" -Dtests.seed=5BA55F046123AF6F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr-CH -Dtests.timezone=Antarctica/Syowa -Druntime.java=17
  2> java.lang.AssertionError: Failure at [index/80_geo_point:60]: hits.total didn't match expected value:

Tests with failures:
 - org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=index/80_geo_point/Multi points test}
 - org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=index/80_geo_point/Single point test}

github-actions · 2022-10-21T21:17:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5240/
CommitID: c0d9171

kotwanikunal · 2022-10-21T21:31:36Z

Tests with failures:
 - org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=index/80_geo_point/Single point test}
 - org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=index/80_geo_point/Multi points test}

kotwanikunal · 2022-10-21T21:32:48Z

#4852

andrross · 2022-10-21T23:36:12Z

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

+    private Queue<RoutingNode> getShuffledRemoteNodes() {
+        Iterator<RoutingNode> nodesIter = routingNodes.mutableIterator();
+        List<RoutingNode> nodeList = new ArrayList<>();
+        while (nodesIter.hasNext()) {


Can you replace the mutable iterator and while loop with the following?

for (RoutingNode rNode : routingNodes) { ... }

andrross · 2022-10-21T23:40:41Z

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

+    }
+
+    /**
+     * Performs heuristic based balancing for remote shards within the cluster.


Can you expand a bit here? For example, what is the heuristic being used here? Doesn't need to be super detailed, just a high level description of exactly what is being done.

andrross · 2022-10-21T23:41:29Z

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

+            UnassignedIndexShards indexShards = unassignedShardMap.get(index);
+            indexShards.addShard(shard);


Nitpick, but can probably just do:

unassignedShardMap.get(index).addShard(shard);

github-actions · 2022-10-22T06:53:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5307/
CommitID: 716c9f9

github-actions · 2022-10-23T22:35:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5325/
CommitID: 0dffbf1

github-actions · 2022-10-23T22:40:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5326/
CommitID: f4f6071

github-actions · 2022-10-24T05:21:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5329/
CommitID: 1b5bb28

github-actions · 2022-10-24T05:53:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5331/
CommitID: 1b5bb28

kotwanikunal · 2022-10-24T17:09:17Z

Seems related since the flag is forcefully turned on for SEARCHABLE_SNAPSHOT -

Tests with failures:
 - org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=snapshot.restore/10_basic/Create a snapshot and then restore it}

Vishalks · 2022-10-27T17:55:21Z

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

+        ObjectIntHashMap<String> primaryShardCount = new ObjectIntHashMap<>();
+        int totalPrimaryShard = 0;
+        for (RoutingNode node : nodeList) {
+            int totalPrimaryPerNode = 0;


What's the difference between totalPrimaryShard above and totalPrimaryPerNode? If the latter conveys totalPrimaryShardPerNode, we can rename it accordingly. If not, maybe a better variable name to distinguish would help here?

Renamed. It was exactly what you figured out.
Simplified the logic and extracted it out.

Vishalks · 2022-10-27T17:56:45Z

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

+
+        ObjectIntHashMap<String> primaryShardCount = new ObjectIntHashMap<>();
+        int totalPrimaryShard = 0;
+        for (RoutingNode node : nodeList) {


Can we extract this logic to a different method?

Vishalks · 2022-10-27T17:58:46Z

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

+        }
+    }
+
+    private void allocateUnassigned(


Can we break this down to smaller methods? Looks difficult to follow.

github-actions · 2022-10-28T03:59:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5832/
CommitID: ab5ce2e
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2022-10-28T04:04:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/5832/

CommitID: ab5ce2e
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Unrelated failure:

REPRODUCE WITH: ./gradlew ':sandbox:plugins:concurrent-search:test' --tests "org.opensearch.search.query.QueryProfilePhaseTests" -Dtests.method="testTerminateAfterEarlyTermination {p0=5 p1=org.opensearch.search.query.ConcurrentQueryPhaseSearcher@1d6a22dd}" -Dtests.seed=E4EDD6626F1EC93B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hu-HU -Dtests.timezone=SystemV/PST8PDT -Druntime.java=17

org.opensearch.search.query.QueryProfilePhaseTests > testTerminateAfterEarlyTermination {p0=5 p1=org.opensearch.search.query.ConcurrentQueryPhaseSearcher@1d6a22dd} FAILED
    java.lang.AssertionError: 
    Expected: a value greater than <0L>
         but: <0L> was equal to <0L>
        at __randomizedtesting.SeedInfo.seed([E4EDD6626F1EC93B:72496FF5662CB552]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.search.query.QueryProfilePhaseTests.lambda$testTerminateAfterEarlyTermination$26(QueryProfilePhaseTests.java:574)
        at org.opensearch.search.query.QueryProfilePhaseTests.lambda$assertProfileData$55(QueryProfilePhaseTests.java:1105)
        at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1156)
        at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1111)
        at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1102)
        at org.opensearch.search.query.QueryProfilePhaseTests.testTerminateAfterEarlyTermination(QueryProfilePhaseTests.java:555)

github-actions · 2022-10-28T04:32:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5833/
CommitID: 3f5641e
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-10-28T07:18:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5836/
CommitID: 3f5641e
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2022-10-28T17:14:52Z

#4881

github-actions · 2022-10-28T17:30:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: https://build.ci.opensearch.org/job/gradle-check/5852/
CommitID: 3f5641e
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-10-28T18:34:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/5861/
CommitID: 01fdb8e

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Part of this commit was developed from code and concepts initially implemented in Amazon OpenSearch Service as part of the UltraWarm feature. Thank you to the following developers and the entire UltraWarm team. Co-authored-by: Min Zhou <minzho@amazon.com> Co-authored-by: Ankit Malpani <malpani@amazon.com> Co-authored-by: Rohit Nair <rohinair@amazon.com> Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Tianru Zhou <tianruz@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Amit Khandelwal <mkhnde@amazon.com> Co-authored-by: Vigya Sharma <vigyas@amazon.com> Co-authored-by: Prateek Sharma <shrprat@amazon.com> Co-authored-by: Venkata Jyothsna Donapati <donapv@amazon.com> Co-authored-by: Vlad Rozov <vrozov@amazon.com> Co-authored-by: Mohit Agrawal <agramohi@amazon.com> Co-authored-by: Shweta Thareja <tharejas@amazon.com> Co-authored-by: Palash Hedau <phhedau@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Piyush Daftary <pdaftary@amazon.com> Co-authored-by: Payal Maheshwari <pmaheshw@amazon.com> Co-authored-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Gulshan Kumar <kumargu@amazon.com> Co-authored-by: Rushi Agrawal <agrrushi@amazon.com> Co-authored-by: Ketan Verma <vermketa@amazon.com> Co-authored-by: Gaurav Chandani <chngau@amazon.com> Co-authored-by: Dharmesh Singh <sdharms@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

github-actions · 2022-11-01T19:30:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6101/
CommitID: 694c89f
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-01T20:28:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6119/
CommitID: 694c89f

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Part of this commit was developed from code and concepts initially implemented in Amazon OpenSearch Service as part of the UltraWarm feature. Thank you to the following developers and the entire UltraWarm team. Co-authored-by: Min Zhou <minzho@amazon.com> Co-authored-by: Ankit Malpani <malpani@amazon.com> Co-authored-by: Rohit Nair <rohinair@amazon.com> Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Tianru Zhou <tianruz@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Amit Khandelwal <mkhnde@amazon.com> Co-authored-by: Vigya Sharma <vigyas@amazon.com> Co-authored-by: Prateek Sharma <shrprat@amazon.com> Co-authored-by: Venkata Jyothsna Donapati <donapv@amazon.com> Co-authored-by: Vlad Rozov <vrozov@amazon.com> Co-authored-by: Mohit Agrawal <agramohi@amazon.com> Co-authored-by: Shweta Thareja <tharejas@amazon.com> Co-authored-by: Palash Hedau <phhedau@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Piyush Daftary <pdaftary@amazon.com> Co-authored-by: Payal Maheshwari <pmaheshw@amazon.com> Co-authored-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Gulshan Kumar <kumargu@amazon.com> Co-authored-by: Rushi Agrawal <agrrushi@amazon.com> Co-authored-by: Ketan Verma <vermketa@amazon.com> Co-authored-by: Gaurav Chandani <chngau@amazon.com> Co-authored-by: Dharmesh Singh <sdharms@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: Min Zhou <minzho@amazon.com> Co-authored-by: Ankit Malpani <malpani@amazon.com> Co-authored-by: Rohit Nair <rohinair@amazon.com> Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Tianru Zhou <tianruz@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Amit Khandelwal <mkhnde@amazon.com> Co-authored-by: Vigya Sharma <vigyas@amazon.com> Co-authored-by: Prateek Sharma <shrprat@amazon.com> Co-authored-by: Venkata Jyothsna Donapati <donapv@amazon.com> Co-authored-by: Vlad Rozov <vrozov@amazon.com> Co-authored-by: Mohit Agrawal <agramohi@amazon.com> Co-authored-by: Shweta Thareja <tharejas@amazon.com> Co-authored-by: Palash Hedau <phhedau@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Piyush Daftary <pdaftary@amazon.com> Co-authored-by: Payal Maheshwari <pmaheshw@amazon.com> Co-authored-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Gulshan Kumar <kumargu@amazon.com> Co-authored-by: Rushi Agrawal <agrrushi@amazon.com> Co-authored-by: Ketan Verma <vermketa@amazon.com> Co-authored-by: Gaurav Chandani <chngau@amazon.com> Co-authored-by: Dharmesh Singh <sdharms@amazon.com> (cherry picked from commit 9119104) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

…5024) * Add remote allocator for searchable snapshots (#4870) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Part of this commit was developed from code and concepts initially implemented in Amazon OpenSearch Service as part of the UltraWarm feature. Thank you to the following developers and the entire UltraWarm team. Co-authored-by: Min Zhou <minzho@amazon.com> Co-authored-by: Ankit Malpani <malpani@amazon.com> Co-authored-by: Rohit Nair <rohinair@amazon.com> Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Tianru Zhou <tianruz@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Amit Khandelwal <mkhnde@amazon.com> Co-authored-by: Vigya Sharma <vigyas@amazon.com> Co-authored-by: Prateek Sharma <shrprat@amazon.com> Co-authored-by: Venkata Jyothsna Donapati <donapv@amazon.com> Co-authored-by: Vlad Rozov <vrozov@amazon.com> Co-authored-by: Mohit Agrawal <agramohi@amazon.com> Co-authored-by: Shweta Thareja <tharejas@amazon.com> Co-authored-by: Palash Hedau <phhedau@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Piyush Daftary <pdaftary@amazon.com> Co-authored-by: Payal Maheshwari <pmaheshw@amazon.com> Co-authored-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Gulshan Kumar <kumargu@amazon.com> Co-authored-by: Rushi Agrawal <agrrushi@amazon.com> Co-authored-by: Ketan Verma <vermketa@amazon.com> Co-authored-by: Gaurav Chandani <chngau@amazon.com> Co-authored-by: Dharmesh Singh <sdharms@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 9119104) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md * Update changelog Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>

…5028) * Add remote allocator for searchable snapshots (#4870) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Part of this commit was developed from code and concepts initially implemented in Amazon OpenSearch Service as part of the UltraWarm feature. Thank you to the following developers and the entire UltraWarm team. Co-authored-by: Min Zhou <minzho@amazon.com> Co-authored-by: Ankit Malpani <malpani@amazon.com> Co-authored-by: Rohit Nair <rohinair@amazon.com> Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Tianru Zhou <tianruz@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Amit Khandelwal <mkhnde@amazon.com> Co-authored-by: Vigya Sharma <vigyas@amazon.com> Co-authored-by: Prateek Sharma <shrprat@amazon.com> Co-authored-by: Venkata Jyothsna Donapati <donapv@amazon.com> Co-authored-by: Vlad Rozov <vrozov@amazon.com> Co-authored-by: Mohit Agrawal <agramohi@amazon.com> Co-authored-by: Shweta Thareja <tharejas@amazon.com> Co-authored-by: Palash Hedau <phhedau@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Piyush Daftary <pdaftary@amazon.com> Co-authored-by: Payal Maheshwari <pmaheshw@amazon.com> Co-authored-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Gulshan Kumar <kumargu@amazon.com> Co-authored-by: Rushi Agrawal <agrrushi@amazon.com> Co-authored-by: Ketan Verma <vermketa@amazon.com> Co-authored-by: Gaurav Chandani <chngau@amazon.com> Co-authored-by: Dharmesh Singh <sdharms@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: Min Zhou <minzho@amazon.com> Co-authored-by: Ankit Malpani <malpani@amazon.com> Co-authored-by: Rohit Nair <rohinair@amazon.com> Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Tianru Zhou <tianruz@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Amit Khandelwal <mkhnde@amazon.com> Co-authored-by: Vigya Sharma <vigyas@amazon.com> Co-authored-by: Prateek Sharma <shrprat@amazon.com> Co-authored-by: Venkata Jyothsna Donapati <donapv@amazon.com> Co-authored-by: Vlad Rozov <vrozov@amazon.com> Co-authored-by: Mohit Agrawal <agramohi@amazon.com> Co-authored-by: Shweta Thareja <tharejas@amazon.com> Co-authored-by: Palash Hedau <phhedau@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Piyush Daftary <pdaftary@amazon.com> Co-authored-by: Payal Maheshwari <pmaheshw@amazon.com> Co-authored-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Gulshan Kumar <kumargu@amazon.com> Co-authored-by: Rushi Agrawal <agrrushi@amazon.com> Co-authored-by: Ketan Verma <vermketa@amazon.com> Co-authored-by: Gaurav Chandani <chngau@amazon.com> Co-authored-by: Dharmesh Singh <sdharms@amazon.com> (cherry picked from commit 9119104) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md * Update changelog Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>

kotwanikunal requested review from a team and reta as code owners October 21, 2022 17:32

kotwanikunal force-pushed the remote-allocator branch from 52660d0 to ac4589e Compare October 21, 2022 17:33

kotwanikunal commented Oct 21, 2022

View reviewed changes

server/src/main/java/org/opensearch/common/util/FeatureFlags.java Outdated Show resolved Hide resolved

kotwanikunal force-pushed the remote-allocator branch from ac4589e to 57ac40e Compare October 21, 2022 18:03

kotwanikunal force-pushed the remote-allocator branch from 57ac40e to 25ed1be Compare October 21, 2022 18:28

andrross reviewed Oct 21, 2022

View reviewed changes

server/src/main/java/org/opensearch/cluster/routing/RoutingPool.java Show resolved Hide resolved

kotwanikunal force-pushed the remote-allocator branch from 25ed1be to c0d9171 Compare October 21, 2022 20:12

kotwanikunal changed the title ~~Add remote allocator for searchable snapshots~~ Add remote shards allocator for searchable snapshots Oct 21, 2022

kotwanikunal mentioned this pull request Oct 21, 2022

[BUG] flaky test index/80_geo_point/Single point test #4852

Closed

andrross reviewed Oct 21, 2022

View reviewed changes

kotwanikunal force-pushed the remote-allocator branch from c0d9171 to 716c9f9 Compare October 22, 2022 06:32

kotwanikunal force-pushed the remote-allocator branch 2 times, most recently from 0dffbf1 to f4f6071 Compare October 23, 2022 22:14

kotwanikunal force-pushed the remote-allocator branch from f4f6071 to 1b5bb28 Compare October 24, 2022 04:54

Vishalks reviewed Oct 27, 2022

View reviewed changes

kotwanikunal added the v2.4.0 'Issues and PRs related to version v2.4.0' label Oct 27, 2022

kotwanikunal force-pushed the remote-allocator branch from a54f7a3 to ab5ce2e Compare October 28, 2022 03:51

kotwanikunal force-pushed the remote-allocator branch from ab5ce2e to 3f5641e Compare October 28, 2022 04:04

kotwanikunal force-pushed the remote-allocator branch from 3f5641e to 01fdb8e Compare October 28, 2022 17:46

kotwanikunal force-pushed the remote-allocator branch from 01fdb8e to 694c89f Compare November 1, 2022 19:00

kotwanikunal added the backport 2.x Backport to 2.x branch label Nov 1, 2022

kotwanikunal merged commit 9119104 into opensearch-project:main Nov 1, 2022

opensearch-trigger-bot bot mentioned this pull request Nov 1, 2022

[Backport 2.x] Add remote shards allocator for searchable snapshots #5024

Merged

kotwanikunal added the backport 2.4 Backport to 2.4 branch label Nov 1, 2022

opensearch-trigger-bot bot mentioned this pull request Nov 1, 2022

[Backport 2.4] Add remote shards allocator for searchable snapshots #5028

Merged

kotwanikunal mentioned this pull request Nov 2, 2022

[Searchable Snapshots] Add support for RemoteShardsBalancer #4777

Closed

RS146BIJAY mentioned this pull request Jul 2, 2024

Removing unnecessary List.removeAll from LocalShardsBalancer to filter remote search shard from relocation decision #14613

Merged

3 tasks

RS146BIJAY mentioned this pull request Aug 5, 2024

[BUG] Removing unnecessary List.removeAll from LocalShardsBalancer to filter remote search shard from relocation decision #15106

Closed

		UnassignedIndexShards indexShards = unassignedShardMap.get(index);
		indexShards.addShard(shard);

Add remote shards allocator for searchable snapshots #4870

Add remote shards allocator for searchable snapshots #4870

Conversation

kotwanikunal commented Oct 21, 2022

Description

Issues Resolved

Check List

github-actions bot commented Oct 21, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 21, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 21, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 21, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 21, 2022

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Oct 21, 2022

github-actions bot commented Oct 21, 2022

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Oct 21, 2022

kotwanikunal commented Oct 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 22, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 24, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 24, 2022

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Oct 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 28, 2022

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Oct 28, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 28, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 28, 2022

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Oct 28, 2022

github-actions bot commented Oct 28, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 28, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 1, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 1, 2022

Gradle Check (Jenkins) Run Completed with: