[Remote Store] Primary/Replica side changes to support Dual Replication #12821

shourya035 · 2024-03-21T09:29:49Z

Description

This PR adds support for dual mode replication during remote store migration; Changes made:

Change all implementations of indexShard.isRemoteTranslogStoreEnabled to indexShard.indexSettings().isRemoteNode. This ensures that shards started on Remote store enabled nodes performs remote based operations in the replication flow.
Wiring DiscoveryNodes obtained from ClusterState to IndexShard. A Function implementation is passed on to ReplicationTracker as a supplier to check if the shards in a replication group is assigned to a remote or docrep node
Depending on whether the target shard is hosted on a remote enabled or non-remote (docrep enabled) node, the ReplicationModeAwareProxy would determine whether replication actions needs to be dropped or fan out to the corresponding replica shard copy. Replication overrides are bypassed and all actions are fanned out if the target shard copy is residing in a docrep enabled node

Added integration tests to test out the following flows:

Primary in remote, replica in docrep, primary hosting node uploads to remote store but replica hosting node does not
Primary in remote, replica in docrep and remote, primary and one replica hosting node uploads to remote but the docrep replica hosting node does not
Primary in remote, fails over to replica in docrep. Docrep shard continues indexing as is and does not upload anything to remote

Old PR raised on @gbbafna 's fork: gbbafna#152

Related Issues

Partially resolves: #12413

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-03-21T09:51:26Z

Compatibility status:

Checks if related components are compatible with change 0ed42b8

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-03-21T10:01:51Z

❌ Gradle check result for 163635b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-21T10:10:46Z

❌ Gradle check result for 50cb08a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-21T12:28:49Z

❌ Gradle check result for 82f2562: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-21T12:41:08Z

❌ Gradle check result for 8aabf1a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-21T17:55:12Z

❌ Gradle check result for 0a5ef5b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java

server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java

server/src/main/java/org/opensearch/action/support/replication/ReplicationModeAwareProxy.java

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java

...internalClusterTest/java/org/opensearch/remotemigration/DocrepToRemoteDualReplicationIT.java

ashking94 · 2024-03-22T10:01:11Z

@shourya035 Lets add one more usecase for integ test -

Primary in remote, replica in docrep and remote. Primary on remote fails, remote replica should become the new primary?

shourya035 · 2024-03-22T10:09:48Z

@shourya035 Lets add one more usecase for integ test -

* Primary in remote, replica in docrep and remote. Primary on remote fails, remote replica should become the new primary?

@ashking94 We don't have that piece ready yet. As of now, if a primary in remote fails, the failover can happen to either docrep or remote copy. @ltaragi is working on this piece wherein failover preference would be given to the remote copy over the docrep one.

server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java

server/src/main/java/org/opensearch/action/support/replication/ReplicationModeAwareProxy.java

server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java

server/src/main/java/org/opensearch/cluster/routing/ShardRouting.java

server/src/main/java/org/opensearch/index/remote/RemoteStoreUtils.java

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java

server/src/main/java/org/opensearch/indices/replication/checkpoint/PublishCheckpointAction.java

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

server/src/main/java/org/opensearch/index/IndexSettings.java

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-04-01T10:04:20Z

❌ Gradle check result for 3a39232: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-04-01T10:32:53Z

✅ Gradle check result for 04fc5cb: SUCCESS

github-actions · 2024-04-01T10:50:37Z

❌ Gradle check result for b11585a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/support/replication/ReplicationModeAwareProxy.java

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/main/java/org/opensearch/index/shard/StoreRecovery.java

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-04-01T16:02:16Z

✅ Gradle check result for 19e7221: SUCCESS

server/src/test/java/org/opensearch/action/support/replication/ReplicationOperationTests.java

server/src/internalClusterTest/java/org/opensearch/remotemigration/RemoteDualMigrationIT.java

ashking94

lgtm, left some minor comments, pls address them.

server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java

server/src/main/java/org/opensearch/action/support/replication/ReplicationModeAwareProxy.java

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-04-02T10:19:26Z

❌ Gradle check result for 77798c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-04-02T11:12:21Z

❕ Gradle check result for 0ed42b8: UNSTABLE

TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimarySingleReplica
      1 org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting
      1 org.opensearch.index.ShardIndexingPressureSettingsIT.classMethod
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals}

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…cation during Remote Store Migration (opensearch-project#12821) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

…port Dual Replication during Remote Store Migration (#13028) * [Remote Store] Add Primary/Replica side changes to support Dual Replication during Remote Store Migration (#12821) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> * Fix build due to multiple commits to same file causing compilation failure (#13019) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

…cation during Remote Store Migration (opensearch-project#12821) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

…cation during Remote Store Migration (opensearch-project#12821) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions bot added enhancement Enhancement or improvement to existing feature or request RFC Issues requesting major changes Storage:Durability Issues and PRs related to the durability framework Storage:Remote labels Mar 21, 2024

shourya035 force-pushed the dual-mode-rep-pri branch from 163635b to 50cb08a Compare March 21, 2024 09:31

shourya035 added skip-changelog and removed RFC Issues requesting major changes labels Mar 21, 2024

github-actions bot added the RFC Issues requesting major changes label Mar 21, 2024

shourya035 force-pushed the dual-mode-rep-pri branch from 82f2562 to 8aabf1a Compare March 21, 2024 12:20

shourya035 self-assigned this Mar 21, 2024

shourya035 removed enhancement Enhancement or improvement to existing feature or request RFC Issues requesting major changes Storage:Remote Storage:Durability Issues and PRs related to the durability framework labels Mar 21, 2024

gbbafna reviewed Mar 22, 2024

View reviewed changes

github-actions bot added enhancement Enhancement or improvement to existing feature or request RFC Issues requesting major changes Storage:Durability Issues and PRs related to the durability framework Storage:Remote labels Mar 22, 2024

ashking94 requested changes Mar 22, 2024

View reviewed changes

shourya035 added 3 commits April 1, 2024 14:45

Fixing typo in IT log line

4a8f92d

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Removing unused TestLogging directive

c7b2dab

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Addressing comments

04fc5cb

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

shourya035 force-pushed the dual-mode-rep-pri branch from 6a4b944 to 04fc5cb Compare April 1, 2024 09:44

shourya035 commented Apr 1, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/IndexSettings.java Show resolved Hide resolved

Removing unused changes

3a39232

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Applying spotless checks

b11585a

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

gbbafna reviewed Apr 1, 2024

View reviewed changes

Addressing comments

19e7221

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

gbbafna reviewed Apr 2, 2024

View reviewed changes

ashking94 approved these changes Apr 2, 2024

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportShardBulkAction.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/action/support/replication/ReplicationModeAwareProxy.java Outdated Show resolved Hide resolved

Addressing comments

77798c7

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Retrigger integs

0ed42b8

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

gbbafna approved these changes Apr 2, 2024

View reviewed changes

gbbafna merged commit 8def8cb into opensearch-project:main Apr 2, 2024
30 of 32 checks passed

gbbafna mentioned this pull request Apr 2, 2024

Fix build due to multiple commits to same file causing compilation fa… #13019

Merged

8 tasks

shourya035 mentioned this pull request Apr 2, 2024

[Backport 2.x] Remote Store - Add Primary/Replica side changes to support Dual Replication during Remote Store Migration #13028

Merged

8 tasks

gbbafna mentioned this pull request Apr 18, 2024

[RFC] Reconsider the breaking changes check policy #13275

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Primary/Replica side changes to support Dual Replication #12821

[Remote Store] Primary/Replica side changes to support Dual Replication #12821

shourya035 commented Mar 21, 2024 •

edited

Loading

github-actions bot commented Mar 21, 2024 •

edited

Loading

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

ashking94 commented Mar 22, 2024

shourya035 commented Mar 22, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

ashking94 left a comment

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

[Remote Store] Primary/Replica side changes to support Dual Replication #12821

[Remote Store] Primary/Replica side changes to support Dual Replication #12821

Conversation

shourya035 commented Mar 21, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Mar 21, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

ashking94 commented Mar 22, 2024

shourya035 commented Mar 22, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

shourya035 commented Mar 21, 2024 •

edited

Loading

github-actions bot commented Mar 21, 2024 •

edited

Loading