[Remote Store] Update index settings on shard movement during remote store migration

[shourya035]

Description

Reopening this since #13253 got closed during a rebase from main

The IndexMetadataUpdater logic runs on the cluster manager and applies changes in shard state triggered by AllocationService. Depending on the state of the shard after AllocationService decisions, the IndexMetadataUpdater would currently do stuff like update in-sync allocation IDs or bump primary terms in the IndexMetadata which is the cluster manager then publishes to the data nodes as a state update. The IndicesClusterStateService responds to these changes and then performs relevant actions like creating or updating the existing IndexShard instance. We are utilizing this code path and adding logic within the IndexMetadataUpdater to:

• Append remote store path based index metadata when the first primary shard copy moves over to remote
• Apply remote store based index settings once all shard copies of an index moves over to remote store enabled nodes.

Currently this logic would only execute when the cluster is in mixed mode and direction for migration is remote_store

Since the updated metadata would be handed over from the cluster manager to the data nodes during a state publication, this helps us in:

• Ensuring that the latest remote path based changes (Add faster scaling composite hash value encoding for remote path #13155) are utilized during primary seeding since the initial seeding cycle would upload the entire segment data present in the shard copy to remote (which would have a relatively bigger upload payload)

and

• Remote Store features like shallow snapshots can be enabled as soon as all available shard copies of an index moves over to the remote store enabled nodes.

Apart from this:

• Updated the existing ITs under RemoteDualReplicationIT to accommodate the changes being introduced through this PR
• Made additional validations alongside Add validation while updating CompatibilityMode setting #13080 which would prevent switching the cluster to strict from mixed mode if all indices in the cluster does not have the remote store based settings.

Related Issues

Resolves #13252

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for f14e231: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ca4b1d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 667cffe: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for cae1886: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6f9f0c3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 5c8df2c: SUCCESS

[github-actions]

❌ Gradle check result for 98004f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[shourya035]

Flaky tests:

ClusterManagerDisruptionIT.testIsolateClusterManagerAndVerifyClusterStateConsensus #12095

IndicesStoreIntegrationIT.testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}} #12788

[github-actions]

❌ Gradle check result for e4ce087: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for bc6d493: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 455bc59: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for 1179072: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4d37794: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a19c5cd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[shourya035]

❌ Gradle check result for a19c5cd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

❌ Gradle check result for 4d37794: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Flaky test being fixed through: #13457

[github-actions]

✅ Gradle check result for a19c5cd: SUCCESS