Remove remote translog settings to make all remote indices translog enabled

[linuxpi]

Description

• Removes the option to create remote indices without translog remote store enabled. All remote indices created should be with segment and translog remote store.
• Also, refactors segment store settings to move it in a nested structure under "segment" key

Sample json to create remote translog index:

Before this change

{               
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "replication": {
        "type": "SEGMENT"
      },
      "remote_store": {
        "enabled": true,
        "repository": "my-fs-repository", "translog": {"enabled": true, "repository": "my-fs-repository", "buffer_interval": "650ms"}
      }
    }
  }
}

After this change

{               
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "replication": {
        "type": "SEGMENT"
      },
      "remote_store": {
        "enabled": true,
        "segment": {"repository": "my-segment-repository"},
        "translog": {"repository": "my-translog-repository",  "buffer_interval": "650ms"}
      }
    }
  }
}

• A lot of tests were updated which just relied on remote segment or were using remote translog settings, leading to large no files being touched.
• Please do take extra care while reviewing server/src/test/java/org/opensearch/index/shard/SegmentReplicationIndexShardTests.java changes.

Related Issues

Resolves #8769

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20278/
• CommitID: 1edc46c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20279/
• CommitID: f281cb9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20282/
• CommitID: 31a3104
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20391/
• CommitID: 4999d3e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20415/
• CommitID: 5b56edb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20440/
• CommitID: 3b1f449
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20441/
• CommitID: 7f77b5a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20478/
• CommitID: 01809a0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20479/
• CommitID: 4fbf5dc
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20498/
• CommitID: 9b60878
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20502/
• CommitID: ed153bb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20505/
• CommitID: 92eefff
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20507/
• CommitID: 08441f3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20509/
• CommitID: 4d3d7a0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20510/
• CommitID: 98c77fc
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20512/
• CommitID: 48ab547
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[linuxpi]

Flaky tests:

Failing in other builds as well

• RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush test failure #8786
• Refactoring default and best_compression codec to their algorithm name #8745
• Disallowing compression level for lz4 and best_compression codec #8737

   [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush/)
    [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush_2/)
    [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush_3/)
    [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush_4/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary_2/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary_3/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary_4/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush_2/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush_3/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush_4/)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/20598/
• CommitID: 486964b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8719 (954a5fa) into main (6cc8da5) will increase coverage by 0.28%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #8719      +/-   ##
============================================
+ Coverage     70.61%   70.89%   +0.28%     
- Complexity    56909    57148     +239     
============================================
  Files          4771     4771              
  Lines        270253   270235      -18     
  Branches      39502    39497       -5     
============================================
+ Hits         190837   191594     +757     
+ Misses        63250    62540     -710     
+ Partials      16166    16101      -65     

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	93.18% <ø> (ø)
...pensearch/common/settings/IndexScopedSettings.java	100.00% <ø> (ø)
...in/java/org/opensearch/indices/IndicesService.java	70.86% <ø> (+1.31%)	⬆️
...org/opensearch/cluster/metadata/IndexMetadata.java	86.16% <100.00%> (+0.46%)	⬆️
...h/cluster/metadata/MetadataCreateIndexService.java	80.07% <100.00%> (+0.24%)	⬆️
.../main/java/org/opensearch/index/IndexSettings.java	85.10% <100.00%> (-0.04%)	⬇️

... and 497 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicationAfterForceMerge

• URL: https://build.ci.opensearch.org/job/gradle-check/20612/
• CommitID: ef7f650
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20640/
• CommitID: 2ab9c84
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

• URL: https://build.ci.opensearch.org/job/gradle-check/20644/
• CommitID: 954a5fa
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8719-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c47ec1ba79757ff3fdb7ac59c69209c0dcaa3139
# Push it to GitHub
git push --set-upstream origin backport/backport-8719-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8719-to-2.x.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8719-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c47ec1ba79757ff3fdb7ac59c69209c0dcaa3139
# Push it to GitHub
git push --set-upstream origin backport/backport-8719-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8719-to-2.x.