Update the indexRandom function to create more segments

[neetikasinghal]

Description

This is to make sure that the index random function which is used in multiple classes has creation of multiple segments to test the concurrent search code path. Without this change there are 848 IT tests that are running with slice count<=1.

With this change the number of tests having slice<=1 is down to 325. This pr can also be used to add index random in the other tests not running with the higher slice count. e.g. 6157d4e

Related Issues

Resolves #10310

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[neetikasinghal]

@sohami @reta please review.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26362/
• CommitID: 8643f74
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 0837fe8

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26627/
• CommitID: 9d153ad
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29006/
• CommitID: 5ca93a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[neetikasinghal]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29008/
• CommitID: 753008b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

#10902 - flaky
https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+RemoteStoreStatsIT - many tests are flaky

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.reindex.DeleteByQueryBasicTests.testDeleteByQueryOnReadOnlyAllowDeleteIndex

• URL: https://build.ci.opensearch.org/job/gradle-check/29023/
• CommitID: 0837fe8
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10247 (0837fe8) into main (44a9f18) will increase coverage by 0.45%.
Report is 10 commits behind head on main.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main   #10247      +/-   ##
============================================
+ Coverage     71.22%   71.68%   +0.45%     
- Complexity    58721    59108     +387     
============================================
  Files          4872     4872              
  Lines        276682   276682              
  Branches      40219    40219              
============================================
+ Hits         197070   198326    +1256     
+ Misses        63163    62174     -989     
+ Partials      16449    16182     -267     

see 453 files with indirect coverage changes

[sohami]

@reta LGTM. Please let me know if you have any open comments.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/29165/
• CommitID: 0837fe8

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10247-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bc74731a537c461cd1d7666bc3be82fbe18e143d
# Push it to GitHub
git push --set-upstream origin backport/backport-10247-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-10247-to-2.x.