Integrate IO Based AdmissionController to AdmissionControl Framework

[ajaymovva]

Description

• If IO usage percent is breached the threshold we are applying admission controller and start rejecting requests till it back to normal.
• IO Usage Calculation Based on Integrate IO Usage Tracker to the Resource Usage Collector Service and Emit IO Usage Stats #11880
• IO Based AdmissionControl is only available for Linux OS.

Node Stats Output On Linux

"admission_control": {
    "global_io_usage": {
        "transport": {
            "rejection_count": {
                "search": 8
            }
        }
    },
    "global_cpu_usage": {
        "transport": {
            "rejection_count": {}
        }
    }
}

Node Stats Output On Mac

"admission_control": {
    "global_cpu_usage": {
        "transport": {
            "rejection_count": {
                "search": 4
            }
        }
    }
}

Fixed the flaky tests in this PR: #12583. Modification Only in AdmissionControlSingleNodeTests Test File.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for d4ba448: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 2032928

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❌ Gradle check result for 651c2c5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ajaymovva]

❌ Gradle check result for 4717c53: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Test org.opensearch.transport.RemoteClusterClientTests.testConnectAndExecuteRequest is continously failing.
Issue link: #12338

[github-actions]

❌ Gradle check result for 2032928: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12702-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fbcc0d94d1183e15ea3c74c72894c527460b2ce3
# Push it to GitHub
git push --set-upstream origin backport/backport-12702-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12702-to-2.x.

[reta]

@ajaymovva @sachinpkale this pull request was merged with failing Gradle checks and now we have build failures across whole repository, I am reverting this pull request

[ajaymovva]

@reta Even before I merged the PR I saw this is failing for other PR's : #12709.
We retried multiple times to autofix it but it was not.

[ajaymovva]

I have done backport to 2.x as well but I didn't faced this falky test issue there: #12716

[reta]

@reta Even before I merged the PR I saw this is failing for other PR's : #12709.

Thanks @ajaymovva , I am trying to pinpoint the cause of this regression, other PRs (like #12709) are rebased on top of main, with this change included

[ajaymovva]

other PRs (like #12709) are rebased on top of main, with this change included

@reta Atleast at this point where we saw the same test failure: #12709 (comment) I didn't even merged my changes into main.

[reta]

@ajaymovva I am still looking, the test was flaky before, but the nature and the frequencies of the failures have changed: looking what is causing it, if it is not this pull request - nothing will be reverted for sure.