[Backport 2.x] Allow customization of netty channel handles before and during decompression

[cwperks]

Manual backport of #10261 to 2.x.

There are differences in the netty pipeline between main and 2.x (mainly around http2). This PR updates the test to accommodate for the differences.

[github-actions]

Compatibility status:

Checks if related components are compatible with change e7d0bed

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27244/
• CommitID: 0bb45a2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[cwperks]

@peternied Can you re-run the CI on this PR and get it merged to the 2.x line?

[peternied]

@cwperks Could you '+1' existing or create issues for the random failures associated with the first CI run [1]? I've re-trigged the CI run

Test	Params
org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test	{yaml=pit/10_basic/Delete all}
org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
org.opensearch.search.scroll.SearchScrollWithFailingNodesIT.testScanScrollWithShardExceptions	{p0={"search.concurrent_segment_search.enabled":"true"}}
org.opensearch.search.scroll.SearchScrollWithFailingNodesIT.testScanScrollWithShardExceptions	{p0={"search.concurrent_segment_search.enabled":"false"}}

• [1] https://build.ci.opensearch.org/job/gradle-check/27244/testReport/

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27493/
• CommitID: 0bb45a2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[peternied]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':distribution:bwc:minor:buildBwcLinuxTar'.
> Building 2.10.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/minor/build/bwc/checkout-2.10/distribution/archives/linux-tar/build/distributions/opensearch-min-2.10.0-SNAPSHOT-linux-x64.tar.gz

[cwperks]

@peternied all of the flaky tests have corresponding issues associated with them. I added +1 or a comment on each issue.

[DarshitChanpura]

Can we get a prioritized review on this? This is blocking 2.12 version bump: opensearch-project/security#3498 which in turn is blocking build on main due to backward-compatibility test expecting 2.12

[peternied]

@DarshitChanpura I'm be monitoring this

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

• URL: https://build.ci.opensearch.org/job/gradle-check/27496/
• CommitID: e7d0bed
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.