Fixes to prevent segment fault errors arising due to unexpected SDK b…

[vikasvb90]

Description

This PR adds guards against unexpected S3 SDK errors.

1. Guards against access to stream after callback from SDK. Cases were observed in which SDK was trying to access stream even after callback was invoked by it. This was leading to segment fault errors in segment upload path as MMap files were being removed from memory on close and attempts to read invalid references were happening by SDK. Also, JDK 17 is not handing these errors gracefully which was leading to JVM crash.
2. Multiple invocations to close was another cause of segment fault errors.

This is a forked PR from : #11327

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 78cfad7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for e8ef33c: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for d3d01bb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for e2c5146: SUCCESS

[codecov]

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (effc9bc) 71.46% compared to head (1d91527) 71.42%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
...n/annotation/processor/ApiAnnotationProcessor.java	88.18%	1 Missing and 14 partials ⚠️
...e/transfer/stream/OffsetRangeIndexInputStream.java	74.35%	9 Missing and 1 partial ⚠️
...on/blobstore/transfer/RemoteTransferContainer.java	71.42%	5 Missing and 1 partial ⚠️
...bstore/transfer/stream/OffsetRangeInputStream.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11334      +/-   ##
============================================
- Coverage     71.46%   71.42%   -0.05%     
+ Complexity    59176    59170       -6     
============================================
  Files          4903     4904       +1     
  Lines        277987   278147     +160     
  Branches      40382    40419      +37     
============================================
- Hits         198662   198657       -5     
- Misses        62805    62971     +166     
+ Partials      16520    16519       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for c671791: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 02637aa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 29b8222: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d2713b4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

Thanks @vikasvb90. Please rebase this against the latest in main to fix the bwc test failure.

[github-actions]

❕ Gradle check result for 1d91527: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.