Add async read support for S3 plugin #9694

kotwanikunal · 2023-09-01T16:33:15Z

Description

Add async read support for S3 plugin added as a part of Add async blob read and download support using multiple streams #9592

Related Issues

Partially resolves #9031

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-01T16:47:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24128/
CommitID: 8e1fc48
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-01T17:05:32Z

Compatibility status:

Checks if related components are compatible with change 8fbb6bd

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

github-actions · 2023-09-01T17:32:56Z

Compatibility status:

Checks if related components are compatible with change 7963644

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-01T17:43:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/24135/
CommitID: 7963644

codecov · 2023-09-01T17:45:50Z

Codecov Report

Merging #9694 (8fbb6bd) into main (d2ca926) will decrease coverage by 0.06%.
The diff coverage is 89.58%.

@@             Coverage Diff              @@
##               main    #9694      +/-   ##
============================================
- Coverage     71.12%   71.07%   -0.06%     
+ Complexity    58146    58092      -54     
============================================
  Files          4824     4824              
  Lines        273915   273963      +48     
  Branches      39917    39921       +4     
============================================
- Hits         194819   194710     -109     
- Misses        62809    62927     +118     
- Partials      16287    16326      +39

Files Changed	Coverage Δ
...rg/opensearch/repositories/s3/S3BlobContainer.java	`78.19% <88.37%> (-0.21%)`	⬇️
...ensearch/repositories/s3/utils/HttpRangeUtils.java	`85.71% <100.00%> (+35.71%)`	⬆️

... and 428 files with indirect coverage changes

andrross

Let's add some tests here to S3BlobStoreRepositoryTests so we can test this functionality without mocking (the integ tests use minio to simulate S3).

Ultimately the tests should go in https://github.com/opensearch-project/OpenSearch/blob/0c839c3e35b4086acb88a276de9d10ece637d318/test/framework/src/main/java/org/opensearch/repositories/blobstore/OpenSearchBlobStoreRepositoryIntegTestCase.java but they can start in S3 since it is the only implementation at the moment. Alternatively you can add it to the base class and override the method in the other repositories with an @Ignore.

github-actions · 2023-09-06T23:28:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24791/
CommitID: 856763f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2023-09-06T23:31:05Z

* What went wrong:
Execution failed for task ':distribution:bwc:minor:buildBwcLinuxTar'.
> Building 2.10.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/minor/build/bwc/checkout-2.x/distribution/archives/linux-tar/build/distributions/opensearch-min-2.10.0-SNAPSHOT-linux-x64.tar.gz

* Try:

I'll wait for #9856

github-actions · 2023-09-06T23:38:37Z

Compatibility status:

Checks if related components are compatible with change 856763f

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-07T17:02:28Z

Compatibility status:

Checks if related components are compatible with change 3bde359

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-07T17:03:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24980/
CommitID: 3bde359
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2023-09-07T17:07:21Z

https://build.ci.opensearch.org/job/gradle-check/24980/

org.opensearch.search.profile.query.QueryProfilerIT.testProfileMatchesRegular {p0={"search.concurrent_segment_search.enabled":"true"} p1=true}
java.lang.NullPointerException: Cannot invoke "org.opensearch.action.search.SearchResponse.getFailedShards()" because "profileResponse" is null

#9869

github-actions · 2023-09-07T17:50:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/24999/
CommitID: 3bde359
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

CHANGELOG.md

github-actions · 2023-09-11T18:22:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/25328/
CommitID: 738c03b

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

github-actions · 2023-09-13T23:32:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/25536/
CommitID: 8fbb6bd

* Add async read support for S3 plugin Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Move functionality to S3BlobContainer Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 70a582f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add async read support for S3 plugin (cherry picked from commit 03ddc8a) * Move functionality to S3BlobContainer --------- (cherry picked from commit 70a582f) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add async read support for S3 plugin Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Move functionality to S3BlobContainer Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Add async read support for S3 plugin Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Move functionality to S3BlobContainer Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

* Add async read support for S3 plugin Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Move functionality to S3BlobContainer Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Add async read support for S3 plugin Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Move functionality to S3BlobContainer Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

kotwanikunal force-pushed the s3-multipart-main branch from 8e1fc48 to 7963644 Compare September 1, 2023 16:56

andrross reviewed Sep 1, 2023

View reviewed changes

kotwanikunal force-pushed the s3-multipart-main branch from 856763f to 3bde359 Compare September 7, 2023 16:21

andrross reviewed Sep 8, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

kotwanikunal force-pushed the s3-multipart-main branch from 3bde359 to 738c03b Compare September 11, 2023 17:31

kotwanikunal added 2 commits September 13, 2023 15:47

Add async read support for S3 plugin

3e745ed

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 03ddc8a) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Move functionality to S3BlobContainer

8fbb6bd

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

kotwanikunal force-pushed the s3-multipart-main branch from 738c03b to 8fbb6bd Compare September 13, 2023 22:47

github-actions bot added distributed framework enhancement Enhancement or improvement to existing feature or request labels Sep 13, 2023

kotwanikunal mentioned this pull request Sep 13, 2023

Pull requests to inherit labels from parent issues #8199

Closed

andrross approved these changes Sep 15, 2023

View reviewed changes

andrross merged commit 70a582f into opensearch-project:main Sep 15, 2023
13 checks passed

kkmr approved these changes Sep 15, 2023

View reviewed changes

kotwanikunal added the backport 2.x Backport to 2.x branch label Sep 18, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 18, 2023

[Backport 2.x] Add async read support for S3 plugin #10092

Merged

kotwanikunal deleted the s3-multipart-main branch September 21, 2023 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add async read support for S3 plugin #9694

Add async read support for S3 plugin #9694

kotwanikunal commented Sep 1, 2023 •

edited

Loading

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023 •

edited

Loading

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

codecov bot commented Sep 1, 2023 •

edited

Loading

andrross left a comment •

edited

Loading

github-actions bot commented Sep 6, 2023

kotwanikunal commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

kotwanikunal commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 13, 2023

Add async read support for S3 plugin #9694

Add async read support for S3 plugin #9694

Conversation

kotwanikunal commented Sep 1, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 1, 2023 • edited Loading

Codecov Report

andrross left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 13, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Sep 1, 2023 •

edited

Loading

github-actions bot commented Sep 1, 2023 •

edited

Loading

codecov bot commented Sep 1, 2023 •

edited

Loading

andrross left a comment •

edited

Loading