Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding x-amz-content-sha256 header for signed requests #339

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fix version and build ([#254](https://github.com/opensearch-project/opensearch-java/pull/254))
- Update Gradle to 7.6 ([#309](https://github.com/opensearch-project/opensearch-java/pull/309))
- Prevent SPI calls at runtime ([#293](https://github.com/opensearch-project/opensearch-java/pull/293))
- Add support for OpenSearch Serverless ([#339](https://github.com/opensearch-project/opensearch-java/pull/339))

### Deprecated

Expand Down
4 changes: 2 additions & 2 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@ Run integration tests after starting OpenSearch cluster:

#### AWS Transport Integration Tests

To run integration tests for the AWS transport client, ensure working AWS credentials and specify your OpenSearch domain and region as follows:
To run integration tests for the AWS transport client, ensure working AWS credentials in `/.aws/credentials` and specify your OpenSearch domain and region as follows:

```
./gradlew integrationTest --tests "*AwsSdk2*" -Dtests.awsSdk2support.domainHost=search-...us-west-2.es.amazonaws.com -Dtests.awsSdk2support.domainRegion=us-west-2
./gradlew integrationTest --tests "*AwsSdk2*" -Dtests.awsSdk2support.domainHost=search-...us-west-2.es.amazonaws.com -Dtests.awsSdk2support.domainRegion=us-west-2 -Dtests.awsSdk2support.serviceName=es
```

For OpenSearch Serverless, change the signing service name.
Expand Down
6 changes: 4 additions & 2 deletions USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
- [Sample data](#sample-data)
- [IndexData class](#indexdata-class)
- [Create a client](#create-a-client)
- [Create a client using `RestClientTransport`](#create-a-client-using-restclienttransport)
- [Create a client using `ApacheHttpClient5Transport`](#create-a-client-using-apachehttpclient5transport)
- [Create an index](#create-an-index)
- [Index data](#index-data)
- [Search for the documents](#search-for-the-documents)
Expand Down Expand Up @@ -175,7 +177,7 @@ DeleteIndexResponse deleteIndexResponse = client.indices().delete(deleteIndexReq

## Amazon Managed OpenSearch

Use `AwsSdk2Transport` to make requests to Amazon Managed OpenSearch.
Use `AwsSdk2Transport` to make requests to Amazon Managed OpenSearch and OpenSearch Serverless.

```java
SdkHttpClient httpClient = ApacheHttpClient.builder().build();
Expand All @@ -184,7 +186,7 @@ OpenSearchClient client = new OpenSearchClient(
new AwsSdk2Transport(
httpClient,
"search-...us-west-2.es.amazonaws.com", // OpenSearch endpoint, without https://
"es" // signing service name
"es" // signing service name, use "aoss" for OpenSearch Serverless
Region.US_WEST_2, // signing service region
AwsSdk2TransportOptions.builder().build()
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,9 @@ private <RequestT> SdkHttpFullRequest prepareRequest(
}
req.putHeader("Content-Length", String.valueOf(body.getContentLength()));
req.contentStreamProvider(body::getInputStream);
// To add the "X-Amz-Content-Sha256" header, it needs to set as required.
// It is a required header for Amazon OpenSearch Serverless.
req.putHeader("x-amz-content-sha256", "required");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it took me a while to find too. We use Aws4Signer which does not have the sign method overriden like Aws4UnsignedPayloadSigner. According to documentation, the Aws4UnsignedPayloadSigner is similar to Aws4Signer but just adds UNSIGNED-PAYLOAD when protocol is HTTPS. Can we use Aws4UnsignedPayloadSigner?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it works we sure can I think.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried the Aws4UnsignedPayloadSigner but it does not work for Amazon OpenSearch Service since it signs the payload with UNSIGNED_PAYLOAD over https protocol. Works for Amazon OpenSearch Serverless. I can PR this change on their repo may be as a new signer class but for now looks like we might have to use the hard-coded "required".

}

boolean responseCompression = Optional.ofNullable(options)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,7 @@
import org.junit.Assert;
import org.opensearch.client.opensearch.OpenSearchAsyncClient;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch._types.OpType;
import org.opensearch.client.opensearch._types.OpenSearchException;
import org.opensearch.client.opensearch._types.Refresh;
import org.opensearch.client.opensearch.core.IndexRequest;
import org.opensearch.client.opensearch.core.IndexResponse;
import org.opensearch.client.opensearch.core.SearchResponse;
Expand Down Expand Up @@ -51,11 +49,14 @@ void testClient(boolean async) throws Exception {
final OpenSearchClient client = getClient(async, null, null);

SimplePojo doc1 = new SimplePojo("Document 1", "The text of document 1");
addDoc(client, "id1", doc1, false);
addDoc(client, "id1", doc1);
SimplePojo doc2 = new SimplePojo("Document 2", "The text of document 2");
addDoc(client, "id2", doc2, false);
addDoc(client, "id2", doc2);
SimplePojo doc3 = getLongDoc("Long Document 3", 1000000);
addDoc(client, "id3", doc3, true);
addDoc(client, "id3", doc3);

// wait for the document to index
Thread.sleep(1000);

SearchResponse<SimplePojo> response = query(client, "NotPresent", null);
Assert.assertEquals(0, response.hits().hits().size());
Expand All @@ -77,12 +78,15 @@ void testClientAsync(boolean async) throws Exception {
final OpenSearchAsyncClient client = getAsyncClient(async, null, null);

SimplePojo doc1 = new SimplePojo("Document 1", "The text of document 1");
CompletableFuture<IndexResponse> add1 = addDoc(client, "id1", doc1, false);
CompletableFuture<IndexResponse> add1 = addDoc(client, "id1", doc1);
SimplePojo doc2 = new SimplePojo("Document 2", "The text of document 2");
CompletableFuture<IndexResponse> add2 = addDoc(client, "id2", doc2, false);
CompletableFuture<IndexResponse> add2 = addDoc(client, "id2", doc2);
SimplePojo doc3 = getLongDoc("Long Document 3", 1000000);
CompletableFuture<IndexResponse> add3 = CompletableFuture.allOf(add1, add2).thenCompose(
unused -> addDoc(client, "id3", doc3, true));
unused -> addDoc(client, "id3", doc3));

// wait for the document to index
Thread.sleep(1000);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a recipe for flakyness ....


List<SearchResponse<SimplePojo>> results = add3.thenCompose(unused -> {
CompletableFuture<SearchResponse<SimplePojo>> r1 = query(client, "NotPresent", null);
Expand All @@ -107,29 +111,21 @@ void testClientAsync(boolean async) throws Exception {
}


private void addDoc(OpenSearchClient client, String id, SimplePojo doc, boolean wait) throws Exception {
private void addDoc(OpenSearchClient client, String id, SimplePojo doc) throws Exception {
IndexRequest.Builder<SimplePojo> req = new IndexRequest.Builder<SimplePojo>()
.index(TEST_INDEX)
.document(doc)
.id(id)
.opType(OpType.Index);
if (wait) {
req.refresh(Refresh.WaitFor);
}
.id(id);
client.index(req.build());
}

private CompletableFuture<IndexResponse> addDoc(
OpenSearchAsyncClient client, String id, SimplePojo doc, boolean wait
OpenSearchAsyncClient client, String id, SimplePojo doc
) {
IndexRequest.Builder<SimplePojo> req = new IndexRequest.Builder<SimplePojo>()
.index(TEST_INDEX)
.document(doc)
.id(id)
.opType(OpType.Index);
if (wait) {
req.refresh(Refresh.WaitFor);
}
.id(id);
try {
return client.index(req.build());
} catch (Exception e) {
Expand Down