Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport] #324 #335

Merged
merged 1 commit into from
Jan 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@
Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

## [Unreleased]

### Added
- Github workflow for changelog verification ([#239](https://github.com/opensearch-project/opensearch-java/pull/239))
- Github workflow for dependabot PRs ([#247](https://github.com/opensearch-project/opensearch-java/pull/247))
- Introduce new OpenSearchTransport based on Apache HttpClient 5 ([#328](https://github.com/opensearch-project/opensearch-java/pull/328))
- Add Github workflow for changelog verification ([#239](https://github.com/opensearch-project/opensearch-java/pull/239))
- Add Github workflow for dependabot PRs ([#247](https://github.com/opensearch-project/opensearch-java/pull/247))
- Add support for signing service name in AwsSdk2Transport ([#324](https://github.com/opensearch-project/opensearch-java/pull/324))
- Add new OpenSearchTransport based on Apache HttpClient 5 ([#328](https://github.com/opensearch-project/opensearch-java/pull/328))

### Dependencies
- Bumps `grgit-gradle` from 4.0.1 to 5.0.0
Expand All @@ -16,19 +18,19 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Update tests to use JUnit's Assert ([#244](https://github.com/opensearch-project/opensearch-java/pull/244))
- Add support to parse sub-aggregations from filter/nested aggregations ([#234](https://github.com/opensearch-project/opensearch-java/pull/234))
- Add timeout and throttle to the jenkins workflows ([#231](https://github.com/opensearch-project/opensearch-java/pull/231))
- Updating maintainers, admins and documentation ([#248](https://github.com/opensearch-project/opensearch-java/pull/248))
- Update maintainers, admins and documentation ([#248](https://github.com/opensearch-project/opensearch-java/pull/248))
- Update Gradle to 7.6 ([#311](https://github.com/opensearch-project/opensearch-java/pull/311))

### Deprecated

### Removed
- Removed support for unsupported dynamic_templates in bulk ([#276](https://github.com/opensearch-project/opensearch-java/pull/276))
- Remove support for unsupported dynamic_templates in bulk ([#276](https://github.com/opensearch-project/opensearch-java/pull/276))

### Fixed
- Make ChildrenAggregate as a SingleBucketAggregate ([#306](https://github.com/opensearch-project/opensearch-java/pull/306))
- Fix /_nodes/stats, /_nodes/info throwing serialization error ([#315](https://github.com/opensearch-project/opensearch-java/pull/315))
- Fix AwsSdk2TransportOptions.responseCompression ([#322](https://github.com/opensearch-project/opensearch-java/pull/322))

### Security


[Unreleased]: https://github.com/opensearch-project/opensearch-java/compare/2.0...HEAD
15 changes: 15 additions & 0 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- [Run Tests](#run-tests)
- [Unit Tests](#unit-tests)
- [Integration Tests](#integration-tests)
- [AWS Transport Integration Tests](#aws-transport-integration-tests)
- [Use an Editor](#use-an-editor)
- [IntelliJ IDEA](#intellij-idea)
- [Visual Studio Code](#visual-studio-code)
Expand Down Expand Up @@ -64,6 +65,20 @@ Run integration tests after starting OpenSearch cluster:
./gradlew clean integrationTest
```

#### AWS Transport Integration Tests

To run integration tests for the AWS transport client, ensure working AWS credentials and specify your OpenSearch domain and region as follows:

```
./gradlew integrationTest --tests "*AwsSdk2*" -Dtests.awsSdk2support.domainHost=search-...us-west-2.es.amazonaws.com -Dtests.awsSdk2support.domainRegion=us-west-2
```

For OpenSearch Serverless, change the signing service name.

```
./gradlew integrationTest --tests "*AwsSdk2*" -Dtests.awsSdk2support.domainHost=....us-west-2.aoss.amazonaws.com -Dtests.awsSdk2support.domainRegion=us-west-2 -Dtests.awsSdk2support.serviceName=aoss
```

## Use an Editor

### IntelliJ IDEA
Expand Down
35 changes: 32 additions & 3 deletions USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,19 @@

- [User Guide](#user-guide)
- [Sample data](#sample-data)
- [IndexData class](#indexdata-class)
- [Create a client](#create-a-client)
- [Create a client using `RestClientTransport`](#create-a-client-using-restclienttransport)
- [Create a client using `ApacheHttpClient5Transport`](#create-a-client-using-apachehttpclient5transport)
- [Create an index](#create-an-index)
- [Index data](#index-data)
- [Search for the document](#search-for-the-document)
- [Search for the documents](#search-for-the-documents)
- [Search documents using a match query](#search-documents-using-a-match-query)
- [Aggregations](#aggregations)
- [Delete the document](#delete-the-document)
- [Delete the index](#delete-the-index)
- [Aggregations](#aggregations)
- [Using different transport options](#using-different-transport-options)
- [Amazon Managed OpenSearch](#amazon-managed-opensearch)

## Sample data

Expand Down Expand Up @@ -156,4 +160,29 @@ client.delete(d -> d.index(index).id("1"));
```java
DeleteIndexRequest deleteIndexRequest = new DeleteRequest.Builder().index(index).build();
DeleteIndexResponse deleteIndexResponse = client.indices().delete(deleteIndexRequest);
```
```

# Using different transport options

## Amazon Managed OpenSearch

Use `AwsSdk2Transport` to make requests to Amazon Managed OpenSearch.

```java
SdkHttpClient httpClient = ApacheHttpClient.builder().build();

OpenSearchClient client = new OpenSearchClient(
new AwsSdk2Transport(
httpClient,
"search-...us-west-2.es.amazonaws.com", // OpenSearch endpoint, without https://
"es" // signing service name
Region.US_WEST_2, // signing service region
AwsSdk2TransportOptions.builder().build()
)
);

InfoResponse info = client.info();
System.out.println(info.version().distribution() + ": " + info.version().number());

httpClient.close();
```
2 changes: 2 additions & 0 deletions java-client/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,8 @@ val integrationTest = task<Test>("integrationTest") {
systemProperty("password", System.getProperty("password", "admin"))
systemProperty("tests.awsSdk2support.domainHost",
System.getProperty("tests.awsSdk2support.domainHost", null))
systemProperty("tests.awsSdk2support.serviceName",
System.getProperty("tests.awsSdk2support.serviceName", "es"))
systemProperty("tests.awsSdk2support.domainRegion",
System.getProperty("tests.awsSdk2support.domainRegion", "us-east-1"))
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,34 +60,34 @@

/**
* Implementation of the OpenSearchTransport interface that sends signed requests using
* the AWS v2 SDK HTTP clients, to connect to an AWS OpenSearch service using IAM authentication
* the AWS v2 SDK HTTP clients, to connect to an AWS OpenSearch service using IAM authentication.
*/
public class AwsSdk2Transport implements OpenSearchTransport {
/**
* By default, requests that exceed this size will be automatically compressed.
* {@link AwsSdk2TransportOptions} can be used to override this setting or disable compresson.
* {@link AwsSdk2TransportOptions} can be used to override this setting or disable compression.
*/
public static final Integer DEFAULT_REQUEST_COMPRESSION_SIZE = 8192;

private static final byte[] NO_BYTES = new byte[0];
private final SdkHttpClient httpClient;
private final SdkAsyncHttpClient asyncHttpClient;
private final String host;
private final String signingServiceName;
private final Region signingRegion;
private final JsonpMapper defaultMapper;
private final AwsSdk2TransportOptions transportOptions;

/**
* Create an {@link OpenSearchTransport} with a SYNCHRONOUS AWS Http client
* Create an {@link OpenSearchTransport} with a synchronous AWS HTTP client.
* <p>
* Note that asynchronous OpenSearch requests sent through this transport will be dispatched
* *synchronously* on the calling thread.
*
* @param httpClient HTTP client to use for OpenSearch requests
* @param host The fully qualified domain name to connect to
* @param signingRegion The AWS region for which requests will be signed. This should typically match
* the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* @param httpClient HTTP client to use for OpenSearch requests.
* @param host The fully qualified domain name to connect to.
* @param signingRegion The AWS region for which requests will be signed. This should typically match the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* {@link AwsSdk2TransportOptions#builder()} and use these to specify non-default credentials,
* compression options, etc.
*/
Expand All @@ -96,20 +96,42 @@ public AwsSdk2Transport(
@Nonnull String host,
@Nonnull Region signingRegion,
@CheckForNull AwsSdk2TransportOptions options) {
this(httpClient, null, host, signingRegion, options);
this(httpClient, null, host, "es", signingRegion, options);
}

/**
* Create an {@link OpenSearchTransport} with an ASYNCHRONOUS AWS Http client
* Create an {@link OpenSearchTransport} with a synchronous AWS HTTP client.
* <p>
* Note that asynchronous OpenSearch requests sent through this transport will be dispatched
* *synchronously* on the calling thread.
*
* @param httpClient HTTP client to use for OpenSearch requests.
* @param host The fully qualified domain name to connect to.
* @param signingServiceName The AWS signing service name, one of `es` (Amazon OpenSearch) or `aoss` (Amazon OpenSearch Serverless).
* @param signingRegion The AWS region for which requests will be signed. This should typically match the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* {@link AwsSdk2TransportOptions#builder()} and use these to specify non-default credentials,
* compression options, etc.
*/
public AwsSdk2Transport(
@Nonnull SdkHttpClient httpClient,
@Nonnull String host,
@Nonnull String signingServiceName,
@Nonnull Region signingRegion,
@CheckForNull AwsSdk2TransportOptions options) {
this(httpClient, null, host, signingServiceName, signingRegion, options);
}

/**
* Create an {@link OpenSearchTransport} with an asynchronous AWS HTTP client
* <p>
* Note that synchronous OpenSearch requests sent through this transport will be dispatched
* using the asynchronous client, but the calling thread will block until they are complete.
*
* @param asyncHttpClient HTTP client to use for OpenSearch requests
* @param host The target host
* @param signingRegion The AWS region for which requests will be signed. This should typically match
* the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* @param asyncHttpClient HTTP client to use for OpenSearch requests.
* @param host The target host.
* @param signingRegion The AWS region for which requests will be signed. This should typically match region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* {@link AwsSdk2TransportOptions#builder()} and use these to specify non-default credentials,
* compression options, etc.
*/
Expand All @@ -118,7 +140,53 @@ public AwsSdk2Transport(
@Nonnull String host,
@Nonnull Region signingRegion,
@CheckForNull AwsSdk2TransportOptions options) {
this(null, asyncHttpClient, host, signingRegion, options);
this(null, asyncHttpClient, host, "es", signingRegion, options);
}

/**
* Create an {@link OpenSearchTransport} with an asynchronous AWS HTTP client.
* <p>
* Note that synchronous OpenSearch requests sent through this transport will be dispatched
* using the asynchronous client, but the calling thread will block until they are complete.
*
* @param asyncHttpClient HTTP client to use for OpenSearch requests.
* @param host The target host.
* @param signingServiceName The AWS signing service name, one of `es` (Amazon OpenSearch) or `aoss` (Amazon OpenSearch Serverless).
* @param signingRegion The AWS region for which requests will be signed. This should typically match the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* {@link AwsSdk2TransportOptions#builder()} and use these to specify non-default credentials,
* compression options, etc.
*/
public AwsSdk2Transport(
@Nonnull SdkAsyncHttpClient asyncHttpClient,
@Nonnull String host,
@Nonnull String signingServiceName,
@Nonnull Region signingRegion,
@CheckForNull AwsSdk2TransportOptions options) {
this(null, asyncHttpClient, host, signingServiceName, signingRegion, options);
}

/**
* Create an {@link OpenSearchTransport} with both synchronous and asynchronous AWS HTTP clients.
* <p>
* The synchronous client will be used for synchronous OpenSearch requests, and the asynchronous client
* will be used for asynchronous HTTP requests.
*
* @param httpClient HTTP client to use for OpenSearch requests.
* @param asyncHttpClient HTTP client to use for synchronous OpenSearch requests.
* @param host The fully qualified domain name to connect to.
* @param signingRegion The AWS region for which requests will be signed. This should typically match the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* {@link AwsSdk2TransportOptions#builder()} and use these to specify non-default credentials,
* compression options, etc.
*/
public AwsSdk2Transport(
@CheckForNull SdkHttpClient httpClient,
@CheckForNull SdkAsyncHttpClient asyncHttpClient,
@Nonnull String host,
@Nonnull Region signingRegion,
@CheckForNull AwsSdk2TransportOptions options) {
this(httpClient, asyncHttpClient, host, "es", signingRegion, options);
}

/**
Expand All @@ -127,19 +195,20 @@ public AwsSdk2Transport(
* The synchronous client will be used for synchronous OpenSearch requests, and the asynchronous client
* will be used for asynchronous HTTP requests.
*
* @param httpClient HTTP client to use for OpenSearch requests
* @param asyncHttpClient HTTP client to use for synchronous OpenSearch requests
* @param host The fully qualified domain name to connect to
* @param signingRegion The AWS region for which requests will be signed. This should typically match
* the region in `host`.
* @param options Options that apply to all requests. Can be null. Create with
* @param httpClient HTTP client to use for OpenSearch requests.
* @param asyncHttpClient HTTP client to use for synchronous OpenSearch requests.
* @param host The fully qualified domain name to connect to.
* @param signingRegion The AWS region for which requests will be signed. This should typically match the region in `host`.
* @param signingServiceName The AWS signing service name, one of `es` (Amazon OpenSearch) or `aoss` (Amazon OpenSearch Serverless).
* @param options Options that apply to all requests. Can be null. Create with
* {@link AwsSdk2TransportOptions#builder()} and use these to specify non-default credentials,
* compression options, etc.
*/
public AwsSdk2Transport(
@CheckForNull SdkHttpClient httpClient,
@CheckForNull SdkAsyncHttpClient asyncHttpClient,
@Nonnull String host,
@Nonnull String signingServiceName,
@Nonnull Region signingRegion,
@CheckForNull AwsSdk2TransportOptions options) {
if (httpClient == null && asyncHttpClient == null)
Expand All @@ -150,6 +219,7 @@ public AwsSdk2Transport(
this.httpClient = httpClient;
this.asyncHttpClient = asyncHttpClient;
this.host = host;
this.signingServiceName = signingServiceName;
this.signingRegion = signingRegion;
this.transportOptions = options != null ? options : AwsSdk2TransportOptions.builder().build();
this.defaultMapper = Optional.ofNullable(options)
Expand Down Expand Up @@ -314,7 +384,7 @@ private <RequestT> SdkHttpFullRequest prepareRequest(

Aws4SignerParams signerParams = Aws4SignerParams.builder()
.awsCredentials(credentials.resolveCredentials())
.signingName("es")
.signingName(this.signingServiceName)
.signingRegion(signingRegion)
.build();
return Aws4Signer.create().sign(req.build(), signerParams);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,13 +83,15 @@ protected OpenSearchClient getClient(
transport = new AwsSdk2Transport(
getAsyncHttpClient(),
getTestClusterHost(),
getTestClusterServiceName(),
getTestClusterRegion(),
getTransportOptions().build()
);
} else {
transport = new AwsSdk2Transport(
getHttpClient(),
getTestClusterHost(),
getTestClusterServiceName(),
getTestClusterRegion(),
getTransportOptions().build()
);
Expand All @@ -107,13 +109,15 @@ protected OpenSearchAsyncClient getAsyncClient(
transport = new AwsSdk2Transport(
getAsyncHttpClient(),
getTestClusterHost(),
getTestClusterServiceName(),
getTestClusterRegion(),
getTransportOptions().build()
);
} else {
transport = new AwsSdk2Transport(
getHttpClient(),
getTestClusterHost(),
getTestClusterServiceName(),
getTestClusterRegion(),
getTransportOptions().build()
);
Expand All @@ -131,13 +135,15 @@ protected OpenSearchIndicesClient getIndexesClient(
transport = new AwsSdk2Transport(
getAsyncHttpClient(),
getTestClusterHost(),
getTestClusterServiceName(),
getTestClusterRegion(),
getTransportOptions().build()
);
} else {
transport = new AwsSdk2Transport(
getHttpClient(),
getTestClusterHost(),
getTestClusterServiceName(),
getTestClusterRegion(),
getTransportOptions().build()
);
Expand All @@ -150,6 +156,11 @@ protected String getTestClusterHost() {
return cluster;
}

protected String getTestClusterServiceName() {
String cluster = System.getProperty("tests.awsSdk2support.serviceName");
return cluster;
}

protected Region getTestClusterRegion() {
String region = System.getProperty("tests.awsSdk2support.domainRegion");
return region != null ? Region.of(region) : Region.US_EAST_1;
Expand Down