-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Using with AWS Elasticsearch Service
There is the github.com/olivere/elastic/aws/...
subpackage for using Elastic with AWS Elasticsearchservice. You should probably use the v4 signer from the official AWS SDK for Go, which is implemented in github.com/olivere/elastic/aws/v4
. Find an example of connecting to an AWS domain in github.com/olivere/elastic/recipes/aws-connect-v4
.
From our experience, you should simply disable sniffing and health checks when using AWS Elasticsearch Service as it will do load-balancing on the server-side. Here's an example code of how this could be done:
client, err := elastic.NewClient(
elastic.SetURL("..."),
elastic.SetSniff(false),
elastic.SetHealthcheck(false),
...
)
We also advise users to turn off compression, e.g. gzip. We're not exactly sure why but that seems to confuse the AWS Elasticsearch service.
When creating an Elastic client, you can specify an *http.Client
to use instead of the default client.
Since the release of v1.2.0 of the AWS SDK for Go (June 23rd, 2016), signing generic *http.Request
instances has become much easier. A wrapper for standard Go *http.Client
instances has been written that allows outgoing requests to Amazon Elasticsearch Service (or any other service) to be signed before the request is sent.
To install the package, simply go get github.com/sha1sum/aws_signing_client
import (
"github.com/aws/aws-sdk-go/aws/signer/v4"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/sha1sum/aws_signing_client"
"gopkg.in/olivere/elastic.v3"
)
func newElasticClient(creds *credentials.Credentials) (*elastic.Client, error) {
signer := v4.NewSigner(creds)
awsClient, err := aws_signing_client.New(signer, nil, "es", "us-east-1")
if err != nil {
return nil, err
}
return elastic.NewClient(
elastic.SetURL("https://my-aws-endpoint.us-east-1.es.amazonaws.com"),
elastic.SetScheme("https"),
elastic.SetHttpClient(awsClient),
elastic.SetSniff(false), // See note below
)
}
Note: These may no longer work with more recent versions of Elasticsearch and changes in AWS.
Thanks to @mthenw who wrote his own http.Transport
(see #317) based on https://github.com/smartystreets/go-aws-auth:
import (
awsauth "github.com/smartystreets/go-aws-auth"
)
...
type AWSSigningTransport struct {
HTTPClient *http.Client
Credentials awsauth.Credentials
}
// RoundTrip implementation
func (a AWSSigningTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return a.HTTPClient.Do(awsauth.Sign4(req, a.Credentials))
}
Usage:
signingTransport := AWSSigningTransport{
Credentials: awsauth.Credentials{
AccessKeyID: os.Getenv("AWS_ACCESS_KEY"),
SecretAccessKey: os.Getenv("AWS_SECRET_KEY"),
},
HTTPClient: http.DefaultClient,
}
signingClient := &http.Client{Transport: http.RoundTripper(signingTransport)}
return elastic.NewClient(
elastic.SetURL(...),
elastic.SetScheme("https"),
elastic.SetHttpClient(signingClient),
elastic.SetSniff(false),
)
Another library by @edoardo849 uses a RoundTripper
that can be tweaked by adding custom timeouts etc... The library is meant to be used within the Apex.run framework in a Lambda function. It also adds an optional feature to log the http request and response for debugging purposes.
Usage:
import (
"github.com/edoardo849/apex-aws-signer"
"github.com/apex/log"
"github.com/aws/aws-sdk-go/service/elasticsearchservice"
"github.com/aws/aws-sdk-go/aws/session"
"gopkg.in/olivere/elastic.v3"
)
// Example For ElasticSearch
transport := signer.NewTransport(session.New(&aws.Config{Region: aws.String("aws-region")}), elasticsearchservice.ServiceName)
// This is optional, "ctx" is the *apex.Context
transport.Logger = log.WithField("requestID", ctx.RequestID)
httpClient := &http.Client{
Transport: transport,
}
// Use the client with Olivere's elastic client
client, err := elastic.NewClient(
elastic.SetSniff(false),
elastic.SetURL("your-aws-es-endpoint"),
elastic.SetScheme("https"),
elastic.SetHttpClient(httpClient),
)