Aws es signing

[JackFazackerley]

Adds AWS ElasticSearch request signing, closes #346, also merged #333 into my branch as I wanted Go modules and it was already created as a PR.

[JackFazackerley]

I've also just noticed that #350 was created yesterday. If required I can use that Go modules instead.

[k1rk]

any plans for this? would be great to have ability to use exporter with AWS elasticseach

[KyleMasterson]

@JackFazackerley I have taken these changes for a spin on EKS, but haven't had any luck actually reaching our AWS elasticsearch instance. Just wondering if you can confirm that this can support IRSA?

Example: https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

Any examples of how to achieve this in a deployment manifest or the like would be extremely helpful!

[JackFazackerley]

Sorry for only just getting back to you on this guys, only just noticed it.

@k1rk if you wanted to use it, my fork should work via the following commands

AWS_ACCESS_KEY_ID="ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="SECRET_ACCESS_KEY"
go run -mod=vendor main.go tls.go logger.go --es.uri="ES_URI" --log.level="debug" --es.aws --es.aws-region="YOUR_REGION"

Assuming that works you could build and use the binary.

@KyleMasterson From reading the docs it should be possible already if using my fork. As the docs say:

For this feature to work correctly, you’ll need to use an SDK version greater than or equal to the values listed below:

The Go SDK being used is v1.31.3 and the specified version on the docs is v1.23.13 so it should be possible. Note i'm not using IRSA, my credentials are imported via Vault.

If you still wanted an example of what my manifest looks like: (I've added environment vars in the manifest below to work with keys)

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: elasticsearch-exporter
  name: elasticsearch-exporter
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch-exporter
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: elasticsearch-exporter
    spec:
      containers:
        - command:
            - elasticsearch_exporter
            - --es.uri=YOUR_ES_URI
            - --es.all
            - --es.indices
            - --es.indices_settings
            - --es.shards
            - --es.snapshots
            - --es.timeout=30s
            - --web.listen-address=:9108
            - --web.telemetry-path=/metrics
          env:
            - name: AWS_ACCESS_KEY_ID
              value: YOUR_ACCESS_KEY_ID
            - name: AWS_SECRET_ACCESS_KEY
              value: YOUR_SECRET_ACCESS_KEY
            - name: ES_AWS
              value: "true"
            - name: ES_AWS_REGION
              value: YOUR_REGION
          image: elasticsearch-exporter:aws-es-signing
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/bash
                  - -c
                  - sleep 20
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 5
          name: elasticsearch-exporter
          ports:
            - containerPort: 9108
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
            initialDelaySeconds: 1
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 5
          resources: {}
          securityContext:
            capabilities:
              drop:
                - SETPCAP
                - MKNOD
                - AUDIT_WRITE
                - CHOWN
                - NET_RAW
                - DAC_OVERRIDE
                - FOWNER
                - FSETID
                - KILL
                - SETGID
                - SETUID
                - NET_BIND_SERVICE
                - SYS_CHROOT
                - SETFCAP
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30```

[jmtsi]

Thank you for this! I was afraid I'd have to implement this myself 😄 I will test this with IRSA and report back. At least the credential chain seems to prioritise env-credentials, so it should work

[jmtsi]

These changes are needed for IRSA (IAM roles for service accounts) to work

Suggested change

	Credentials: credentials.NewChainCredentials([]credentials.Provider{&credentials.EnvProvider{}, &credentials.SharedCredentialsProvider{}}),
	Credentials: credentials.NewChainCredentials([]credentials.Provider{
	&credentials.EnvProvider{},
	&credentials.SharedCredentialsProvider{},
	stscreds.NewWebIdentityRoleProvider(
	sts.New(sess),
	os.Getenv("AWS_ROLE_ARN"),
	"",
	os.Getenv("AWS_WEB_IDENTITY_TOKEN_FILE"),
	),
	&ec2rolecreds.EC2RoleProvider{
	Client: ec2metadata.New(sess),
	},
	}),

[jmtsi]

@JackFazackerley could you accept these changes so we can at least get this PR in a working state? Thanks!

[AlexMabry]

I could use these changes. Is there anything I can do to help move this along?

[sysadmind]

This PR will need to be rebased. Go module support has been merged already.

[jmtsi]

@sysadmind I rebased and cleaned this up in a separate PR: #443