Support for IAM Roles (Instance Profile/IRSA) for Authentication #96

davivcgarcia · 2021-11-26T14:58:07Z

Is your feature request related to a problem? Please describe.
The current best practices for security in AWS recommends using short-lived credentials through IAM Role (STS) instead of static IAM Credentials (Access Key/Secret). That capacity is critically important when using Amazon EKS, since we can leverage IAM Roles for Service Accounts (IRSA) and improve security posture with Logstash on Kubernetes.

Describe the solution you'd like
Logstash should be able to consume an IAM Role, from an Instance Profile (EC2) or from IRSA (EKS), and use it to authenticate against the OpenSearch environment.

Describe alternatives you've considered
Use the legacy output plugin from awslabs/logstash-output-amazon_es.

Additional context
The current aws-sdk gem used in this plugin (>= 2.11.632) already supports IRSA capability.

The text was updated successfully, but these errors were encountered:

jigar-bhalodia1 · 2022-01-20T13:49:22Z

The legacy plugin doesn't support this either. awslabs/logstash-output-amazon_es#133

asyschikov · 2022-01-20T15:52:20Z

Looks like according to docs the instance profiles are supported: https://opensearch.org/docs/latest/clients/logstash/ship-to-opensearch/#opensearch-output-plugin
Look for optional parameters section:

Instance profile credentials delivered through the Amazon EC2 metadata service

vpulagarwal · 2022-02-06T15:54:02Z

Apparently, the latest plugin does not have a feature to read the Web Identity Token file (used for IRSA) mounted on the pod.

If we pass the token stored in /var/run/secrets/eks.amazonaws.com/serviceaccount/token file on the pod to variable session_token defined here, then it does not authorize the request but basic auth to the domain works just fine.

Logstassh output plugin config for IAM:

output {
  opensearch {
     hosts => "${LOGSTASH_OS_HOST}:${LOGSTASH_OS_PORT}"
     index => "${LOGSTASH_OS_INDEX}"
     auth_type => {    
          type => 'aws_iam'     
          session_token => "${LOGSTASH_OS_TOKEN}"     
          region => 'us-east-1'         
      }
      ssl => true
     document_id => "%{id}"
  }
}

VijayanB · 2022-02-07T02:45:44Z

@vpulagarwal Thanks for reporting. Will check why session token is not accepted by config

oridool · 2022-03-08T19:12:35Z

Please check
awsdocs/amazon-eks-user-guide#281
It means the aws sdk first needs to upgraded to 3.58+

vpulagarwal · 2022-03-09T14:01:51Z

@VijayanB Any plans to upgrade this plugin on Ruby 3.58+ and start supporting IRSA?

VijayanB · 2022-03-09T16:02:55Z

@vpulagarwal is it trivial? If so will you be able to contribute?

amine250 · 2022-03-25T10:25:03Z

I'm sorry I don't understand why upgrading the aws-sdk to 3.58.0 would take this much time.
Maybe some non-regression testing is all what's needed to be done.
Can we at least get an ETA ?

Thank you.

Fix opensearch-project#96 by upgrading aws sdk

Fix opensearch-project#96 by upgrading aws sdk Signed-off-by: Gong Yi <topikachu@163.com>

vpulagarwal · 2022-04-13T15:10:02Z

I am no ruby expert, but I feel to support IRSA, there should be an option to provide the path of the service account token file located at /var/run/secrets/eks.amazonaws.com/serviceaccount/token in this function

dblock · 2022-05-24T23:12:17Z

I see @topikachu closed #138, it wasn't merged.

Looking for someone to contribute here. Happy to help review and merge PRs. I would want/start with integration test CI, #76 and then it will make it easier to add features.

topikachu · 2022-05-29T13:44:23Z

I see @topikachu closed #138, it wasn't merged.

Looking for someone to contribute here. Happy to help review and merge PRs. I would want/start with integration test CI, #76 and then it will make it easier to add featurebreak othe aws plugins.

I see @topikachu closed #138, it wasn't merged.

Looking for someone to contribute here. Happy to help review and merge PRs. I would want/start with integration test CI, #76 and then it will make it easier to add features.

I realize it breaks all other aws plugins because they are all using the old aws sdk. :(

trunet · 2022-05-31T13:29:19Z

I can see on my opensearch audit logs the EKS worker role being used to try to authenticate leading to a 403.

Definitely this plugin is using the instance metadata to get the instance profile iam role instead of AWS_WEB_IDENTITY_TOKEN_FILE.

dblock · 2022-06-20T13:19:18Z

@topikachu What should we do about this? What else needs to be upgraded at the same time? Care to help?

trunet · 2022-06-20T13:57:36Z

@dblock My guess is that aws-sdk ruby gem needs to be update to v3. There's an upgrade guide for this, we need to be sure that all the requires are updated across this plugin.

mikef-nl · 2022-09-08T07:25:19Z

Is there any further progress on this? I currently have it setup using an IAM user but would prefer to use a role if possible. Has anyone tested this using a container execution role in AWS Fargate?

keymon · 2022-10-12T13:01:14Z

Any update on logstash-output-opensearch supporting the Web Identity Token file (used for IRSA), ie by reading the file in AWS_WEB_IDENTITY_TOKEN_FILE?

the PR #171 has been merged and the aws-sdk v3 is meant to support this. I think it was added here: aws/aws-sdk-ruby#2075

From what I read, it seems that the library might just work and pull the credentials out of the box. It is one of the default authentication providers but we need to test it.

andrejvanderzee · 2023-03-14T21:55:26Z

We require this too.

Thamizhvanan-R · 2023-04-17T10:22:57Z

Any update on logstash-output-opensearch supporting the Web Identity Token file (used for IRSA), ie by reading the file in AWS_WEB_IDENTITY_TOKEN_FILE?

the PR #171 has been merged and the aws-sdk v3 is meant to support this. I think it was added here: aws/aws-sdk-ruby#2075

From what I read, it seems that the library might just work and pull the credentials out of the box. It is one of the default authentication providers but we need to test it.

I am able to get it working by below configuration, the logstash has a default value for the region input, if you pass the correct region then it works

 auth_type => {
      type => 'aws_iam'
      region => <region>
    }

zentavr · 2023-10-11T20:29:28Z

should work in EKS just with:

output {
		    opensearch {
		        hosts => [ "https://search-kibana-stage-xxx.us-east-1.es.amazonaws.com:443" ]
		        http_compression => true
		        index => "clb-logs-test-%{+YYYY.MM.dd}"
		        ecs_compatibility => disabled
		        auth_type => {
		            type => "aws_iam"
		            region => "us-east-1"
		            service_name => "es"
		        }
		        legacy_template => true
		        template => "/opt/bitnami/logstash/templates/alb-access-logs.json"
		        template_name => "alb-access-logs"
		    }
		    stdout { codec => rubydebug }
		}

joewragg · 2024-04-04T15:24:06Z

+1 on this, tried a lot of the "workarounds" on both these issues and none work. I will have to revert to basic auth for now unfortunately

dblock · 2024-04-06T15:15:22Z

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work?
@joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

Thamizhvanan-R · 2024-04-06T15:57:41Z

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock

The Opensearch project uses aws-sdk-ruby for aws authentication.

logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

Line 85 in d60a58f

@credentials = Aws::CredentialProviderChain.new(credential_config).resolve

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain.

https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143
Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.

I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.

@joewragg I would suggest you check if these env variables are set in your case.

dblock · 2024-04-08T07:38:35Z

the proper fix is to read the AWS_REGION from env and fall back to default if no such environment variable is set.

In the code you pointed out does setting the region to nil work? If so then the fact that we pass through the us-east-1 default is incorrect, the default should be nil everywhere.

Whichever the fix is here, do you think you could write a test for it that shows that the regions configured by the AWS_REGION environment variable produce a different value here? This makes an easy fix.

joewragg · 2024-04-08T07:41:54Z

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock

The Opensearch project uses aws-sdk-ruby for aws authentication.

logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

Line 85 in d60a58f

@credentials = Aws::CredentialProviderChain.new(credential_config).resolve

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain.
https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.

I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.

@joewragg I would suggest you check if these env variables are set in your case.

I have set the correct ENV vars as mentioned (redacted):

AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::ACCOUNTID:role/MYROLE
AWS_ACCOUNT_ID=ACCOUNTID
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional

My config (redacted):

output {
  if [type] == "cloudfront" {
    opensearch {
      hosts => ["https://MYURL:443"]
      index => "cloudfront-logs-${ENV}-%{+YYYY-MM-dd}"
      auth_type => {
        type => 'aws_iam'     
        region => "${AWS_REGION}"
      }         
      ecs_compatibility => disabled
      }
  }
}

My IAM role is mapped to all_access in opensearch for testing

I am seeing 403s coming from opensearch in the logs

Thamizhvanan-R · 2024-04-08T07:54:07Z

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock

The Opensearch project uses aws-sdk-ruby for aws authentication.

logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

Line 85 in d60a58f

@credentials = Aws::CredentialProviderChain.new(credential_config).resolve

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain.
https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.

I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.

@joewragg I would suggest you check if these env variables are set in your case.

I have set the correct ENV vars as mentioned (redacted):
AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::ACCOUNTID:role/MYROLE
AWS_ACCOUNT_ID=ACCOUNTID
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional
My config (redacted):
output {
  if [type] == "cloudfront" {
    opensearch {
      hosts => ["https://MYURL:443"]
      index => "cloudfront-logs-${ENV}-%{+YYYY-MM-dd}"
      auth_type => {
        type => 'aws_iam'     
        region => "${AWS_REGION}"
      }         
      ecs_compatibility => disabled
      }
  }
}
My IAM role is mapped to all_access in opensearch for testing

I am seeing 403s coming from opensearch in the logs

One more thing we can check is if your role has assumerolepolicy and it allows your eks service account

joewragg · 2024-04-08T08:05:54Z

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock
The Opensearch project uses aws-sdk-ruby for aws authentication.

logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

Line 85 in d60a58f

@credentials = Aws::CredentialProviderChain.new(credential_config).resolve

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain.
https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.
I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.
@joewragg I would suggest you check if these env variables are set in your case.

I have set the correct ENV vars as mentioned (redacted):
AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::ACCOUNTID:role/MYROLE
AWS_ACCOUNT_ID=ACCOUNTID
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional
My config (redacted):
output {
  if [type] == "cloudfront" {
    opensearch {
      hosts => ["https://MYURL:443"]
      index => "cloudfront-logs-${ENV}-%{+YYYY-MM-dd}"
      auth_type => {
        type => 'aws_iam'     
        region => "${AWS_REGION}"
      }         
      ecs_compatibility => disabled
      }
  }
}
My IAM role is mapped to all_access in opensearch for testing
I am seeing 403s coming from opensearch in the logs
One more thing we can check is if your role has assumerolepolicy and it allows your eks service account

Yeah I definitely have the assumerolepolicy working as it's already using this role to S3 GET in the pod

dblock · 2024-04-08T08:57:46Z

@joewragg Is it 100% of requests? Can you paste a (redacted) log line here?

joewragg · 2024-04-09T07:07:52Z

@joewragg Is it 100% of requests? Can you paste a (redacted) log line here?

Pretty sure it's 100% of requests.

If I use username and password it works.

I see this log line over and over that's it

[2024-04-09T07:06:02,353][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://MYURL:443/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '403' contacting OpenSearch at URL 'https://MYURL:443/'"}

dblock · 2024-04-10T07:09:01Z

@joewragg Turn on debug level logging, let's see the complete error?

I think https://forum.opensearch.org/t/enable-debug-logging-on-logstash/12194 or passing --debug to logstash will give you that.

joewragg · 2024-04-10T08:10:30Z

@joewragg Turn on debug level logging, let's see the complete error?

I think https://forum.opensearch.org/t/enable-debug-logging-on-logstash/12194 or passing --debug to logstash will give you that.

Does this have something to do with these settings in AWS:

Relevant logs I see repeating:

[2024-04-10T08:04:53,137][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,138][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Young Generation"}
[2024-04-10T08:04:53,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,138][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2024-04-10T08:04:53,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,143][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:54,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,145][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:55,137][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,143][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:56,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,145][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:56,923][DEBUG][logstash.outputs.opensearch][main] Waiting for connectivity to OpenSearch cluster, retrying in 32s
[2024-04-10T08:04:57,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,145][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:57,187][DEBUG][logstash.outputs.opensearch][main] Running health check to see if an OpenSearch connection is working {:url=>"https://MYURL:443/", :path=>"/"}
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.client.protocol.RequestAuthCache][main] Auth cache not set in the context
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection request: [route: {s}->https://MYURL:443][total available: 1; route allocated: 1 of 100; total allocated: 1 of 1000]
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection leased: [id: 0][route: {s}->https://MYURL:443][total available: 0; route allocated: 1 of 100; total allocated: 1 of 1000]
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: set socket timeout to 60000
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: set socket timeout to 60000
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.impl.execchain.MainClientExec][main] Executing request HEAD / HTTP/1.1
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.impl.execchain.MainClientExec][main] Proxy auth state: UNCHALLENGED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> HEAD / HTTP/1.1
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> Connection: Keep-Alive
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> content-type: application/json
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> host: MYURL
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> x-amz-date: 20240410T080457Z
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> x-amz-security-token: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> x-amz-content-sha256: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20240410/eu-west-1/es/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> User-Agent: Manticore 0.9.1
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> Accept-Encoding: gzip,deflate
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "content-type: application/json[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "host: MYURL[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "x-amz-date: 20240410T080457Z[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "x-amz-security-token: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "x-amz-content-sha256: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20240410/eu-west-1/es/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "User-Agent: Manticore 0.9.1[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Date: Wed, 10 Apr 2024 08:04:57 GMT[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Content-Type: application/json[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Content-Length: 251[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Connection: keep-alive[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "x-amzn-requestid: REDACTED
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "access-control-allow-origin: *[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << HTTP/1.1 403 Forbidden
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Date: Wed, 10 Apr 2024 08:04:57 GMT
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Content-Type: application/json
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Content-Length: 251
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Connection: keep-alive
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << x-amzn-requestid: REDACTED
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << access-control-allow-origin: *
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.execchain.MainClientExec][main] Connection can be kept alive indefinitely
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection [id: 0][route: {s}->https://MYURL:443] can be kept alive indefinitely
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: set socket timeout to 0
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection released: [id: 0][route: {s}->https://MYURL:443][total available: 1; route allocated: 1 of 100; total allocated: 1 of 1000]
[2024-04-10T08:04:57,212][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://MYURL:443/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '403' contacting OpenSearch at URL 'https://MYURL:443/'"}

dblock · 2024-04-10T13:38:49Z

@joewragg Can you curl or awscurl your server with these credentials (https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html) and if so with what exact parameters/settings? I'm sure you're reading https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html already, but it's worth another pass in case something is incorrect in your setup.

joewragg · 2024-04-16T10:02:51Z

@joewragg Can you curl or awscurl your server with these credentials (https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html) and if so with what exact parameters/settings? I'm sure you're reading https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html already, but it's worth another pass in case something is incorrect in your setup.

Yeah awscurl works

dblock · 2024-04-16T11:46:54Z

@joewragg Can you curl or awscurl your server with these credentials (https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html) and if so with what exact parameters/settings? I'm sure you're reading https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html already, but it's worth another pass in case something is incorrect in your setup.

Yeah awscurl works

Good. Knowing that the credentials are valid clears that. The next step is to debug the code in this library. It's doing its own signing here, I'd ensure that the parameters passed in match what's in the environment (maybe add some log lines?).

I'm happy to try to help if this can be reproduced easily.

zentavr · 2024-05-13T23:51:04Z

@dblock it's like it tries to use instance profile instead of STS...

dblock · 2024-05-14T11:27:54Z

I think it was mentioned above:

Definitely this plugin is using the instance metadata to get the instance profile iam role instead of AWS_WEB_IDENTITY_TOKEN_FILE.

I am hoping someone can implement this, I doubt I'll get to it any time soon.

vpulagarwal mentioned this issue Mar 8, 2022

auth_type 'aws_iam' with IAM Roles For Service Account (IRSA) #127

Open

topikachu pushed a commit to topikachu/logstash-output-opensearch that referenced this issue Apr 11, 2022

Upgrade aws sdk to latest v3

99de505

Fix opensearch-project#96 by upgrading aws sdk

topikachu pushed a commit to topikachu/logstash-output-opensearch that referenced this issue Apr 11, 2022

Upgrade aws sdk to latest v3

b107543

Fix opensearch-project#96 by upgrading aws sdk Signed-off-by: Gong Yi <topikachu@163.com>

topikachu added a commit to topikachu/logstash-output-opensearch that referenced this issue Apr 11, 2022

Upgrade aws sdk to latest v3

1193d20

Fix opensearch-project#96 by upgrading aws sdk Signed-off-by: Gong Yi <topikachu@163.com>

topikachu mentioned this issue Apr 11, 2022

Upgrade aws sdk to latest v3 #138

Closed

5 tasks

hardproblems mentioned this issue Aug 26, 2022

Upgrade to aws sdk v3 #171

Merged

5 tasks

dblock added the enhancement Increases software capabilities beyond original client specifications label Feb 13, 2024

Support for IAM Roles (Instance Profile/IRSA) for Authentication #96

Support for IAM Roles (Instance Profile/IRSA) for Authentication #96

Comments

davivcgarcia commented Nov 26, 2021 • edited Loading

jigar-bhalodia1 commented Jan 20, 2022

asyschikov commented Jan 20, 2022

vpulagarwal commented Feb 6, 2022

VijayanB commented Feb 7, 2022

oridool commented Mar 8, 2022

vpulagarwal commented Mar 9, 2022

VijayanB commented Mar 9, 2022

amine250 commented Mar 25, 2022

vpulagarwal commented Apr 13, 2022

dblock commented May 24, 2022

topikachu commented May 29, 2022

trunet commented May 31, 2022

dblock commented Jun 20, 2022

trunet commented Jun 20, 2022

mikef-nl commented Sep 8, 2022

keymon commented Oct 12, 2022 • edited Loading

andrejvanderzee commented Mar 14, 2023

Thamizhvanan-R commented Apr 17, 2023

zentavr commented Oct 11, 2023

joewragg commented Apr 4, 2024 • edited Loading

dblock commented Apr 6, 2024

Thamizhvanan-R commented Apr 6, 2024

dblock commented Apr 8, 2024

joewragg commented Apr 8, 2024 • edited Loading

Thamizhvanan-R commented Apr 8, 2024

joewragg commented Apr 8, 2024

dblock commented Apr 8, 2024

joewragg commented Apr 9, 2024 • edited Loading

dblock commented Apr 10, 2024

joewragg commented Apr 10, 2024 • edited Loading

dblock commented Apr 10, 2024

joewragg commented Apr 16, 2024

dblock commented Apr 16, 2024

zentavr commented May 13, 2024

dblock commented May 14, 2024

davivcgarcia commented Nov 26, 2021 •

edited

Loading

keymon commented Oct 12, 2022 •

edited

Loading

joewragg commented Apr 4, 2024 •

edited

Loading

joewragg commented Apr 8, 2024 •

edited

Loading

joewragg commented Apr 9, 2024 •

edited

Loading

joewragg commented Apr 10, 2024 •

edited

Loading