AWS S3 Input custom endpoint handling broken in 8.15 #40792
Labels
bug
Team:Cloud-Monitoring
Label for the Cloud Monitoring team
Team:Elastic-Agent-Data-Plane
Label for the Agent Data Plane team
8.15 no longer has the fix for AWS S3 endpoint handling that was present in 8.14
8.15:
beats/x-pack/filebeat/input/awss3/input.go
Lines 51 to 60 in 2f0dda8
8.14:
beats/x-pack/filebeat/input/awss3/input.go
Lines 83 to 104 in 7b6cfad
In AWS, the
endpoint
field is supposed to act kind of like a "base url" where service URLs are built using the value in the endpoint field. So when the SQS client makes a request, an endpoint field ofs3.us-east1.amazonaws.com
is transformed intosqs.us-east1.amazonaws.com
, etc.The 8.15 code forces all endpoints to use the value in the
endpoint
field instead of relying on the resolver to use the endpoint to "build" each service's endpoint (s3, sqs, etc). In the example above, this would cause the SQS client to directly querys3.us-east1.amazonaws.com
Even the 8.14 code has an issue that crops up with some customers. We should likely switch to only using a custom endpoint resolver when a user explicitly tells us to, for example by introducing a new setting called "static endpoint" or something similar, that when set to true, sets the endpoint resolver as it is set currently. This would be a breaking change.
An alternative would be introducing a setting called, "dynamic_endpoint" or something similar which, when set, sets the
endpoint
field without using a resolver.The text was updated successfully, but these errors were encountered: