Permissions on ca-central-1 lambda bucket

[lucasteligioridis]

:(

aws_lambda_function.lambda: Error modifying Lambda Function Code: AccessDeniedException: Your access has been denied by S3, please make sure your request credentials have permission to GetObject for buildkite-lambdas-ca-central-1/buildkite-agent-scaler/v0.4.0/handler.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied

[lox]

It looks like this might be related to the user you are using to create the CloudFormation stack. AWS provided this insight:

1. While accessing a public S3 object present in a bucket within the same account, we don't need to specify any explicit permission (GetObject) in the CloudFormation service role / the role which is trying to create/update the stack.
2. However, even if an S3 object is publicly accessible, but if it's present in a different account (Say a third party account), we still need to give permission (GetObject) for that particular S3 object in the CloudFormation service role's policy / the role's policy which is trying to create/update the stack.

[lucasteligioridis]

Hmmm, I'm using a different set of users in my stack, but its the same IAM permissions as the other regions and they worked all fine.

I thought those buckets/paths were fully publicly accessible?

[lox]

The key element here is the user that is creating the stack. Yes those bucket/paths are fully accessible, but check out point 2 in the above from AWS.

[lucasteligioridis]

I'm misunderstanding, I don't see what the Cloudformation stack or role is relevant with the bucket policy? 2 examples below:

For example, here is a random account I'm authenticating against in ap-southeast-2:

$ export AWS_DEFAULT_REGION=ap-southeast-2
$ aws s3 cp s3://buildkite-lambdas-ap-southeast-2/buildkite-agent-scaler/v0.4.0/handler.zip .
download: s3://buildkite-lambdas-ap-southeast-2/buildkite-agent-scaler/v0.4.0/handler.zip to ./handler.zip

Now here is another random account with ca-central-1:

$ export AWS_DEFAULT_REGION=ca-central-1
$ aws s3 cp s3://buildkite-lambdas-ca-central-1/buildkite-agent-scaler/v0.4.0/handler.zip .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

[lox]

So many face palms. You are right, we are getting this same error for some that do exist, but in this case you are correct and we haven't published the scaler there! Sorry! Seeing zebras when I should be seeing horses 🦓

[lucasteligioridis]

Haha, great. I thought I was going crazy for a second. IAM will do that to you.
Glad we got it though :)

I just put a workaround in place by moving the lambda in my stack code and push it up as a file rather than fetching from S3.

[lox]

I'll get buildkite/buildkite-agent-scaler#11 into 4.3.2 on Monday