InvalidAccessKeyId when using instance profile for S3 upload

[benbc]

I am seeing this error when uploading a large number of files to S3 from an EC2 instance, using an instance role for authentication.

upload failed: ./image.part.130 to s3:///agent-ami/image.part.130
A client error (InvalidAccessKeyId) occurred when calling the CompleteMultipartUpload operation: The AWS Access Key Id you provided does not exist in our records.

I do this operation quite a lot and don't normally see this problem. I have seen it twice today.

I wonder if there is a race condition when the role's credentials are rotated.

I am using the latest version of awscli from PyPI.

[akaczorek]

Today, I experienced this exact behavior during repeated sequential S3 downloads using instance roles. I'm wondering if this behavior is related to the IAM instance role functionality of autorotating the keys. SDK and tool implementations have to be able to handle the transient authentication failure and retry with the new credentials pulled down from instance metadata when the rotation occurs. It is possible that our failures were a result of unlucky timing related to the automatic key rotation.

[toastdriven]

Working on this as part of boto/botocore#251...

[toastdriven]

This should be fixed in the next release of the CLI (& botocore).

[felipe-vvoosh]

I'm having the exact same problem... But this was fixed right? I have an instance with the newest AMI possible and fully updated. The aws s3 sync starts fine and after some time I get this message "The AWS Access Key Id you provided does not exist in our records."

[felipe-vvoosh]

Can you guys reopen this one?

[k-f-]

I was having this exact issue earlier today. The issue for me was that the credentials I was using were for my AWS account, not my IAM user.

[SooLee]

I've been using IAM role with no problem so far and today I'm getting this error when uploading to S3 from lambda: An error occurred (InvalidAccessKeyId) when calling the PutObject operation

[justnance]

@SooLee - Thank you for reporting this error. It would help if you could post the exact CLI command here so I can investigate. It would also help if you could post the sanitized output after using the --debug flag. It appears to be an issue with the credentials in use as the CLI is echoing an error returning from the service. Can you recheck your Access Key and Secret Key configuration? As mentioned in #1946, "quick way to figure out what credentials are being used is by running: aws configure list. That will show where the credentials are being pulled from in the CLI."