InvalidSignatureException: Signature expired #527

evansolomon · 2015-03-09T20:50:21Z

First, I know there have been several similar issues in the past. I've tried to read through them all but they all seem to be either for issues that are closed or for different problems.

I have a bunch of apps that run on AWS + Docker and at some point will randomly start throwing these errors. I think the error usually comes up when I haven't worked on the app for a couple days, so the "expiration" is pretty extreme. For example, the one I just got is "expired" by 2 days.

Signature expired: 20150307T194740Z is now earlier than 20150309T203545Z

Some details that are probably relevant:

These apps are using static API keys in ~/.aws/
The docker container links the .aws directory along the lines of docker run --volume "${HOME}/.aws":/home/app/.aws whatever-image-name
Eventually through some random combination of re-installing dependencies, running the app outside of docker, and reading chicken entrails, things do start working again — unfortunately, I can never figure out exactly why

I've dug through the internals of the SDK a bit, but I don't quite know what I'm looking for and hoping you can point me in a more productive direction.

Currently I'm using v2.1.14 (I know there are a couple newer patch releases in the last couple days). Here is a recent stack track.

InvalidSignatureException: Signature expired: 20150307T194740Z is now earlier than 20150309T203545Z (20150309T204045Z - 5 min.)
  at Request.extractError (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/protocol/json.js:43:27)
  at Request.callListeners (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:100:18)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:604:14)
  at Request.transition (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:21:12)
  at AcceptorStateMachine.runTo (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:14:12)
  at /home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:26:10
  at Request.&lt;anonymous&gt; (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:22:9)
  at Request.&lt;anonymous&gt; (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:606:12)
  at Request.callListeners (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:104:18)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
  at Request.emit (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:604:14)
  at Request.transition (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:21:12)
  at AcceptorStateMachine.runTo (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:14:12)
  at /home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/state_machine.js:26:10
  at Request.&lt;anonymous&gt; (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:22:9)
  at Request.&lt;anonymous&gt; (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/request.js:606:12)
  at Request.callListeners (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:104:18)
  at callNextListener (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/sequential_executor.js:90:14)
  at IncomingMessage.onEnd (/home/app/contents/node_modules/galileofive-common/node_modules/aws-sdk/lib/event_listeners.js:183:11)
  at IncomingMessage.emit (events.js:117:20)
  at _stream_readable.js:943:16
  at process._tickDomainCallback (node.js:463:13)

The text was updated successfully, but these errors were encountered:

odeke-em · 2015-03-10T06:20:20Z

Just curious are you able to synchronize your NTP server/clock with AWS' ? Several months ago, I encountered this error a couple of times, when creating signatures on my local machine. Hopefully this doc will be relevant: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

evansolomon · 2015-03-10T06:47:44Z

I've looked into the clock thing before but don't think that's it. In the stack trace above, the date is off by like 48 hours, so that would be a lot of clock skew.

lsegal · 2015-03-10T07:08:44Z

@evansolomon I'm not sure why clock skew would be ruled out, it seems to coincide precisely with your usage pattern described above:

I think the error usually comes up when I haven't worked on the app for a couple days,

If Docker images only resync NTP when they are re-activated through commands and they have not been activated in multiple days, that would explain the skew pretty accurately. The SDK definitely relies on "current system time" when making requests, so it needs to be synced prior to sending requests. You might see the exact same behavior on a development machine right after waking the machine up from sleep (prior to clock sync), for example.

I would recommend taking stock of the date command in your Docker image next time you run into this issue. Knowing the current system time would be very helpful to diagnosing the issue.

evansolomon · 2015-03-10T07:15:07Z

I have tried running date in the past when it's happened and always had correct results. I've also rebuilt the docker image without cache and had the same problem.

I don't know how the internals of docker image clocks work, but I just took a couple of old (and not recently used) docker images on my machine and ran docker run old-image-name date for each one. They all came out with correct dates.

I agree it does seem like a good theory, but I haven't been able to find any evidence of it.

AdityaManohar · 2015-03-10T18:15:24Z

@evansolomon
Circling back to the error stack, it looks like the skew is actually 5 min and not 48 hours. Would you be able to re-run the date command on your Docker image and let me know what you find?

Do you also encounter the error when you run the same code outside your Docker image?

Alternatively, would you be able to share your Docker image? This would help debug the issue.

evansolomon · 2015-03-10T18:25:18Z

it looks like the skew is actually 5 min and not 48 hours

I assume you mean this line

20150307T194740Z is now earlier than 20150309T203545Z (20150309T204045Z - 5 min.)

I parsed that as March 7, 2015 19:47 is earlier than March 9, 2015 20:40. I don't really know what the 5 minute thing on the end means, I was thinking maybe that was the threshold.

Would you be able to re-run the date command on your Docker image and let me know what you find?

Just tried again on an image I haven't used since yesterday and got "Tue Mar 10 18:22:48 UTC 2015" (which is correct at the time I'm writing this).

Do you also encounter the error when you run the same code outside your Docker image?

I'm not sure. Our workflow is pretty heavily Docker-centric, so this isn't much of a use case for me.

would you be able to share your Docker image?

It is based on https://github.com/phusion/passenger-docker. The only differences are some nginx configs, the shared ~/.aws directory, and my apps.

kyriesent · 2015-06-11T20:38:23Z

@evansolomon did you ever find the cause/solution for this? I've got the same issue on a docker machine running on debian:jessie. Pretty much same exact stack trace as yours.

evansolomon · 2015-06-12T00:18:51Z

@kyriesent Sorry, never really figured it out

kyriesent · 2015-06-12T14:28:42Z

Turns out mine was a clock syncing issue on my VM. Running a sync between the VM and the host cleared it up.

AdityaManohar · 2015-09-15T20:32:48Z

@evansolomon I've just pushed a patch that offsets the SDK clock when a clock skew error is detected. Simply set the correctClockSkew option when constructing a service client.

Here's an example:

var dynamodb = new AWS.DynamoDB({correctClockSkew: true});

Let me know if this works for you. Thanks for your patience.

evansolomon · 2015-09-15T20:34:54Z

Thanks for the update. Is there a reason this shouldn't be on by default?

AdityaManohar · 2015-09-16T00:43:53Z

@evansolomon This is turned off by default because there may be other customers who are applying a clock skew correction using the AWS.config.systemClockOffset property.

evansolomon · 2015-09-16T01:38:06Z

Are there cases where systemClockOffset is set to 0 and the user would not want the skew correction to be applied? It seems like that would be a sensible default, but maybe I'm missing something.

lraulier · 2016-02-08T11:12:19Z

hi all,

same issue 'Signature expired' on aws command line
resolved by 'calling ntpd' (The Network Time Protocol (NTP) is used to synchronize the time of a computer).

hope it helps
regards

creeefs · 2016-08-09T18:00:58Z

I encountered this issue locally, and restarting my docker daemon fixed it.

tonyjiang · 2017-02-22T21:55:00Z

@Clee681 thank you for sharing your experience and solution! Restarting docker daemon fixed my problem as well - could take much longer to solve it without seeing your comment.

mankins · 2017-03-07T21:28:14Z

None of the above mentions workarounds worked for me, even though the environment was similar.

One difference may be that I was uploading via the --zip-file method of the aws-cli over a slow network. The upload was actually taking more than 5 minutes and consequently giving the SignatureDoesNotMatch error.

I was able to work around this slow network by uploading first to s3, and then instead of the zip-file command line option using the --s3-bucket and --s3-key options. Perhaps this is an unrelated issue but it took me long enough to solve that I thought I'd document it here in case.

afeld · 2017-03-27T16:53:31Z

More discussion: https://forums.docker.com/t/automated-build-on-docker-hub-is-there-a-way-to-specify-a-dockerfile/1078

sbeam · 2017-05-11T13:33:32Z

if you are running docker on OS X then this is probably the cause, a long-standing issue with docker not resyncing the clock after sleep. Fix coming in next release.

docker/for-mac#17

harish-swamy · 2017-05-17T06:33:07Z

go to vm settings -> date&time -> enable date and time with internetaccess ... this issue should be fix.

varenc · 2017-07-18T02:58:54Z

would love to see correctClockSkew default to True in the future since this fixes a problem that virtually everyone will eventually run in to. Having it default to false is like embedding a non-deterministic bug that causes problem for ~0.1% of users in everyone's AWS implementation. Any site with lots of diverse users will eventually have to deal with this likely after their implementation has gone into production. I've opened an issue regarding this for discussion: #1632

saikiran91 · 2018-02-17T09:07:07Z

I have an exactly same issue. I run Ubuntu terminal on Windows 10. Any solution for this?

Kovaloff · 2018-09-01T15:01:54Z

a restart of the VM helped me

zeeshanjamal16 · 2019-01-14T13:12:27Z

Run the command to sync clock
ntpdate pool.ntp.org

abidulrmdn · 2019-02-27T12:51:56Z

restarting docker daemon fixed it

sudo systemctl start docker
or
sudo service docker start

lock · 2019-09-28T03:55:22Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

lsegal added Question labels Mar 10, 2015

AdityaManohar mentioned this issue Sep 9, 2015

Add/clock skew retries #706

Merged

AdityaManohar closed this as completed in #706 Sep 15, 2015

slmoloch mentioned this issue Oct 3, 2015

lots of 403 Forbidden errors aws/aws-sdk-mobile-analytics-js#7

Closed

EthanCai mentioned this issue Mar 3, 2016

A hopefully useable status of dlog topicai/dlog#2

Merged

goliatone mentioned this issue May 31, 2016

We can now pass AWS.CloudWatchLogs options lazywithclass/winston-cloudwatch#39

Merged

LiuJoyceC mentioned this issue Oct 11, 2016

SignatureDoesNotMatch: Signature expired: 20161010T120646Z is now earlier than 20161010T151154Z (20161010T152654Z - 15 min.) #1174

Closed

varenc mentioned this issue Jul 18, 2017

correctClockSkew should default to True #1632

Closed

komuw mentioned this issue Aug 22, 2017

Option to upload zips to S3 to bypass size limit & multi-part uploads apex/up#248

Closed

evgenyfadeev mentioned this issue Nov 21, 2017

add ntpd to time sync the server. puckel/docker-airflow#130

Open

saikiran91 mentioned this issue Feb 17, 2018

I have exact same issue (InvalidSignatureException: Signature expired) #1938

Closed

pgavlin mentioned this issue Apr 28, 2018

Large function contexts can cause lambda creation timeouts pulumi/pulumi-cloud#468

Closed

almeidaah mentioned this issue Sep 20, 2018

InvalidSignatureException: Signature expired aws/aws-sdk-java#1764

Closed

srchase added guidance Question that needs advice or information. and removed Question labels Jan 4, 2019

davidsbailey mentioned this issue Jun 5, 2019

correct clock skew in firehose js client code-dot-org/code-dot-org#28959

Merged

lock bot locked as resolved and limited conversation to collaborators Sep 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

InvalidSignatureException: Signature expired #527

InvalidSignatureException: Signature expired #527

evansolomon commented Mar 9, 2015

odeke-em commented Mar 10, 2015

evansolomon commented Mar 10, 2015

lsegal commented Mar 10, 2015

evansolomon commented Mar 10, 2015

AdityaManohar commented Mar 10, 2015

evansolomon commented Mar 10, 2015

kyriesent commented Jun 11, 2015

evansolomon commented Jun 12, 2015

kyriesent commented Jun 12, 2015

AdityaManohar commented Sep 15, 2015

evansolomon commented Sep 15, 2015

AdityaManohar commented Sep 16, 2015

evansolomon commented Sep 16, 2015

lraulier commented Feb 8, 2016

creeefs commented Aug 9, 2016

tonyjiang commented Feb 22, 2017 •

edited

Loading

mankins commented Mar 7, 2017

afeld commented Mar 27, 2017

sbeam commented May 11, 2017

harish-swamy commented May 17, 2017

varenc commented Jul 18, 2017 •

edited

Loading

saikiran91 commented Feb 17, 2018

Kovaloff commented Sep 1, 2018

zeeshanjamal16 commented Jan 14, 2019

abidulrmdn commented Feb 27, 2019

lock bot commented Sep 28, 2019

InvalidSignatureException: Signature expired #527

InvalidSignatureException: Signature expired #527

Comments

evansolomon commented Mar 9, 2015

odeke-em commented Mar 10, 2015

evansolomon commented Mar 10, 2015

lsegal commented Mar 10, 2015

evansolomon commented Mar 10, 2015

AdityaManohar commented Mar 10, 2015

evansolomon commented Mar 10, 2015

kyriesent commented Jun 11, 2015

evansolomon commented Jun 12, 2015

kyriesent commented Jun 12, 2015

AdityaManohar commented Sep 15, 2015

evansolomon commented Sep 15, 2015

AdityaManohar commented Sep 16, 2015

evansolomon commented Sep 16, 2015

lraulier commented Feb 8, 2016

creeefs commented Aug 9, 2016

tonyjiang commented Feb 22, 2017 • edited Loading

mankins commented Mar 7, 2017

afeld commented Mar 27, 2017

sbeam commented May 11, 2017

harish-swamy commented May 17, 2017

varenc commented Jul 18, 2017 • edited Loading

saikiran91 commented Feb 17, 2018

Kovaloff commented Sep 1, 2018

zeeshanjamal16 commented Jan 14, 2019

abidulrmdn commented Feb 27, 2019

lock bot commented Sep 28, 2019

tonyjiang commented Feb 22, 2017 •

edited

Loading

varenc commented Jul 18, 2017 •

edited

Loading