lots of 403 Forbidden errors

[slmoloch]

For about 20% of my users I see lots of 403 FORBIDDEN POST https://mobileanalytics.us-east-1.amazonaws.com/2014-06-05/events error. Reproducible in all browsers.

Usually it is preceded with something like this:

Ajax POST
Url: https://cognito-identity.us-east-1.amazonaws.com/
Response: 200 1083 milliseconds elapsed

Note the time of the response.

[cheruvian]

Are you able to enable the logger and provide the log messages?

Additionally, when the SDK sees a 403 it should keep the events in the local storage and attempt to resubmit them again later. Do you know if those event batches are retried and if they are successful on retry? If not, can you provide a sample batch that failed when receiving a 403?

(Please be sure to redact application_key, cognito_identity_pool, cognito_id and any other potentially sensitive fields)

[slmoloch]

I will enable the logger on the site. Unfortunately I can not provide a batch because the error is not reproducible on our local environments. We see it appear sporadically at our end user machines in our js error log monitors.

[slmoloch]

here is what I got:

https://mobileanalytics.us-east-1.amazonaws.com/2014-06-05/events
Response: Pending

InvalidSignatureException: Signature expired: 20151003T005631Z is now earlier than 20151003T014131Z (20151003T015631Z - 15 min.),

[Function:(AMA.Client).clearBatchById]
batchId:914c5853-c45e-0df7-c124-f1a09f32d5cc

[Function:(AWS.MobileAnalyticsClient.Storage).saveToLocalStorage]

LocalStorage Cache: {"AWSMobileAnalyticsGlobalAttributes":{},"AWSMobileAnalyticsGlobalMetrics":{},
"AWSMobileAnalyticsClientId":"fbb85dab-5061-cace-f73f-11aa7bc1c4fd",
"AWSMobileAnalyticsEventStorage":[],"AWSMobileAnalyticsBatchStorage":{},"AWSMobileAnalyticsBatchIndexStorage":[],
"MobileAnalyticsSessionIda322a3d3-da60-f5ec-6cdf-1265d241327a":"a322a3d3-da60-f5ec-6cdf-1265d241327a",
"MobileAnalyticsSessionExpirationa322a3d3-da60-f5ec-6cdf-1265d241327a":1443834378782,
"MobileAnalyticsSessionStartTimeStampa322a3d3-da60-f5ec-6cdf-1265d241327a":"2015-10-03T00:56:18.782Z"}

403 Forbidden: POST https://mobileanalytics.us-east-1.amazonaws.com/2014-06-05/events

[Function:(AMA.Client).submitEvents]
options:{}

[Function:(AMA.Client).submitAllBatches]
options:{}

[slmoloch]

Looks very similar to aws/aws-sdk-js#527

[cheruvian]

Interesting, I went ahead and was able to reproduce the issue by setting my clock to the past (OSX + chrome). I then tried hardcoding the correctClockSkew to true but I still get the InvalidSignatureException.

The line to change can be found here.

[slmoloch]

I did the same and can confirm that setting correctClockSkew wouldn't fix the problem. Any other suggestions?

[cheruvian]

Hmmm still investigating. Is there a way to verify that the clients showing the error do have a clock skewed?

[cheruvian]

I've identified it as a Server side issue, I will post here once that fix has been deployed.

[slmoloch]

Thanks! looking forward to it

[cheruvian]

Can you confirm if you are seeing any improvement with your forbidden rate?

[slmoloch]

Hi, I confirm - the problem with 403 response dissapeared

[cheruvian]

In v0.9.1 autoCorrectClockSkew is now enabled by default. To disable it or provide other low level client configurations, you can use the new clientOptions configuration when constructing the AMA client(s).

new AMA.Manager({ appId: APPLICATION_ID, clientOptions: { region: 'us-east-1', } })

[abidulrmdn]

restarting docker daemon fixed it

sudo systemctl start docker
or
sudo service docker start