Connections to S3 timeout too quickly after upgrading to latest HTTP.jl

[ianfiske]

I'm not sure whether to file this issue with HTTP.jl or AWSS3.jl. When using HTTP.jl 0.6.8, everything works fine and I can upload files to S3 (even from docker/AWS Batch). However, upgrading HTTP.jl to 0.6.9 causes this error. It seems that it's related to my application having authenticated earlier (to retrieve data from S3) and then that connection times out later before it has a chance to upload results back to S3.

I am using latest AWSS3 (0.3.7) and AWSCore (0.3.8) on Julia 0.6.2.

ERROR: LoadError: RequestTimeTooSkewed -- The difference between the request time and the current time is too large.
HTTP.ExceptionRequest.StatusError(403, HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden
x-amz-request-id: DA48EXXXXXXX
x-amz-id-2: eLxPBqGgXXXXXXXX
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Mon, 07 May 2018 14:30:32 GMT
Connection: close
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><RequestTime>20180507T141448Z</RequestTime><ServerTime>2018-05-07T14:30:34Z</ServerTime><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>DA48EXXXXXXX</RequestId><HostId>eLxPBqGgXXXXXXXXXXX=</HostId></Error>""")
Stacktrace:
[1] #request#1(::Array{Any,1}, ::Function, ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N) at /root/.julia/v0.6/HTTP/src/ExceptionRequest.jl:22
[2] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::String) at ./<missing>:0
[3] #request#1(::VersionNumber, ::String, ::Void, ::Void, ::Array{Any,1}, ::Function, ::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::String) at /root/.julia/v0.6/HTTP/src/MessageRequest.jl:44
[4] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::String) at ./<missing>:0
[5] macro expansion at /root/.julia/v0.6/AWSCore/src/http.jl:31 [inlined]
[6] macro expansion at /root/.julia/v0.6/Retry/src/repeat_try.jl:206 [inlined]
[7] http_request(::Dict{Symbol,Any}) at /root/.julia/v0.6/AWSCore/src/http.jl:22
[8] macro expansion at /root/.julia/v0.6/AWSCore/src/AWSCore.jl:421 [inlined]
[9] macro expansion at /root/.julia/v0.6/Retry/src/repeat_try.jl:206 [inlined]
[10] do_request(::Dict{Symbol,Any}) at /root/.julia/v0.6/AWSCore/src/AWSCore.jl:399
[11] macro expansion at /root/.julia/v0.6/AWSS3/src/AWSS3.jl:107 [inlined]
[12] macro expansion at /root/.julia/v0.6/Retry/src/repeat_try.jl:206 [inlined]
[13] #s3#1(::Dict{String,String}, ::String, ::Dict{String,String}, ::String, ::String, ::Bool, ::Bool, ::Function, ::Dict{Symbol,Any}, ::String, ::String) at /root/.julia/v0.6/AWSS3/src/AWSS3.jl:81
[14] (::AWSS3.#kw##s3)(::Array{Any,1}, ::AWSS3.#s3, ::Dict{Symbol,Any}, ::String, ::String) at ./<missing>:0
…

[samoconnor]

It is possible that this is related JuliaLang/MbedTLS.jl#145, does the problem go away if you use latest master of MbedTLS.jl?

If not, please call HTTP.ConnectionPool.showpool(STDOUT) right before the upload and also post the output with AWSCore.set_debug_level(3) and DEBUG_LEVEL = 2 (in HTTP.jl).

[ianfiske]

Thank you for the suggestion @samoconnor --- looks like running with the latest master of MBedTLS.jl (1c8c0a1b8d0e4baa1774a44794447659fd91124b) lets everything work with latest HTTP.jl and AWSS3.jl.