ReadError: The server aborted pending request

[mifi]

Initial checklist

• I understand this is a bug report and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Link to runnable example

https://uppy.io/examples/dashboard/

Steps to reproduce

I'm able to reproduce the problem very consistently, it's very easy to reproduce:

• https://uppy.io/examples/dashboard/
• select dropbox
• select 4 files, 1mb each, resulting in 4 concurrent downloads/uploads.

If I run it locally and slow down/throttle the tcp responses it happens even with just a single file. The timeouts happen after about 30 sec or so (though not consistent timing).

The code I use to throttle locally:
Replace this line

uppy/packages/@uppy/companion/src/server/Uploader.js

Line 300 in 233435f

const ret = await this.uploadStream(stream)

...with this:

async function* transform () {
  for await (const chunk of stream) {
    console.log('chunk',chunk.length)
    await new Promise((resolve) => setTimeout(resolve, 500))
    yield chunk
  }
}

const transformedStream = require('stream').Readable.from(transform())

const ret = await this.uploadStream(transformedStream)

// same also happens when outputting to file btw:
// const outstream = require('fs').createWriteStream(`/Users/mifi/Desktop/companion${Math.random()}`)
// await pipeline(transformedStream, outstream)

Expected behavior

Uploads should complete successfully.

Actual behavior

Upload of some or all of the files fails.

Downloads from these providers are consistently failing with ReadError: The server aborted pending request - meaning the server abruptly disconnects Companion's http request to download the file from the provider.

Here's the most interesting part:
If I vpn to the US (or Asia which I am in now) using NordVPN, the issue completely disappears. I'm not able to make a single upload fail, even with throttling enabled and multiple files!

Interestingly the problem also happens on our https://uppy.io/examples/dashboard/ - and this server runs on heroku servers in the US. At first I thought that dropbox, google etc have some DoS protection again requests coming from certain regions like asia, but that seems not to be the case.

What is my theory now is that these providers have DoS protection against certain IP address ranges, where they will be more inclined to cut off "long" running reqeusts coming from these ranges in order to protect themselves. Heroku servers and an internet cafe in Asia are both considered "unsafe" IP addresses, but nordVPN and other VPN providers' sole job is to provide customers with IP addresses that cloud providers cannot block (to watch netflix in any country for example, bypassing netflix's blocks), so using a VPN gives an IP address that providers like dropbox and google are also not able to flag and protect against.

Not sure how to solve this. I think retry is not going to help, because the problem seems very consistent, and retrying will probably just end up getting companion's IP blocked.

[mifi]

I have tried to replace all this code

uppy/packages/@uppy/companion/src/server/provider/dropbox/index.js

Line 73 in d19e6d8

const stream = (await getClient({ token })).stream.post('files/download', {

with this, to see if the problem is related to got:

const http = require('node:https')
const res = await new Promise((resolve, reject) => {
  const req = http.request({
    method: 'POST',
    host: 'content.dropboxapi.com',
    path: '/2/files/download',
    headers: {
      Authorization: `Bearer ${token}`,
      'Dropbox-API-Arg': httpHeaderSafeJson({ path: String(id) }),
    },
  }, (res) => {
    console.log(`STATUS: ${res.statusCode}`)
    console.log(`HEADERS: ${JSON.stringify(res.headers)}`)
    resolve(res)
  })
  req.on('error', (err) => {
    reject(err)
  })
  req.write(Buffer.alloc(0))
  req.end()
})

return { stream: res }

...same problem, but the error is more familiar:

companion2: 2023-03-17T07:27:26.328Z [error] 241cef21 uploader.error Error: aborted
    at connResetException (node:internal/errors:705:14)
    at TLSSocket.socketCloseListener (node:_http_client:454:19)
    at TLSSocket.emit (node:events:525:35)
    at TLSSocket.emit (node:domain:489:12)
    at node:net:301:12
    at TCP.done (node:_tls_wrap:588:7) {
  code: 'ECONNRESET'
}

...so the bug is not in got.

I'm not able to reproduce the problem with curl on the command line, so for fun I tried to replace the code with:

const curl = `curl -X POST https://content.dropboxapi.com/2/files/download --header "Authorization: Bearer ${token}" --header 'Dropbox-API-Arg: ${httpHeaderSafeJson({ path: String(id) })}' -o -`
const cmd = execa(curl, { shell: true, buffer: false, encoding: null });
return { stream: cmd.stdout }

and it works! good ol' curl!

need to investigate more why only curl works and not node.js

[mifi]

Finally I managed to identify and solve the problem!
After trying basically everything with the streams, no matter what I did, dropbox would terminate the HTTP request before the node stream was finished processing the data, and ECONNRESET got thrown from the HTTP response stream.

In the end I compared which HTTP headers curl sends vs headers sent by Node.js http.request(). I went through every single header and thru trial and error I found that the culprit is Connection header.

• curl doesn't send any Connection header, however in HTTP/1.1 Connection: keep-alive is the default
• Node.js by default sends Connection: close

So when I in got or http.request set the header to Connection: keep-alive, it works! no more connection resets.

This seems like it's not a problem with other providers like Google or Box (with the default Connection: close), so it leads me to believe that something is wrong with Dropbox's HTTP implementation.

I tried to change curl also to use Connection: close, and then it also showed the same problem, if I run two curl in parallel:

curl -H 'Connection: close' -vvv -X POST https://content.dropboxapi.com/2/files/download --header "Authorization: Bearer token" --header 'Dropbox-API-Arg: {"path":"/transloadit xls test/training000188.xls"}' -o out.xls &
curl -H 'Connection: close' -vvv -X POST https://content.dropboxapi.com/2/files/download --header "Authorization: Bearer token" --header 'Dropbox-API-Arg: {"path":"/transloadit xls test/training000188.xls"}' -o out.xls

< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: no-cache
< Dropbox-Api-Result: {"name": "TRAINING000188.xls", "path_lower": "/transloadit xls test/training000188.xls", "path_display": "/transloadit xls test/TRAINING000188.xls", "id": "id:NBIhAz1dRdQAAAAAAAAB6w", "client_modified": "2023-03-06T16:07:42Z", "server_modified": "2023-03-15T03:29:09Z", "rev": "5f6e7f277febe02e474a1", "size": 940032, "is_downloadable": true, "content_hash": "65e4356436ce73ff1ca4eefbb2bafc3c7214ad560906b82ded01b7401201c4d1"}
< Etag: W/"5f6e7f277febe02e474a1"
< Original-Content-Length: 940032
< X-Server-Response-Time: 324
< Content-Type: application/octet-stream
< Accept-Encoding: identity,gzip
< Date: Sun, 19 Mar 2023 02:15:28 GMT
< Server: envoy
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Robots-Tag: noindex, nofollow, noimageindex
< Content-Length: 940032
< X-Dropbox-Response-Origin: far_remote
< X-Dropbox-Request-Id: 8bba861a24a74a5eb2b0d4d2c7794c4f
< Connection: close
< 
{ [15393 bytes data]
 19  918k   19  176k    0     0  55013      0  0:00:17  0:00:03  0:00:14 55063* transfer closed with 743424 bytes remaining to read
 20  918k   20  192k    0     0  56247      0  0:00:16  0:00:03  0:00:13 56302
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
} [2 bytes data]
curl: (18) transfer closed with 743424 bytes remaining to read

Note that it's easier to reproduce this problem with a throttled connection (e.g. Network Link Conditioner on Macos).

So this must be a problem with dropbox's HTTP server, and I think we should add the Connection: keep-alive workaround for that provider only.

I cannot explain why a VPN helps but I'm thinking VPN might batch TCP packets together differently, so that it doesn't trigger the connection reset before packets have been read.

If we still have ECONNRESET issues in the future we might want to pass the data thru a Passthrough stream with a high highWaterMark like a few megabytes