Copying large files fails after 5s timeout

[Licenser]

It seems one of the processes in the chain of storing a file chunk is set up with the default 5s timeout, this leads to the situation where copying large files fails with timeouts to the point where it's impossible to uplaod anything larger

Schrödinger:heinz heinz$ s3cmd put zmq.tar.gz s3://heinz
zmq.tar.gz -> s3://heinz/zmq.tar.gz  [1 of 1]
  5181440 of 14353743    36% in    5s   992.70 kB/s  failed
WARNING: Upload failed: /zmq.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
zmq.tar.gz -> s3://heinz/zmq.tar.gz  [1 of 1]
  5275648 of 14353743    36% in    5s   999.79 kB/s  failed
WARNING: Upload failed: /zmq.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
zmq.tar.gz -> s3://heinz/zmq.tar.gz  [1 of 1]
  1871872 of 14353743    13% in    5s   356.83 kB/s  failed
WARNING: Upload failed: /zmq.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
ozmq.tar.gz -> s3://heinz/zmq.tar.gz  [1 of 1]
   405504 of 14353743     2% in    5s    77.21 kB/s  failed
WARNING: Upload failed: /zmq.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
zmq.tar.gz -> s3://heinz/zmq.tar.gz  [1 of 1]
    86016 of 14353743     0% in    5s    16.56 kB/s  failed
WARNING: Upload failed: /zmq.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
zmq.tar.gz -> s3://heinz/zmq.tar.gz  [1 of 1]
    24576 of 14353743     0% in    6s     3.81 kB/s  failed

[Licenser]

A workaround for this issue is to change the large_object.chunked_obj_len setting.

[yosukehara]

Thank you for posting this issue.
Do you think you could share "~/.s3cfg” and "app.config (gateway)"?

[Licenser]

here you go: https://gist.github.com/Licenser/7370772

The chnage I did was dividing the sizes in the app.config by 5 that way each chunk took less them 5s but this is specific to my connection speed, someone with a slower connection might still face the problem.

[yosukehara]

Thank you for sharing the configurations. I'll check that on our environment.

[Licenser]

This will only affect you when your connection is slow enough so that a single chunk takes > 5s, in a LAN w/o throtteling you will propably not be reproducable w/o setting the chunk size very high

[yosukehara]

I fixed this issue but it's a temporary solution.
So I asked Cowboy creator, Loïc to modify and support it.

1. leo-project/cowboy@4a5ad0e
2. leo-project/cowboy@a941e7f

[Licenser]

I wonder if it would be possible to read the big chunk in smaller chunks. That would make it possible to not timeout over the whole but over (configurable amount of data) chunks. So the DB could still store 5MB but the chunks read by cowboy are much smaller to handle the timeout.

An algorithm would come to mind that reads chunks, starting small, and growing them until the time to read that chunk takes approximately 50% of the timeout, and scales them down when it reaches 75% (just a mad thought).

[yosukehara]

It's a good idea. We have been still considering this issue. I'm planning to support as the follows:

• 1st step: Able to be configurable "read chunk size", which default value is 5MB (In case of using narrow band network/ low speed network, you're able to set less 5MB)
• 2nd step: Able to dynamically change "read chunk size" depends on using network.

We're going to implement the 1st step at first.

Thanks a lot.

[Licenser]

Just a update here, the issue has changed slighty but still persists,
in current development the timeout the following error is shown for uploads:

[E]     gateway_0@127.0.0.1     2013-12-06 04:44:47.143154 +0000        1386305087      leo_gateway_http_commons:put_large_object/7     479     key:fifo/base64-13.2.1.zfs.gz
1, cause:timeout
[E]     gateway_0@127.0.0.1     2013-12-06 04:44:47.144849 +0000        1386305087      leo_gateway_large_object_handler:delete_chunked_objects/2       280     key:fifo/base64-13.2.1.zfs.gz
1, index:1, cause:not_found

This only gets triggered after an full part of the multipart upload is done and does not interrupt the part in the middle, still changing the chunk size in the gateway config will work around this.

[yosukehara]

Thank you for your report. We're planning to solve it with the next minor version as the 1st step.

[Licenser]

I know :) just wanted to update the symtoms since they changed.

[yosukehara]

I have implemented the 1st step
but I does not merge the master branch. It will be included in the next minor version.

[yosukehara]

I've close this issue. If you and we need the 2nd step you or we will reopen it.