Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws s3 cp keeps uploading multipart parts when a previous part has failed #834

Closed
jlopex opened this issue Jul 2, 2014 · 1 comment
Closed

aws s3 cp keeps uploading multipart parts when a previous part has failed #834

jlopex opened this issue Jul 2, 2014 · 1 comment

Comments

@jlopex
Copy link
Contributor

jlopex commented Jul 2, 2014

I noticed that the tool keeps uploading following parts even if a previous (necessary) part has failed while being uploaded.

Retry mechanism is provided by botocore, if botocore is not able to complete the upload operation before max_attempts threshold is reached, the operate call will raise a ConnectionError exception.

Current UploadTask logic handles this case properly so whenever it gets an Exception, the failed task sets the shared MultipartUploadContext state to _CANCELLED.

Unfortunately queued UploadTasks doesn't seem to honor the MultipartUploadContext state and keeps trying to upload more parts, these uploadTask are useless as the uploaded file won't be complete.
jamesls added a commit that referenced this issue Jul 7, 2014
@jamesls
Merge branch 'jlopex-cancel_pending_tasks' into develop
195569b 
* jlopex-cancel_pending_tasks:
  Added test_cancel_after_upload_id to test_tasks
  Fixes issue #834
jamesls added a commit that referenced this issue Jul 7, 2014
@jamesls
Add changelog entry for #834
043f126
@jamesls
Copy link
Member

jamesls commented Jul 8, 2014

Fixed via #835

@jamesls jamesls closed this as completed Jul 8, 2014
jamesls added a commit that referenced this issue Jul 10, 2014
@jamesls
Merge branch 'release-1.3.22'
0acbafd 
* release-1.3.22:
  Bumping version to 1.3.22
  Update CHANGELOG with the latest features
  Update completer test with new services
  Update changelog with #825
  Add changelog entry for #834
  Fix changelog entry for merge of #831
  Added test_cancel_after_upload_id to test_tasks
  Update changelog with fix for #549
  Disable fix_s3_host when --endpoint-url is given
  Fixes issue #834
  Update changelog with bugfix
  Add validation to ensure we don't mv a file onto itself
  Let aws.cmd find python.exe on paths with spaces.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jamesls @jlopex