Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry (exponential backoff) for storage api functions #108

Closed
brfulu opened this issue Apr 15, 2020 · 3 comments · Fixed by #287
Closed

Add retry (exponential backoff) for storage api functions #108

brfulu opened this issue Apr 15, 2020 · 3 comments · Fixed by #287
Assignees
Labels
api: storage Issues related to the googleapis/python-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@brfulu
Copy link

brfulu commented Apr 15, 2020

While using the python-storage library, we were getting 503 or 504 errors from time to time when deleting or moving objects.
I looked up the docs for cloud storage and came across this:

Clients should use truncated exponential backoff for all requests to Cloud Storage that return HTTP 5xx and 429 response codes, including uploads and downloads of data or metadata.

At the bottom of the page there is also:

Examples of backoff implemented in client libraries that you can use with Cloud Storage include:

  • Retrying library for Python.
  • Google Cloud Client Libraries for Node.js can automatically use backoff strategies to retry requests with the autoRetry parameter.

It would be super useful and in my opinion necessary to include retrying functionality into the python-storage library.
It was also a bit confusing to me that the official docs are recommending a third party retrying library while already having google.api.core.retry inside google cloud python sdk.

The google.api.core.retry docs mention this:

Some client library methods apply retry automatically. These methods can accept a retry parameter that allows you to configure the behavior

A potential solution could be to add the retry parameter of type google.api.core.retry to public api functions or while creating the storage client. Developers could specify if they want exponential backoff and set the parameters as they like (maximum delay, deadline, etc.)

Java and Node.js libraries already have similar functionality and it seems that this can be useful when cloud storage is facing some issues or high traffic load, or in the case where users are hitting the limits of 1k writes or 5k reads per second and GCS is autoscaling.

If this is already supported in the library, please direct me to some docs, I wasn't able to find it...

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/python-storage API. label Apr 15, 2020
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Apr 16, 2020
@frankyn frankyn added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed triage me I really want to be triaged. labels Apr 17, 2020
@brfulu
Copy link
Author

brfulu commented May 6, 2020

Hey everyone, is there any update about this?

@frankyn frankyn self-assigned this May 12, 2020
@frankyn
Copy link
Member

frankyn commented May 12, 2020

I'm working on this and don't have an update yet to share. I will update this issue when I do. Thank you the ping.

@simonvanderveldt
Copy link

simonvanderveldt commented Jul 14, 2020

There's a prototype PR for this now #197

We'd very much welcome functionality like this, currently we have to implement the retrying ourselves which is a bit tedious.

@tseaver tseaver changed the title Storage: Add retry (exponential backoff) for storage api functions Add retry (exponential backoff) for storage api functions Aug 17, 2020
@frankyn frankyn assigned andrewsg and unassigned frankyn Sep 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants