-
Notifications
You must be signed in to change notification settings - Fork 748
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
object_store: Retry on connection duration timeouts? #6287
Comments
The golang GCS client supports automatic retrying here via keeping track of how many bytes the client has already seen and creating a new ranged read request to resume the read where the client left off: |
So apparently I feel like the docstring of I'm not that happy yet I'm also not sure if the calls to |
take |
take |
**Is your feature request related to a problem or challenge?
I'm using
object_store
to stream large(r) files to in this case, the body of a HTTP .I essentially do a
store.get(path).await?.into_stream()
to get the data stream.When using it with the Azure backend, I noticed that Azure reliably closes the connection after 30 seconds. Other providers (S3) also explicitly inject errors, but keep most of the error handling in their SDKs.
I know there's some retry logic for some error cases in
object_store
, but the "connection closed while getting data as a stream" part doesn't seem to be covered. I think there should be a high-level function that retries receiving the remaining data in these error cases (verifying the etag is still the same).Describe alternatives you've considered
Manually dealing with error handling and retries in all
object_store
consumersThe text was updated successfully, but these errors were encountered: