-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(bigquery/storage/managedwriter): context refactoring #8275
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR refactors how contexts are retained in the managedwriter package. Prior to multiplexing, each writer had a dedicated connection, and context could be shared between writer and the connection. Now, the relationship between writers and their backing connections is more nuanced. Now, we use the context provided as part of client instantiation (e.g. NewClient) for connection and pool management, and context provided in NewManagedWriter is scoped to _just_ that specific writer. This corrects the previous problem where the context for the first writer in a connection pool would be used for pool and connection management, and thus a cancellation of a single writer's context could trigger writes from other writer instances to fail.
product-auto-label
bot
added
size: m
Pull request size is medium.
api: bigquery
Issues related to the BigQuery API.
labels
Jul 17, 2023
alvarowolfx
approved these changes
Jul 18, 2023
product-auto-label
bot
added
size: l
Pull request size is large.
and removed
size: m
Pull request size is medium.
labels
Jul 20, 2023
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
Jul 24, 2023
🤖 I have created a release *beep* *boop* --- ## [1.53.0](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.52.0...bigquery/v1.53.0) (2023-07-24) ### Features * **bigquery/analyticshub:** Promote to GA ([130c571](https://github.com/googleapis/google-cloud-go/commit/130c5713dcbac7f670cb92ea113dd53d8029c960)) * **bigquery/connection:** Add support for Salesforce connections, which are usable only by allowlisted partners ([bac978a](https://github.com/googleapis/google-cloud-go/commit/bac978ace43bb58db7c0b1475e41c8fdf8c49a29)) * **bigquery/datapolicies:** Promote to GA ([130c571](https://github.com/googleapis/google-cloud-go/commit/130c5713dcbac7f670cb92ea113dd53d8029c960)) * **bigquery/storage:** Add ResourceExhausted to retryable error for Write API unary calls ([#8214](https://github.com/googleapis/google-cloud-go/issues/8214)) ([8ff13bf](https://github.com/googleapis/google-cloud-go/commit/8ff13bf87397ad524019268c1146e44f3c1cd0e6)) ### Bug Fixes * **bigquery/storage/managedwriter:** Context refactoring ([#8275](https://github.com/googleapis/google-cloud-go/issues/8275)) ([c4104ea](https://github.com/googleapis/google-cloud-go/commit/c4104eaab0d7291c15aba37b78e71ce3cbb9f77a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Aug 1, 2023
shollyman
added a commit
to shollyman/google-cloud-go
that referenced
this pull request
Aug 29, 2023
The context refactoring in googleapis#8275 introduced two possible sources of deadlocks in the ManagedStream AppendRows call path. This PR addresses those, and augments deadlock testing to cover this case. Fixes: googleapis#8505
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
Aug 29, 2023
The context refactoring in #8275 introduced two possible sources of deadlocks in the ManagedStream AppendRows call path. This PR addresses those, and augments deadlock testing to cover this case. Fixes: https://github.com/googleapis/google-cloud-go/issues/8505
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors how contexts are propagated/retained in the managedwriter package. Prior to multiplexing, each writer had a dedicated connection, and context could be shared between writer and the connection. Multiplexing decoupled these, and this PR addresses some issues with those changes.
Most notably, the library (incorrectly) used the context provided when instantiating a writer via NewManagedStream to derive the context for the connection pool, which would cause issues if said writer was closed before other writers in the pool completed/closed.
Similarly, (re)-opening a connection would reuse the same context, and certain failure scenarios (namely, server-close on an idle connection) could cause slow leak of grpc resources due to fully signalling disposal of the connection from the client side.
This CL corrects these issues by refactoring some of the key context lifecycles.
First, we use the context provided when instantiating the client (e.g. NewClient) as the root context when constructing connection pool and connections. It also preserves the desired property of making shutdown straightforward.
In the second, two key changes of note:
Finally, this CL adds the requisite additional context expiry checks as we have potentially three contexts of note: the connection, the writer instance, and the request context (for a specific AppendRows request).
Fixes: #8232
Fixes: #8272