Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bigquery/storage/managedwriter: Potential deadlock on error of ManagedStream in AppendRows #8505

Closed
jonomacd opened this issue Aug 29, 2023 · 1 comment · Fixed by #8507
Closed
Assignees
Labels
api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@jonomacd
Copy link

jonomacd commented Aug 29, 2023

Client

BigQuery

Environment

local

Go Environment

any go env

Code

https://github.com/googleapis/google-cloud-go/blob/bigquery/v1.54.0/bigquery/storage/managedwriter/managed_stream.go#L293-L297

	ms.mu.Lock()
	if ms.err != nil {
		return nil, ms.err
	}
	ms.mu.Unlock()

Expected behavior

The managed writer should check an error and release the lock

Actual behavior

If an error case is detected on the ManagedStream we return early and will not release the mutex. Other writers using this stream will deadlock.

Suggested Fix

	ms.mu.Lock()
        err := ms.err
        ms.mu.Unlock()
	
       if err != nil {
		return nil, err
	}

EDIT: The same issue can be seen here in appendWithRetry
https://github.com/googleapis/google-cloud-go/blob/bigquery/v1.54.0/bigquery/storage/managedwriter/managed_stream.go#L199-L203

@jonomacd jonomacd added the triage me I really want to be triaged. label Aug 29, 2023
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the BigQuery API. label Aug 29, 2023
@shollyman
Copy link
Contributor

Thanks for the report. Appears I introduced these while dealing with the context changes in #8275, will have a PR out shortly.

@shollyman shollyman added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. and removed triage me I really want to be triaged. labels Aug 29, 2023
shollyman added a commit to shollyman/google-cloud-go that referenced this issue Aug 29, 2023
The context refactoring in googleapis#8275 introduced two possible sources of
deadlocks in the ManagedStream AppendRows call path.  This PR addresses
those, and augments deadlock testing to cover this case.

Fixes: googleapis#8505
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants