Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2185] [Bug] 502 BadGateway error should retry #562

Closed
2 tasks done
patkearns10 opened this issue Feb 26, 2023 · 1 comment · Fixed by #563
Closed
2 tasks done

[CT-2185] [Bug] 502 BadGateway error should retry #562

patkearns10 opened this issue Feb 26, 2023 · 1 comment · Fixed by #563
Labels
bug Something isn't working

Comments

@patkearns10
Copy link
Contributor

patkearns10 commented Feb 26, 2023

Is this a new bug in dbt-bigquery?

  • I believe this is a new bug in dbt-bigquery
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

User experiences intermittent failure on a job. Some runs are just fine, and occasionally one will error out with below:

Unable to generate access token, if you're using impersonate_service_account, make sure your initial account has the "roles/iam.serviceAccountTokenCreator" role on the account you are trying to impersonate.

and

14:03:39      <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
14:03:39      <p><b>502.</b> <ins>That's an error.</ins>
14:03:39      <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That's all we know.</ins>

Expected Behavior

The error itself says to retry in 30 seconds, so we should add this to the retry logic so that it happens automatically.

Steps To Reproduce

It is very intermittent, so unable to reproduce on my end.

Relevant log output

Unable to generate access token, if you're using impersonate_service_account, make sure your initial account has the "roles/iam.serviceAccountTokenCreator" role on the account you are trying to impersonate.

and

14:03:39      <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
14:03:39      <p><b>502.</b> <ins>That's an error.</ins>
14:03:39      <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That's all we know.</ins>

Environment

- OS:
- Python:
- dbt-core: 1.3
- dbt-bigquery: 1.3

Additional Context

ticket: https://dbtcloud.zendesk.com/agent/tickets/42996

RETRYABLE_ERRORS = (
google.cloud.exceptions.ServerError,
google.cloud.exceptions.BadRequest,
ConnectionResetError,
ConnectionError,
)

does not have the BadGateway 502 in the list of retryable cases and it probably should.
see: https://googleapis.dev/python/google-api-core/latest/exceptions.html

related, but not the same:
#263 (not retried during job timeout)
#231 (larger piece on refactoring bq retries).

@patkearns10 patkearns10 added bug Something isn't working triage labels Feb 26, 2023
@github-actions github-actions bot changed the title [Bug] 502 BadGateway error should retry [CT-2185] [Bug] 502 BadGateway error should retry Feb 26, 2023
@dbeatty10
Copy link
Contributor

@patkearns10 thanks for documenting this issue and opening a PR to fix it! 🤩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants