Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle case of 0 connect retries better for HTTP Spark Connections #132

Merged
merged 1 commit into from
Dec 31, 2020

Conversation

danielvdende
Copy link
Contributor

resolves #131

Description

With a default of 0 connect_retries, confusing log messages show up. (see example below)

Running with dbt=0.18.1
Found 5 models, 12 tests, 0 snapshots, 0 analyses, 158 macros, 0 operations, 0 seed files, 3 sources

Warning: TEMPORARILY_UNAVAILABLE: Cluster <my-cluster-name> is temporarily unavailable. The current cluster state is Terminated. Please retry your request after 30 seconds.
        Retrying in 10 seconds (0 of 0)
Encountered an error:
Runtime Error
  Runtime Error
    TEMPORARILY_UNAVAILABLE: Cluster <my-cluster-name> is temporarily unavailable. The current cluster state is Terminated. Please retry your request after 30 seconds.

We encountered this while working with Databricks, where we had a cluster that had not been started yet.
The logs suggest that a retry is going to be carried out, but it isn't (because the retry count defaults to 0). Also, a sleep is carried out, even if no retries are to happen. This check prevents this from happening.

Checklist

  • [ x ] I have signed the CLA
  • [ x ] I have run this code in development and it appears to resolve the stated issue
  • [ x ] This PR includes tests, or tests are not required/relevant for this PR
  • [ x ] I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

@cla-bot cla-bot bot added the cla:yes label Dec 23, 2020
@jtcohen6
Copy link
Contributor

Thanks for the contribution @danielvdende! This change makes sense. Could you add a changelog entry, and add yourself to the list of contributors?

The failing test looks unrelated—seems to be an issue with executing a set statement through the new Databricks SQL endpoint.

With a default of 0 connect_retries, confusing log messages show up.
The logs suggest that a retry is going to be carried out, but it isn't.
Also, a sleep is carried out, even if no retries are to happen. This
check prevents this from happening.
@danielvdende
Copy link
Contributor Author

Done @jtcohen6 ! Thanks for the swift response 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Confusing log messages when using default connect_retries for HTTP connection
2 participants