BigQuery project reversion in 0.19.1

[max-sixty]

Describe the bug

0.19.1 reverts the behavior around whether a database / project is required in profiles.yml.

For example:

foo:
  target: user
  outputs:
    user:
      type: bigquery
      method: oauth
      dataset: max
      timeout_seconds: 3600
      threads: 24
      # project: x

...fails with

Encountered an error:
Field "database" of type typing.Union[str, NoneType] is missing in dbt.adapters.bigquery.connections.BigQueryCredentials instance

Un-commenting project above works.

Expected behavior

As above, which worked in 0.19.0

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.19.1
   latest version: 0.19.1

Up to date!

Plugins:
  - bigquery: 0.19.1
  - snowflake: 0.19.1
  - redshift: 0.19.1
  - postgres: 0.19.1

The operating system you're using:
MacOS
The output of python --version:
3.8

Additional context

Add any other context about the problem here.

[jtcohen6]

@max-sixty You're right—I'm sorry I didn't catch this during the v0.19.1 testing process.

Let's figure out a fix for this, and a way to add an automated test for this—even if we can't fully integration-test it (since the integration tests run via service account)—to ensure that this doesn't happen again in a future version.

[jtcohen6]

So this still works:

my-bigquery:
  outputs:
    oauth:
      type: bigquery
      method: oauth
      project: "{{ 'None' | as_native }}"
      dataset: dbt_jcohen
  target: oauth

The project field is indeed allowed to be None, and when it is, dbt will use the gcloud-configured default project (per the work in #2908). The issue is when the database/project key is missing entirely, it raises this error:

Field "database" of type typing.Union[str, NoneType] is missing in dbt.adapters.bigquery.connections.BigQueryCredentials instance

@gshank @kwigley Any chance there's an immediate spark of recognition for either of you in this error message?

[kwigley]

The logic to set the project when not provided relied on a hook when creating the credential object that worked well when serializing data before we introduced mashumaro (#2862). mashumaro provides the same hooks, but are named differently.

I added a test to check to cover this case and a quick fix here, it still needs some polish -> 0.19.latest...fix/bigquery-no-project

[jtcohen6]

@kwigley Someone in Slack just saw the same error for their Snowflake profile when authenticator is excluded:

Field "authenticator" of type typing.Union[str, NoneType] is missing in dbt.adapters.snowflake.connections.SnowflakeCredentials instance

Here's the relevant hook:
https://github.com/fishtown-analytics/dbt/blob/cee0bfbfa2596520032b766fd1027fe748777c75/plugins/snowflake/dbt/adapters/snowflake/connections.py#L45-L54

Essentially, it seems like we should find all the places in the codebase that call __post_init__ (8 by my count) and check to see if they should be swapped out for __pre_deserialize__.

UPDATE: False alarm! This was an interaction between dbt-core==0.19.1 and dbt-snowflake==0.19.0.

[jtcohen6]

Hey @max-sixty, we're including a fix for this in v0.19.2, for which we just cut a release candidate. We've been able to add a test for this, and it worked locally for me, but just to be triple sure—any chance you could test locally and confirm that it resolves the regression?

pip install dbt==0.19.2rc1
# or
brew install dbt@0.19.2-rc1

[edited: typo]

[max-sixty]

More than happy to test but are you sure that's right re pip?

❯ pip install --upgrade 0.19.2rc1
ERROR: Could not find a version that satisfies the requirement 0.19.2rc1
ERROR: No matching distribution found for 0.19.2rc1
WARNING: You are using pip version 21.0.1; however, version 21.1 is available.


❯ pip install --upgrade 0.19.2-rc1
ERROR: Could not find a version that satisfies the requirement 0.19.2-rc1
ERROR: No matching distribution found for 0.19.2-rc1
WARNING: You are using pip version 21.0.1; however, version 21.1 is available.

[jtcohen6]

@max-sixty Sorry, silly typo on my part! Just edited

[max-sixty]

@jtcohen6 sorry, not sure how I copy & pasted so blindly!

This works with 0.19.2-rc1, thanks for fixing!