Disabling package models by default leads to unexpected testing behavior

[soltanianalytics]

Note: I suspect that this behavior is not intended, but I might be wrong, so feel free to close this issue if so.

Describe the bug

It seems that a package that disables all models by default and re-enables them via config and variables leads to disabled schema tests for enabled models.

Steps To Reproduce

I am developing a small package and am currently creating a model in this package. This model is called gemma_fx.sql. The first line of the model is {{ config(enabled=var("gemma:fx:enabled")) }}.

The model is also documented and has tests as follows:

version: 2

models:
  - name: gemma_fx
    description: "A model containing exchange rate data: one row per date and currency."
    tests:
      - unique:
          column_name: concat(date, currency)
    columns:
      - name: date
        tests:
          - not_null
      - name: currency
        tests:
          - not_null
      - name: fx_rate_usd
        description: USD/XXX exchange rate. Divide an amount in the original currency by this number of get a USD amount. Multiply a USD amount with this number to get an amount in the respective currency.
        tests:
          - not_null

The package's dbt_project.yml looks as follows:

config-version: 2

name: 'gemma_dbt_utils'
version: '0.2.8'

require-dbt-version: '>=0.17.0'

profile: 'gemma_dbt_utils_profile'

target-path: "target"
clean-targets:
  - "target"
  - "dbt_modules"
log-path: "logs"

macro-paths: ["macros"]
source-paths: ["models"]

vars:
  # Variables related to the fx model
  'gemma:fx:currencies':
    EUR: 'gemma:fx:eur'  # define currency as other variables, or ...
    CHF: 'raw_fx.daily_fx_rates_usd_chf'  # ... name table directly
  'gemma:fx:eur': "{{ source('fx', 'eur') }}"  # must use this approach to use source macro
  'gemma:fx:enabled': false  # overwrite this variable to enable the fx model

models:
  gemma_dbt_utils:
    # Defaults for all models in package
    +schema: gemma_dbt_utils
    +tags: gemma_dbt_utils
    +materialized: table
    +enabled: false  # models need to be activated on purpose

Therefore, I disabled the model by default, but any project that uses the model can activate the model by setting the variable gemma:fx:enabled to true.

Very much like the dbt utils, I created an integration_tests subfolder where I install the package via symlink to develop and test it. That project's dbt_project.yml file vars section looks like that:

vars:
  # Set variables to configure the Gemma dbt utils package
  'gemma:fx:currencies':
    EUR: 'gemma:fx:eur'
    CHF: 'gemma:fx:chf'
  'gemma:fx:eur': "{{ source('fx', 'eur') }}"
  'gemma:fx:chf': "{{ source('fx', 'chf') }}"
  'gemma:fx:enabled': true

Running dbt run works properly. Running dbt test does not run the tests I have defined.

Interestingly enough, if I delete the last line in the package's dbt_project.yml file, +enabled: false # models need to be activated on purpose, it does work.

Expected behavior

In the above setup, changing the last line of the package's dbt_project.yml file should have no effect, since the model is still disabled via variable by default and enabled by variable in the project that uses the package.

I'd also prefer if this worked properly, because that way all models in the package can be disabled by default, with or without the use of variables.

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.19.0
   latest version: 0.19.0

Up to date!

Plugins:
  - bigquery: 0.19.0
  - postgres: 0.19.0
  - redshift: 0.19.0
  - snowflake: 0.19.0

The operating system you're using: Win 10

The output of python --version: Python 3.7.2

[jtcohen6]

Thanks @soltanianalytics! This is a tricky one. I think you've identified a real bug that exists at the intersection of tests, vars, packages, and model configs. I really appreciate the detailed write-up and reproduction case.

For the specific use case you're describing, I'd recommend a different approach. Instead of using a variable ('gemma:fx:enabled') to switch on the models in gemma_dbt_utils (off by default), I'd recommend that the root project simply add:

models:
  gemma_dbt_utils:
    +enabled: true

This config overrides any enabled: false configs set within the package, both gemma_dbt_utils/dbt_project.yml and in-model config(). Both dbt run and dbt test work as expected.

Since there's a reasonable workaround today, fixing this specific issue isn't going to be a huge priority, but I think we may end up with another indirect resolution. For the next version of dbt, we're planning work around tests, including the ability to enable or disable both schema and data tests from dbt_project.yml. That work would also make it possible to disable tests from an installed package, or (as in this case) vice versa.

[soltanianalytics]

Thanks for the workaround! I'm staying with the one I described above, though. If I do it via root project, I would have two different places in the root where I'd have to make changes to activate the model (once in models and once in vars). I'd rather have it all in once place (i.e., vars) - just personal preference. This works as long as I don't deactivate all models in the package's dbt_project.yml, which would then cause the buggy behavior.

[jtcohen6]

I just triaged a very similar issue (#3591), and I realized you had it right all along:

Interestingly enough, if I delete the last line in the package's dbt_project.yml file, +enabled: false # models need to be activated on purpose, it does work.

In older versions of dbt, tests were lumped in with project-level models configs. In v0.20, we introduced a test-specific project config (#3253)—and incidentally fixed this issue.

I confirmed by trying the setup described above with v0.20.0:

$ dbt test
Running with dbt=0.20.0
Found 1 model, 4 tests, 0 snapshots, 1 analysis, 334 macros, 0 operations, 0 seed files, 0 sources, 0 exposures

12:17:58 | Concurrency: 1 threads (target='dev')
12:17:58 |
12:17:58 | 1 of 4 START test not_null_gemma_fx_currency......................... [RUN]
12:17:58 | 1 of 4 PASS not_null_gemma_fx_currency............................... [PASS in 0.05s]
12:17:58 | 2 of 4 START test not_null_gemma_fx_date............................. [RUN]
12:17:58 | 2 of 4 PASS not_null_gemma_fx_date................................... [PASS in 0.02s]
12:17:58 | 3 of 4 START test not_null_gemma_fx_fx_rate_usd...................... [RUN]
12:17:58 | 3 of 4 PASS not_null_gemma_fx_fx_rate_usd............................ [PASS in 0.03s]
12:17:58 | 4 of 4 START test unique_gemma_fx_concat_date_currency_.............. [RUN]
12:17:58 | 4 of 4 PASS unique_gemma_fx_concat_date_currency_.................... [PASS in 0.03s]
12:17:58 |
12:17:58 | Finished running 4 tests in 0.48s.

[soltanianalytics]

I very much appreciate you getting back to me with the update!
I’ll be taking 0.20 for a spin shortly :)