[Bug] Not able to select unit tests via the --resource-type flag

[thijs-nijhuis]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

We are implementing the new 'unit test' feature; which is great by the way! I have added multiple unit test and tagged them with with 'my_unit_test_tag'.
When I run this, all unit test run and succeed: dbt test --select tag:my_unit_test_tag . Using 'dbt build' instead of 'dbt test' also works fine.
When I run this though, I get the "Nothing to do. Try checking your model configs and model specification args" message: dbt build --resource-type unit_test . I alaso tried with 'resource-types' instead of 'resource-type'; same message.

Expected Behavior

According to this page, this command should build all unit test: https://docs.getdbt.com/reference/global-configs/resource-type

Steps To Reproduce

1. Create a unit test and make sure it runs properly
2. execute dbt build --resource-type unit_test . This should run the ut.

Relevant log output

No response

Environment

- OS: Windows 11 enterprise
- Python: 3.11.4
- dbt-core: 1.8.2

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

I am using dbt-databricks 1.8.1 (so dbt-spark I guess).

[dbeatty10]

Thanks for reporting this @thijs-nijhuis-shell !

Reprex

✅ When using this simple example, these all work for me:

dbt list --select "resource_type:unit_test"
dbt list --select "test_type:unit" 
dbt list --select "test_name:test_hello_world" 
dbt list --select "test_hello_world" 
dbt list --resource-type unit_test

✅ And so do all of these (except for the --resource-type one because that flag isn't supported by dbt test):

dbt test --select "resource_type:unit_test"
dbt test --select "test_type:unit" 
dbt test --select "test_name:test_hello_world" 
dbt test --select "test_hello_world" 
dbt test --resource-type unit_test

❌ But none of these work -- I get " Nothing to do" for all of them:

dbt build --select "resource_type:unit_test"
dbt build --select "test_type:unit" 
dbt build --select "test_name:test_hello_world" 
dbt build --select "test_hello_world" 
dbt build --resource-type unit_test

But let's say I add a data test like the following:

models/_models.yml

models:
  - name: hello_world
    columns:
      - name: hello
        tests:
          - not_null

Most of these commands will work (see below for more explanation on the exceptions):

dbt build --select "resource_type:test"
dbt build --select "test_type:data" 
dbt build --select "test_name:not_null" 
dbt build --select "not_null_hello_world_hello" 
dbt build --resource-type test

❌ These two will execute without error, but the included types are not consistent (only data tests for the former but both data tests and unit tests for the latter):

dbt build --select "resource_type:test"
dbt build --resource-type test

❌ But using data_test doesn't work at all:

dbt build --select "resource_type:data_test"
dbt build --resource-type data_test

This will include both unit tests and data tests (which seems rational since building a model includes running its unit tests first, materializing the model and then running its data tests):

dbt build --select "resource_type:model"

[thijs-nijhuis]

Hi @dbeatty10 , thanks for elaborate response and testing on the one!
One difference I noticed on my end is this command dbt test --resource-type unit_test. For me this yields an error Error: No such option: --resource-type. When I run dbt test -h the 'resource-type(s)', and 'exclude-resource-type(s)' are all not present as an option. Would be a nice addition to have these available for the test command as well btw!

[dbeatty10]

@thijs-nijhuis-shell I got the same thing as you with dbt test --resource-type unit_test -- Error: No such option: --resource-type. We should get the same error message if we do dbt run --resource-type unit_test.

Then

Historically, is presumably because dbt build and dbt list both support multiple resource types, but dbt test and dbt run each only supported one resource type (data tests and models, respectively).

Now

But as-of v1.8, dbt test includes two resource types: data tests and unit tests.

Workaround

So I could see us adding --resource-type / --exclude-resource-type to dbt test, but it probably wouldn't be a high priority since unit tests and data tests can be separated like the following:

dbt test --select "resource_type:unit_test"
dbt test --select "resource_type:test"

[thijs-nijhuis]

@dbeatty10 , makes sense. Thanks!
And indeed, dbt test --select "resource_type:unit_test" works for me as well.

[seub]

Piggybacking, I noted similar bugs:

dbt ls --resource-type=test
dbt ls --select resource_type:test

The first command picks up unit tests, the second doesn't.

dbt ls --resource-type=model
dbt ls --select resource_type:model

The first command only picks up models, the second picks up models and tests.

Tested with dbt 1.8.3

[dbeatty10]

@seub to your point, I agree that we should rationalize --resource-type vs. --select resource_type: across dbt sub-commands and resource types so that the behavior is both understandable and consistent.

In the meantime, see below for example workarounds on a sub-command & resource type basis.

Example commands

Here are some examples to select only select a specific resource type for a handful of dbt commands (list, run, test, and build) and resource types (unit_test, model, and data_test):

dbt list

dbt list unit tests

dbt list --select "resource_type:unit_test"
dbt list --select "test_type:unit"
dbt list --resource-type unit_test

dbt list models

dbt list --resource-type model
dbt list --select "resource_type:model" --exclude "test_type:unit" "test_type:data"

dbt list data tests

dbt list --select "resource_type:test"
dbt list --select "test_type:data"
dbt list --resource-type test --exclude "test_type:unit"

dbt run

dbt run unit tests

Note

N/A since dbt run only runs models (and doesn't run unit tests) -- see dbt test or dbt build to execute unit tests.

dbt run models

dbt run
dbt run --select "resource_type:model"

dbt run data tests

Note

N/A since dbt run only runs models (and doesn't run data tests) -- see dbt test or dbt build to execute data tests.

dbt test

dbt test unit tests

dbt test --select "resource_type:unit_test"
dbt test --select "test_type:unit"

dbt test models

Since dbt test only runs tests (and doesn't run models), the following example is expected to do nothing:

dbt test --select "resource_type:model" --exclude "test_type:unit" "test_type:data"

dbt test data tests

dbt test --select "resource_type:test"
dbt test --select "test_type:data"

dbt build

dbt build unit tests

Important

I didn't find any examples that worked to build only unit tests using the dbt build subcommand -- I needed to use one of the dbt test examples above instead.

dbt build models

dbt build --resource-type model
dbt build --select "resource_type:model" --exclude "test_type:unit" "test_type:data"

dbt build data tests

dbt build --select "resource_type:test"
dbt build --select "test_type:data"
dbt build --resource-type test --exclude "test_type:unit"

[dbeatty10]

A few different things came up in the course of this thread:

1. dbt build --resource-type unit_test always says "Nothing to do" no matter what (original issue) [1]
2. dbt test doesn't support --resource-type / --exclude-resource-type flags (but it could!) [2]
3. inconsistencies between --select "resource_type:" and --resource-type [3]

For the first one, the relevant code was introduced in #9273. It looks to me like dbt build will only run unit tests whose associated model is also included in the selection. If possible/practical, we'll want to update the behavior so that unit tests can be included in dbt build even when their associated model is not included in the selection.

The second is covered by #10656.

I'm not sure the root cause(s) of the third one, and we'll want to create a separate issue for it.

[dbeatty10]

Would be a nice addition to have these available for the test command as well btw!

@thijs-nijhuis good news: this was implemented in #10706, and it will be available in v1.9

I'm not sure the root cause(s) of the third one, and we'll want to create a separate issue for it.

@seub Here's the issue for the 3rd one: #10753