Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3513] [Bug] tags in properties/schema/etc yml files do not support dbt run --select tag:my_tag #9317

Closed
2 tasks done
IlyaDropit opened this issue Dec 26, 2023 · 2 comments
Labels
bug Something isn't working node selection Functionality and syntax for selecting DAG nodes wontfix Not a bug or out of scope for dbt-core

Comments

@IlyaDropit
Copy link

IlyaDropit commented Dec 26, 2023

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When adding tags in schema.yml/some_yml_transformed_tables.yml, they are not seeing by dbt run command. However, dbt test is working. And I want the super clear here - I apply tags to fact or dimension, tables I transformed with dbt. It was tested with both variants from dbt docs with dbt version 1.7.3 and 1.7.4 (bellow are code blocks from docs, I was applying tags to actual models so names maybe misleading for my case):

  1. Without config property
version: 2

exposures:
  - name: my_exposure
    tags: ['exposure_tag']
    ...

sources: (in my specific case - models only)
  - name: source_name
    tags: ['top_level']

    tables:
      - name: table_name
        tags: ['table_level']

        columns:
          - name: column_name
            tags: ['column_level']
            tests:
              - unique:
                  tags: ['test_level']
  1. Config property (second tab on the very top of tags docs - even though it is a bit confusing, since at the end of the page docs mention that "resources do not yet support the config property, so you'll need to specify the tags as a top-level key instead."):
version: 2

models:
  - name: model_name
    config:
      tags: <string> | [<string>]

    columns:
      - name: column_name
        tags: [<string>]
        tests:
          <test-name>:
            config:
              tags: <string> | [<string>]

Please, note that this is not about dbt_project.yml tags which are working fine.

Tried also running models that are downstream or upstream - didn't work (dbt run -s tag:my_tag+ or dbt run -s +tag:shopify)

Expected Behavior

According to documentation, these commands are supported:

  • dbt run --select tag:my_tag
  • dbt seed --select tag:my_tag
  • dbt snapshot --select tag:my_tag
  • dbt test --select tag:my_tag

There is no information on restrictions of tags defined in yml files in dbt's documentation.

Apart from the documentation, I found related tickets that also suggest that tags are supported in the latest versions of dbt in some_file.yml:

#2957 (comment)
#8749

There is also a post in dbt Slack community describing exactly the same problem.

Steps To Reproduce

  1. Need a project with several models so that there can be a source or model with several tables
  2. Recent versions of dbt with Postgres adapter (1.7.3 or 1.7.4)
  3. Make sure there are not tags in your models in config blocks or that they have different names from what you will place in yml
  4. Add tags under models name in your .yml file where you keep your transformed table and tests without config property:
models:
  - name: fact_name
    tags: ['tag_name']
  1. Try dbt run -s tag:tag_name or dbt run -s tag:shopify+ - whatever is relevant for your project
  2. Try dbt test -s tag:tag_name or dbt test -s +tag:shopify+
  3. Now try the same, but under table name

dbt run suppose to return Nothing to do. Try checking your model configs and model specification args, but dbt test should work.

Relevant log output

12:51:58  Running with dbt=1.7.4
12:51:58  Registered adapter: postgres=1.7.4
12:51:58  Found xx models, xx analyses, xx tests, xx seeds, xx operations, xx sources, xx exposures, xx metrics, xxx macros, x groups, x semantic models
12:51:58  
12:51:58  Nothing to do. Try checking your model configs and model specification args

Environment

- OS: macOS Sonoma 14.0
- Python: 3.11.16
- dbt: dbt-core 1.7.4

Which database adapter are you using with dbt?

postgres

Additional Context

Up to now tags were in dbt_project.yml defined on each folder level. Occasionally, tags were defined in config blocks inside models. Both are working fine, no bugs were noticed.
For convenience and efficiency I need to define tags on a more granular level, preferably in yml file where I keep tests and descriptions for materialized tablea to add them under model/source name. Unfortunately, it didn't work. Tried to remove all tags from config blocks - didn't work either.

@IlyaDropit IlyaDropit added bug Something isn't working triage labels Dec 26, 2023
@github-actions github-actions bot changed the title [Bug] tags in properties/sources yml files do not support dbt run --select tag:my_tag [CT-3513] [Bug] tags in properties/sources yml files do not support dbt run --select tag:my_tag Dec 26, 2023
@IlyaDropit
Copy link
Author

UPD: after additional checks, I also found that dbt seed does not work with such tags and dbt build would do only test and would not proceed to run.

Will test on a different machine to see if it is also a case with other os or python version.

@IlyaDropit IlyaDropit changed the title [CT-3513] [Bug] tags in properties/sources yml files do not support dbt run --select tag:my_tag [CT-3513] [Bug] tags in properties/schema/etc yml files do not support dbt run --select tag:my_tag Dec 28, 2023
@dbeatty10 dbeatty10 self-assigned this Jan 2, 2024
@dbeatty10
Copy link
Contributor

Thanks for reaching out @IlyaDropit !

Are you trying to point out that column-level tags affect dbt test but not dbt run?

If so, this is expected behavior rather than a bug.

The key insight is that depending on which "level" you are adding a tag, it is either being applied to a model resource or a test resource. Both model and test resources are considered during dbt test, but only model resources are considered during dbt run.

So if you want a tag to affect dbt run, then it needs to be at the model level rather than at the column level.

Example

Suppose you have the following project files:

Click here to toggle project files

dbt_project.yml

name: "my_project"
version: "1.0.0"
config-version: 2
profile: "postgres"

models:
  my_project:
    tags: ["all_project_models_tag"]

models/_properties.yml

models:
  - name: model_a
    config:
      tags: "model_properties_tag"
    columns:
      - name: id
        tags: ["column_tag"]

        tests:
          - unique:
              tags: ["column_test_tag"]

models/model_a.sql

{{ config(tags="model_config_tag") }}

select 1 as id

Each of the following commands will select at least one resource (which will be the model resource type for the first 3 and test resource type for the last 2):

dbt list --select tag:all_project_models_tag
dbt list --select tag:model_config_tag
dbt list --select tag:model_properties_tag
dbt list --select tag:column_tag
dbt list --select tag:column_test_tag

The following commands will run all the test resources that are associated with the given tags:

dbt test --select tag:all_project_models_tag
dbt test --select tag:model_config_tag
dbt test --select tag:model_properties_tag
dbt test --select tag:column_tag
dbt test --select tag:column_test_tag

The following commands will run all the model resources that are associated with the given tags:

dbt run --select tag:all_project_models_tag
dbt run --select tag:model_config_tag
dbt run --select tag:model_properties_tag
dbt run --select tag:column_tag
dbt run --select tag:column_test_tag

Because models don't inherit tags from column tests, the last two commands will both output something like:

Nothing to do. Try checking your model configs and model specification args

This is expected behavior, and we aren't planning to change it at this time. So I'm going to close this as "not planned". Just let me know if I've misinterpreted something, and we can take another look.

@dbeatty10 dbeatty10 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 2, 2024
@dbeatty10 dbeatty10 added wontfix Not a bug or out of scope for dbt-core node selection Functionality and syntax for selecting DAG nodes and removed triage labels Jan 2, 2024
@dbeatty10 dbeatty10 removed their assignment Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working node selection Functionality and syntax for selecting DAG nodes wontfix Not a bug or out of scope for dbt-core
Projects
None yet
Development

No branches or pull requests

2 participants