Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] --warn-error-options doesn't support configuration of turning source freshness warnings into failures #11116

Open
2 tasks done
aerielsoriano opened this issue Dec 9, 2024 · 1 comment
Assignees
Labels
bug Something isn't working cloud Issues related to dbt Cloud freshness related to the dbt source freshness command

Comments

@aerielsoriano
Copy link

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

This is related to a previous issue where source freshness have also the same dynamic capabilities like tests. While a fix was merged, it wasn't extended to source freshness. The docs mentions source freshness is supported.

Running dbt --warn-error source freshness still returns a WARN rather than an ERROR

Expected Behavior

dbt --warn-error source freshness should return as an ERROR if the freshness falls within the warn_after value

Steps To Reproduce

#models/sources.yml

version: 2

sources:
  - name: jaffle_shop
    description: This is a replica of the Postgres database used by our app.
    database: raw
    schema: jaffle_shop
    tables:
      - name: orders
        description: One record per order. Includes cancelled and deleted orders.
        freshness: 
          warn_after: {count: 1000, period: day}

$ dbt --warn-error source freshness
...
23:32:29  Pulling freshness from warehouse metadata tables for 1 sources
23:32:32  Concurrency: 1 threads (target='dev')
23:32:32  
23:32:32  1 of 1 START freshness of jaffle_shop.orders ................................... [RUN]
23:32:32  1 of 1 WARN freshness of jaffle_shop.orders .................................... [WARN in 0.00s]
23:32:33  
23:32:33  Finished running 1 source in 0 hours 0 minutes and 6.92 seconds (6.92s).
23:32:33  Done.

^^ WARN still appeared even if I included the --warn-error flag

Relevant log output

No response

Environment

- OS:macOS
- Python: 3.12.3
- dbt: dbt-core==1.8.9 / dbt-snowflake==1.8.4

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

@aerielsoriano aerielsoriano added bug Something isn't working triage labels Dec 9, 2024
@dbeatty10 dbeatty10 changed the title [Bug] --warn-error-options doesn't support configuration of turning source freshness warnings into failures [Bug] --warn-error-options doesn't support configuration of turning source freshness warnings into failures Dec 10, 2024
@dbeatty10 dbeatty10 added the freshness related to the dbt source freshness command label Dec 10, 2024
@dbeatty10
Copy link
Contributor

Thank you for reporting this @aerielsoriano !

See below for full details to reproduce this scenario.

Reprex

Create these files:

models/my_model.sql

{{ config(materialized="view") }}

select
    {{ dbt.dateadd(datepart="minute", interval=-60, from_date_or_timestamp=dbt.current_timestamp()) }} as hour_ago,
    1 as id

models/_sources.yml

sources:
  - name: test_project
    database: "{{ target.database }}"  
    schema: "{{ target.schema }}"  
    tables:
      - name: my_model
        freshness:
          warn_after: {count: 59, period: minute}
          error_after: {count: 60, period: minute}
        loaded_at_field: "hour_ago"

Run these commands:

dbt run -s my_model
dbt source freshness --warn-error
echo $?

Get this output:

$ dbt source freshness --warn-error

15:32:29  Running with dbt=1.9.0
15:32:32  Registered adapter: postgres=1.9.0
15:32:32  Found 1 model, 1 source, 429 macros
15:32:32  
15:32:32  Concurrency: 5 threads (target='postgres')
15:32:32  
15:32:33  1 of 1 START freshness of test_project.my_model ................................ [RUN]
15:32:33  1 of 1 WARN freshness of test_project.my_model ................................. [WARN in 0.04s]
15:32:33  
15:32:33  Finished running 1 source in 0 hours 0 minutes and 1.08 seconds (1.08s).
15:32:33  Done.

$ echo $?                          

0

@dbeatty10 dbeatty10 added cloud Issues related to dbt Cloud and removed triage labels Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cloud Issues related to dbt Cloud freshness related to the dbt source freshness command
Projects
None yet
Development

No branches or pull requests

3 participants