Allow project-level severity config for specific test types

[joellabes]

Describe the feature

I want up to 10* incorrect records to be allowed on a model before almost any generic test should fail.

Right now, I can:

• set that threshold for all tests. This would cause issues for things like dbt_utils.at_least_one that only return a single "failing row". I could deal with that in turn by overriding those individually, but additional instances of the test would no-op unless we remember to special-case it and that's more fragile than I'd like.
• override the unique test with my own implementation. I don't like to do this either, because it pretty quickly expands to reimplementing my own copy of every test.

Instead, I'd like to be able to do something along the lines of this in dbt_project.yml

tests:
  - name: unique
      +error_if: ">10"
  - name: relationships
      +warn_if: ">10"
      +error_if: ">20"
  dbt_utils:
     - name: at_least_one
        +severity: warn

*I would also like this to be configurable, probably via an environment variable - e.g. I'd like to be able to have a tighter internal rule but our exposure tile job might allow 30 failing records to reflect our true SLA, so that we have time to take action before the tiles turn red

Describe alternatives you've considered

Described above

Who will this benefit?

People who don't like overriding the built in tests

Are you interested in contributing this feature?

No, but I'm also happy to wait for it to come in as part of a wider test configuration revamp

[jtcohen6]

@joellabes Thanks for opening! A totally reasonable thing to want.

I do think the best today is to override/reimplement the generic test definitions, which can set default configs for all specific tests that use it.

As a side effect of the way we've implemented generic tests in dbt, and some of the tests in dbt-utils—separating the test definition from a (dispatched) macro containing its core logic—I think you could do this a bit more subtly than you might think:

-- macros/test_config_overrides.sql

{% test unique(model, column_name) %}
    {{ config(error_if = ">10") }}
    {{ dbt.default__test_unique(model, column_name) }}
{% endtest %}

{% test relationships(model, column_name, to, field) %}
    {{ config(warn_if = ">10", error_if = ">20") }}
    {{ dbt.default__test_relationships(model, column_name, to, field) }}
{% endtest %}

{% test at_least_one(model, column_name) %}
    {{ config(severity = 'warn') }}
    {{ dbt_utils.default__at_least_one(model, column_name) }}
{% endtest %}

As a current workaround, that doesn't feel too shabby. (There are some project parse timing implications of overriding the unique test. We created a "shortcut" for the built-in version, since it's so ubiquitous; a custom implementation just takes slightly longer at parse time.)

I agree that the long-term sustainable answer for this looks much more like:

• Make it possible to define resource properties for tests (e.g. Configurable description in yaml config for generic tests #3249)
• Those resource properties could also include the config: block (introduced in Configs in schema files #3616), functionally equivalent to setting config() inside the test's SQL definition (taking precedence over project-level configs, and preceded by configs on specific tests)
• Make it possible to override properties (including config) for resources defined in packages. Maybe that looks like source overrides, maybe not:

# tests/test_config_overrides.yml

tests:
  - name: unique
    overrides: dbt  # perhaps just call this `package`?
    config:
      error_if: ">10"
  - name: relationships
    overrides: dbt
    config:
      warn_if: ">10"
      error_if: ">20"
   - name: at_least_one
     overrides: dbt_utils
     config:
       severity: warn

[joellabes]

I think you could do this a bit more subtly than you might think

Yeah this is really slick! I'll do that for now 🤩 thanks!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.