schema tests defined by macros

[drewbanin]

• use "globally" scoped macros instead of hardcoded sql in python files
• supports user-defined schema tests (via macros) custom schema tests (via macros) #200

To create a custom schema test, create macros prefixed with test_. The test_ macros must take model as the first argument. If the test takes a single argument (as with not null & unique tests), then the second argument should be called arg. Otherwise, the remaining arguments can be named as you like. Test configurations stated in schema.yml will be mapped to the macro arguments as long as the names match. For example:

-- macros/tests.sql
{% macro test_something(model, arg) %}

select *
from {{ ref(model) }}

where id = '{{ arg }}'

{% endmacro %}


{% macro test_something_complicated(model, first_argument, another_argument) %}

select *
from {{ ref(model) }}

where id = '{{ first_argument }}'
   or id = '{{ another_argument }}'

{% endmacro %}

some_model:
  constraints:
    my_project.test_something:
      - abc
      - def

    my_project.test_something_complicated:
      - {first_argument: abc, another_argument: def}

[cmcarthur]

can this be anything but a dict? looks like below we make sure it's a dict before calling this. parts = kwargs.values() should cover all the cases, right?

[drewbanin]

This func takes arbitrary schema test configs and spits out a human readable, unique name.

my_model:
  constraints:
    test_something:
      - { some_value: True, other_thing: ['abc', 'def'] }

So this func would operate on { some_value: True, other_thing: ['abc', 'def'] } and spit out:

test_something_my_model_abc_def__True

I actually think that in practice this is kind of annoying/confusing. This "nice" name becomes the compiled filename and shows up in the dbt test output. Ideally, we'd keep the test config args around and show something like:

ERROR running test "test_something" for model "my_model" with args:
  some_value: True
  other_thing: ['abc', 'def']

but that's not really how things work currently. Something to consider for the future though

[drewbanin]

oh gosh, and to answer your question, yes: The arg_val var will be the value of each item in the supplied dict (args). So here it is a bool, then a dict. could also be a string/list/int etc

[cmcarthur]

use dbt.compat.basestring

[drewbanin]

👍

[cmcarthur]

is there a time when we would not want to include globals?

[drewbanin]

nope, i'll remove this

[drewbanin]

@cmcarthur how do you feel about this?

[cmcarthur]

i feel ok about it. not in love with regex parsing but what can you do.

is the intention to support ref and var in custom schema tests? it doesn't look like those would be passed in here unless i'm missing something

[cmcarthur]

i feel ok about it. not in love with regex parsing but what can you do.

is the intention to support ref and var in custom schema tests? it doesn't look like those would be passed in here unless i'm missing something

[cmcarthur]

ahh i understand now. is this a breaking change in how schema tests are defined?

[drewbanin]

Yeah, sorry for the lack of context. It is a breaking change, but we don't really have much of a choice! I think it's more canonical like this anyway -- kind of weird that the ref was implicit before. I ran it by @jthandy and he agreed

…

On Mar 25, 2017, at 8:28 PM, Connor McArthur ***@***.***> wrote: @cmcarthur commented on this pull request. In dbt/parser.py: > - raw_sql = QUERY_VALIDATE_REFERENTIAL_INTEGRITY.format( - child_field=child_field, - child_ref="{{ref('"+model_name+"')}}", - parent_field=parent_field, - parent_ref=("{{ref('"+parent_model+"')}}")) +def as_kwarg(key, value): + test_value = to_string(value) + is_function = re.match(r'^\s*(ref|var)\(.+\)$', test_value) is not None i feel ok about it. not in love with regex parsing but what can you do. is the intention to support ref and var in custom schema tests? it doesn't look like those would be passed in here unless i'm missing something In test/integration/008_schema_tests_test/models/schema.yml: > @@ -29,7 +29,7 @@ table_summary: - { field: favorite_color, values: ['blue', 'green'] } relationships: - - { from: favorite_color, to: table_copy, field: favorite_color } + - { from: favorite_color, to: ref('table_copy'), field: favorite_color } ahh i understand now. is this a breaking change in how schema tests are defined? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.