Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add invocation args column to invocation models #222

Merged
merged 12 commits into from
Oct 17, 2022
Merged
4 changes: 2 additions & 2 deletions .github/workflows/ci_test_package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,13 @@ jobs:
name: Approve Integration Tests

steps:
- name: Checkout branch
- name: Checkout Branch
uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }} # Check out the code of the PR

- name: Install Python packages
run: python -m pip install dbt-snowflake~=1.2.0 sqlfluff-templater-dbt
run: python -m pip install dbt-snowflake~=1.3.0 sqlfluff-templater-dbt

- name: Test database connection
run: dbt debug
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/main_test_package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ jobs:
uses: actions/checkout@v2

- name: Install Python packages
run: python -m pip install dbt-snowflake~=1.2.0 sqlfluff-templater-dbt
run: python -m pip install dbt-snowflake~=1.3.0 sqlfluff-templater-dbt

- name: Test database connection
run: dbt debug
Expand Down
17 changes: 12 additions & 5 deletions macros/upload_invocations.sql
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
nullif({{ adapter.dispatch('column_identifier', 'dbt_artifacts')(14) }}, ''),
nullif({{ adapter.dispatch('column_identifier', 'dbt_artifacts')(15) }}, ''),
{{ adapter.dispatch('parse_json', 'dbt_artifacts')(adapter.dispatch('column_identifier', 'dbt_artifacts')(16)) }},
{{ adapter.dispatch('parse_json', 'dbt_artifacts')(adapter.dispatch('column_identifier', 'dbt_artifacts')(17)) }}
{{ adapter.dispatch('parse_json', 'dbt_artifacts')(adapter.dispatch('column_identifier', 'dbt_artifacts')(17)) }},
{{ adapter.dispatch('parse_json', 'dbt_artifacts')(adapter.dispatch('column_identifier', 'dbt_artifacts')(18)) }}
from values
(
'{{ invocation_id }}', {# command_invocation_id #}
Expand Down Expand Up @@ -56,10 +57,13 @@
{% for dbt_var in var('dbt_vars') %}
{% do dbt_vars_dict.update({dbt_var: var(dbt_var)}) %}
{% endfor %}
'{{ tojson(dbt_vars_dict) }}' {# dbt_vars #}
'{{ tojson(dbt_vars_dict) }}', {# dbt_vars #}
{% else %}
null {# dbt_vars #}
null, {# dbt_vars #}
{% endif %}

'{{ tojson(invocation_args_dict) }}' {# invocation_args #}

)
{% endset %}
{{ invocation_values }}
Expand Down Expand Up @@ -101,10 +105,13 @@
{% for dbt_var in var('dbt_vars') %}
{% do dbt_vars_dict.update({dbt_var: var(dbt_var)}) %}
{% endfor %}
parse_json('{{ tojson(dbt_vars_dict) }}') {# dbt_vars #}
parse_json('{{ tojson(dbt_vars_dict) }}'), {# dbt_vars #}
{% else %}
null {# dbt_vars #}
null, {# dbt_vars #}
{% endif %}

parse_json('{{ tojson(invocation_args_dict) }}') {# invocation_args #}

)
{% endset %}
{{ invocation_values }}
Expand Down
6 changes: 6 additions & 0 deletions models/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,3 +376,9 @@ Total time spent executing the node's last run (seconds).
The meta field of the config associated with the node.

{% enddocs %}

{% docs invocation_args %}

Key-value pairs of args passed to invocation.

{% enddocs %}
3 changes: 2 additions & 1 deletion models/fct_dbt__invocations.sql
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ invocations as (
dbt_cloud_run_reason_category,
dbt_cloud_run_reason,
env_vars,
dbt_vars
dbt_vars,
invocation_args
from base

)
Expand Down
2 changes: 2 additions & 0 deletions models/fct_dbt__invocations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ models:
description: '{{ doc("env_vars") }}'
- name: dbt_vars
description: '{{ doc("dbt_vars") }}'
- name: invocation_args
description: '{{ doc("invocation_args") }}'
3 changes: 2 additions & 1 deletion models/sources/invocations.sql
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ select
cast(null as {{ type_string() }}) as dbt_cloud_run_reason_category,
cast(null as {{ type_string() }}) as dbt_cloud_run_reason,
cast(null as {{ type_json() }}) as env_vars,
cast(null as {{ type_json() }}) as dbt_vars
cast(null as {{ type_json() }}) as dbt_vars,
cast(null as {{ type_json() }}) as invocation_args
from dummy_cte
where 1 = 0
3 changes: 2 additions & 1 deletion models/staging/stg_dbt__invocations.sql
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ enhanced as (
dbt_cloud_run_reason_category,
dbt_cloud_run_reason,
env_vars,
dbt_vars
dbt_vars,
invocation_args
from base

)
Expand Down
2 changes: 2 additions & 0 deletions models/staging/stg_dbt__invocations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ models:
description: '{{ doc("env_vars") }}'
- name: dbt_vars
description: '{{ doc("dbt_vars") }}'
- name: invocation_args
description: '{{ doc("invocation_args") }}'
12 changes: 6 additions & 6 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ rules = L001,L003,L004,L005,L006,L007,L008,L010,L011,L012,L014,L015,L017,L018,L0

deps =
sqlfluff-templater-dbt
dbt-snowflake~=1.2.0
dbt-snowflake~=1.3.0

[sqlfluff:rules:L004]
indent_unit = space
Expand Down Expand Up @@ -112,33 +112,33 @@ deps = {[sqlfluff]deps}
commands = sqlfluff fix models --ignore parsing

[testenv:generate_docs]
deps = dbt-snowflake~=1.2.0
deps = dbt-snowflake~=1.3.0
commands = dbt docs generate --profiles-dir integration_test_project

[testenv:integration_snowflake]
changedir = integration_test_project
deps = dbt-snowflake~=1.2.0
deps = dbt-snowflake~=1.3.0
commands =
dbt deps
dbt build --target snowflake

[testenv:integration_databricks]
changedir = integration_test_project
deps = dbt-databricks~=1.2.0
deps = dbt-databricks~=1.3.0
commands =
dbt deps
dbt build --target databricks

[testenv:integration_bigquery]
changedir = integration_test_project
deps = dbt-bigquery~=1.2.0
deps = dbt-bigquery~=1.3.0
commands =
dbt deps
dbt build --target bigquery

[testenv:integration_spark]
changedir = integration_test_project
deps = dbt-spark[ODBC]~=1.2.0
deps = dbt-spark[ODBC]~=1.3.0
commands =
dbt deps
dbt build --target spark
Expand Down