Skip to content

Commit

Permalink
fix: session db/schema and macro references (#25)
Browse files Browse the repository at this point in the history
* Need to set current database and schema for Snowflake to create
tags in the correct place and look for tags in the correct place

* When calling macros from the packaged, they need to be prefixed
with the package name for DBT to find them

* Add note about tag location in readme

Closes issue 24
#24
  • Loading branch information
calleo authored Jan 18, 2023
1 parent 6ff0e8e commit 37f03fc
Show file tree
Hide file tree
Showing 3 changed files with 23 additions and 6 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ target/
dbt_modules/
dbt_packages/
logs/
.idea
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ schema.yml
models:
- name: ACCOUNT
+schema: FINANCE
config:
meta:
database_tags:
Expand All @@ -135,6 +136,9 @@ The above means:
The Snowflake table ACCOUNT will have the tag 'accounting_row_string' set to the value 'visible'.
Its columns ACCOUNT_NAME and ACCOUNT_NUMBER will both have the tag 'accounting_col_string' set to the value 'visible'

All tags are created in the schema of the model where they are added. In the above example the tags will end up
in the FINANCE schema (name depends on how [DBT has been configured](https://docs.getdbt.com/docs/build/custom-schemas)).

The macro must be called as part of on-run-end, so add the following to dbt_project.yml:
```
on-run-end: "{{ snowflake_utils.apply_meta_as_tags(results) }}"
Expand Down
24 changes: 18 additions & 6 deletions macros/apply_meta_as_tags.sql
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,21 @@
#}
{%- set tags_by_schema = {} -%}
{% for res in results -%}
{% if model_contains_tag_meta(res.node) %}
{% if snowflake_utils.model_contains_tag_meta(res.node) %}

{%- set model_database = res.node.database -%}
{%- set model_schema = res.node.schema -%}
{%- set model_schema_full = model_database+'.'+model_schema -%}
{%- set model_alias = res.node.alias -%}

{%- call statement('set_database', fetch_result=True) -%}
USE DATABASE {{model_database}}
{%- endcall -%}

{%- call statement('set_schema', fetch_result=True) -%}
USE SCHEMA {{model_schema}}
{%- endcall -%}

{% if model_schema_full not in tags_by_schema.keys() %}
{{ log('need to fetch tags for schema '+model_schema_full, info=True) }}
{%- call statement('main', fetch_result=True) -%}
Expand All @@ -39,6 +47,8 @@
node: {{ res.node.unique_id }}; status: {{ res.status }} (message: {{ res.message }})
model level database tags: {{ model_meta.database_tags}}
materialized: {{ res.node.config.materialized }}
database: {{ model_database }}
schema: {{ model_schema }}
{%- endset %}
{{ log(line, info=True) }}
{#
Expand All @@ -52,22 +62,24 @@
{{ log(existing_tags_for_table, info=True) }}

{% for table_tag in model_meta.database_tags %}
{{ create_tag_if_missing(current_tags_in_schema,table_tag|upper) }}
{{ snowflake_utils.create_tag_if_missing(current_tags_in_schema,table_tag|upper) }}
{% set desired_tag_value = model_meta.database_tags[table_tag] %}
{{set_table_tag_value_if_different(model_alias|upper,table_tag,desired_tag_value,existing_tags_for_table)}}
{{ snowflake_utils.set_table_tag_value_if_different(model_alias|upper,table_tag,desired_tag_value,existing_tags_for_table) }}
{% endfor %}
{% for column in res.node.columns %}
{% for column_tag in res.node.columns[column].meta.database_tags %}
{{log(column_tag,info=True)}}
{{create_tag_if_missing(current_tags_in_schema,column_tag|upper)}}
{{ snowflake_utils.create_tag_if_missing(current_tags_in_schema,column_tag|upper)}}
{% set desired_tag_value = res.node.columns[column].meta.database_tags[column_tag] %}
{{set_column_tag_value_if_different(model_alias|upper,column|upper,column_tag,desired_tag_value,existing_tags_for_table)}}
{{ snowflake_utils.set_column_tag_value_if_different(model_alias|upper,column|upper,column_tag,desired_tag_value,existing_tags_for_table)}}
{% endfor %}
{% endfor %}
{{ log("========== Finished processing tags for "+model_alias+" ==========", info=True) }}
{% endif %}
{% endfor %}
{% endif %}
-- Need to return something other than None, since DBT will try to execute it as SQL statement
{{ return('') }}
{% endmacro %}

{#
Expand Down Expand Up @@ -165,4 +177,4 @@
{%- endcall -%}
{{ log(load_result('main').data, info=True) }}
{% endif %}
{% endmacro %}
{% endmacro %}

0 comments on commit 37f03fc

Please sign in to comment.