Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug/fix issue with single quotes in env vars #299

Merged
merged 7 commits into from
May 12, 2023
Merged

Bug/fix issue with single quotes in env vars #299

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
406c721
Add additional env vars for testing
Apr 28, 2023
21fc676
Escape single quotes
Apr 28, 2023
1b4f303
Update bigquery macro
Apr 28, 2023
d50a204
Update CI tests
Apr 28, 2023
629566f
Add default values
May 12, 2023
cd75ee2
Pass env variables to tox
May 12, 2023
6f7f0a0
Merge branch 'main' of github.com:brooklyn-data/dbt_artifacts into bu…
May 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci_test_package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ env:
DBT_CLOUD_PROJECT_ID: 123
DBT_CLOUD_JOB_ID: ABC
DBT_CLOUD_RUN_REASON: "String with 'quotes' !"
TEST_ENV_VAR_1: TEST_VALUE
TEST_ENV_VAR_NUMBER: 3
TEST_ENV_VAR_EMPTY: ""
TEST_ENV_VAR_WITH_QUOTE: "Triggered via Apache Airflow by task 'trigger_dbt_cloud_job_run' in the airtable_ingest DAG."
DBT_ENV_CUSTOM_ENV_FAVOURITE_DBT_PACKAGE: dbt_artifacts

jobs:
Expand Down
18 changes: 6 additions & 12 deletions integration_test_project/dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
name: 'artifacts_integration_tests'
version: '1.0'
name: "artifacts_integration_tests"
version: "1.0"
config-version: 2

profile: 'dbt_artifacts'
profile: "dbt_artifacts"

model-paths: ["models"]
analysis-paths: ["analysis"]
Expand All @@ -18,15 +18,9 @@ vars:
test_dbt_vars_1: dbt_vars_1
test_dbt_vars_2: dbt_vars_2
test_dbt_vars_3: dbt_vars_3
env_vars: [
'TEST_ENV_VAR_1'
]
dbt_vars: [
'test_dbt_vars_1',
'test_dbt_vars_2',
'test_dbt_vars_3'
]

env_vars:
["TEST_ENV_VAR_NUMBER", "TEST_ENV_VAR_EMPTY", "TEST_ENV_VAR_WITH_QUOTE"]
dbt_vars: ["test_dbt_vars_1", "test_dbt_vars_2", "test_dbt_vars_3"]

models:
+persist_docs:
Expand Down
4 changes: 3 additions & 1 deletion integration_test_project/example-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,7 @@ export DBT_CLOUD_JOB_ID=
export DBT_CLOUD_RUN_ID=
export DBT_CLOUD_RUN_REASON_CATEGORY=
export DBT_CLOUD_RUN_REASON=
export TEST_ENV_VAR_1=
export TEST_ENV_VAR_NUMBER=3
export TEST_ENV_VAR_EMPTY=
export DBT_ENV_CUSTOM_ENV_FAVOURITE_DBT_PACKAGE=dbt_artifacts
export TEST_ENV_VAR_WITH_QUOTE="Triggered via Apache Airflow by task 'trigger_dbt_cloud_job_run' in the airtable_ingest DAG."
21 changes: 15 additions & 6 deletions macros/upload_invocations.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
{% if var('env_vars', none) %}
{% set env_vars_dict = {} %}
{% for env_variable in var('env_vars') %}
{% do env_vars_dict.update({env_variable: env_var(env_variable)}) %}
{% do env_vars_dict.update({env_variable: (env_var(env_variable, '') | replace("'", "''"))}) %}
{% endfor %}
'{{ tojson(env_vars_dict) }}', {# env_vars #}
{% else %}
Expand All @@ -56,15 +56,20 @@
{% if var('dbt_vars', none) %}
{% set dbt_vars_dict = {} %}
{% for dbt_var in var('dbt_vars') %}
{% do dbt_vars_dict.update({dbt_var: var(dbt_var)}) %}
{% do dbt_vars_dict.update({dbt_var: (var(dbt_var, '') | replace("'", "''"))}) %}
{% endfor %}
'{{ tojson(dbt_vars_dict) }}', {# dbt_vars #}
{% else %}
null, {# dbt_vars #}
{% endif %}

'{{ tojson(invocation_args_dict) | replace('\\', '\\\\') }}', {# invocation_args #}
'{{ tojson(dbt_metadata_envs) | replace("'", "\\'") }}' {# dbt_custom_envs #}

{% set metadata_env = {} %}
{% for key, value in dbt_metadata_envs.items() %}
{% do metadata_env.update({key: (value | replace("'", "''"))}) %}
{% endfor %}
'{{ tojson(metadata_env) | replace('\\', '\\\\') }}' {# dbt_custom_envs #}

)
{% endset %}
Expand Down Expand Up @@ -95,7 +100,7 @@
{% if var('env_vars', none) %}
{% set env_vars_dict = {} %}
{% for env_variable in var('env_vars') %}
{% do env_vars_dict.update({env_variable: env_var(env_variable)}) %}
{% do env_vars_dict.update({env_variable: (env_var(env_variable, '') | replace("'", "''"))}) %}
{% endfor %}
parse_json('{{ tojson(env_vars_dict) }}'), {# env_vars #}
{% else %}
Expand All @@ -105,7 +110,7 @@
{% if var('dbt_vars', none) %}
{% set dbt_vars_dict = {} %}
{% for dbt_var in var('dbt_vars') %}
{% do dbt_vars_dict.update({dbt_var: var(dbt_var)}) %}
{% do dbt_vars_dict.update({dbt_var: (var(dbt_var, '') | replace("'", "''"))}) %}
{% endfor %}
parse_json('{{ tojson(dbt_vars_dict) }}'), {# dbt_vars #}
{% else %}
Expand All @@ -119,7 +124,11 @@
{% endif %}
{# invocation_args_dict.vars, in the absence of any vars, results in the value "{}\n" as a string which results in an error. safe.parse_json accomodates for this gracefully. #}
safe.parse_json('{{ tojson(invocation_args_dict) }}'), {# invocation_args #}
parse_json('{{ tojson(dbt_metadata_envs) }}') {# dbt_custom_envs #}
{% set metadata_env = {} %}
{% for key, value in dbt_metadata_envs.items() %}
{% do metadata_env.update({key: (value | replace("'", "''"))}) %}
{% endfor %}
parse_json('{{ tojson(metadata_env) | replace('\\', '\\\\') }}') {# dbt_custom_envs #}

)
{% endset %}
Expand Down
4 changes: 3 additions & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,9 @@ passenv =
DBT_CLOUD_RUN_ID
DBT_CLOUD_RUN_REASON_CATEGORY
DBT_CLOUD_RUN_REASON
TEST_ENV_VAR_1
TEST_ENV_VAR_NUMBER
TEST_ENV_VAR_EMPTY
TEST_ENV_VAR_WITH_QUOTE

[testenv:lint]
deps = {[sqlfluff]deps}
Expand Down