Bug: Error when uploading results to BigQuery, when using vars from CLI.

[bendiktv2]

Hello,

We are getting errors when uploading the invocations.
DBT Version 1.3
dbt_artifacts version 2.2.2
Invoking dbt in the following way: dbt run --select mymodel -- vars '{"my_var": "my value"}' (or similar with dbt test, dbt seed, dbt build)

It produces the following SQL:

insert into `my_project.my_dataset.invocations`
        VALUES
        (
        'feda02be-a309-42d7-873b-29f66f2330cb', 
        '1.3.2', 
        'artifacts_integration_tests', 
        '2023-01-20 12:06:47.124097+00:00', 
        'build', 
        False, 
        'dbt_artifacts', 
        'bigquery', 
        'my_dataset', 
        8, 

        '', 
        '', 
        '', 
        '', 
        '', 
            parse_json('{"TEST_ENV_VAR_1": ""}'), 
            parse_json('{"test_dbt_vars_1": "dbt_vars_1", "test_dbt_vars_2": "dbt_vars_2", "test_dbt_vars_3": "dbt_vars_3"}'), 
        parse_json('{"write_json": true, "use_colors": true, "printer_width": 80, "version_check": true, "partial_parse": true, "static_parser": true, "profiles_dir": "my_path/dbt_artifacts/integration_test_project", "send_anonymous_usage_stats": false, "event_buffer_size": 100000, "quiet": false, "no_print": false, "cache_selected_only": false, "target": "bigquery", "vars": "{\"my_var\": \"my value\"}", "indirect_selection": "eager", "resource_types": [], "which": "build", "rpc_method": "build"}'), 
        parse_json('{}') 
        )

Which, when ran, gives the following error from BigQuery: Invalid input: syntax error while parsing object - invalid literal; last read: '"{"m'; expected '}'; error in PARSE_JSON expression

Which corresponds to the "vars": "{\"my_var\": \"my value\"}", which appears to be a correctly escaped json-string, but BigQuery will not accept it.

---

I propose to add custom handling for BigQuery (and other databases, if the problems exist there too), where we actually parse the invocation_args_dict.vars field, and convert it from a string to a proper python dictionary, before passing it to DBT's tojson.

---

Something along the lines of this in macros/upload_incovations.sql:

        {% if invocation_args_dict.vars %}
            {% set parsed_inv_args_vars = json.loads(invocation_args_dict.vars) %}
            {% set invocation_args_dict.vars = parsed_inv_args_vars %}
        {% endif %}

        parse_json('{{ tojson(invocation_args_dict) }}'), {# invocation_args #}

But the --vars argument accepts any yaml dictionary, so all the following methods of invoking dbt are valid and equivalent:

dbt run --vars '{"key": "value", "date": 20180101}'
dbt run --vars '{key: value, date: 20180101}'
dbt run --vars 'key: value'

[bendiktv2]

Update:

I was a bit hasty on the pseudo-code above, which assumed that the vars was a proper json object, but the dbt docs states that it is a valid yaml object.

The following snippet works for the three --vars scenarios I listed above:
It uses the dbt built-infromyaml function (https://docs.getdbt.com/reference/dbt-jinja-functions/fromyaml), and the same dict.update that is used to set dbt_vars

        {% if invocation_args_dict.vars %}
            {% set parsed_inv_args_vars = fromyaml(invocation_args_dict.vars) %}
            {% do invocation_args_dict.update({'vars': parsed_inv_args_vars}) %}
        {% endif %}