Concurrency Issue

[robinreeves]

Perhaps I am misunderstanding this package's intended use, but I'm getting what appear to be concurrency issues when jobs using the dbt_artifacts package run around the same time. I believe the issue is with running the actual models, but the docs say, "Always run the dbt_artifacts models in every dbt invocation which uses the upload_results macro. This ensures that the source models always have the correct fields in case of an update." If I need to run these models each time, how does the process handle concurrent execution? I'm getting a variety of errors like the ones below. Sometimes objects don't exist, and other times they exist when they shouldn't, so it seems like the processes are all trying to manipulate the same objects in Snowflake.

We currently have this setup with a single warehouse for all our jobs, so there is a lot of potential for things running concurrently using the same artifact objects in Snowflake.

19:50:25 Completed with 3 errors and 0 warnings:
19:50:25
19:50:25 Database Error in model dim_dbt__models (models\dim_dbt__models.sql)
19:50:25 002003 (42S02): SQL compilation error:
19:50:25 Object 'DBT_ARTIFACTS.STAGING.STG_DBT__MODELS' does not exist or not authorized.
19:50:25 compiled Code at target\run\dbt_artifacts\models\dim_dbt__models.sql
19:50:25
19:50:25 Database Error in model dim_dbt__snapshots (models\dim_dbt__snapshots.sql)
19:50:25 002003 (42S02): SQL compilation error:
19:50:25 Object 'DBT_ARTIFACTS.STAGING.STG_DBT__SNAPSHOTS' does not exist or not authorized.
19:50:25 compiled Code at target\run\dbt_artifacts\models\dim_dbt__snapshots.sql
19:50:25
19:50:25 Database Error in model dim_dbt__sources (models\dim_dbt__sources.sql)
19:50:25 002003 (42S02): SQL compilation error:
19:50:25 Object 'DBT_ARTIFACTS.STAGING.STG_DBT__SOURCES' does not exist or not authorized.
19:50:25 compiled Code at target\run\dbt_artifacts\models\dim_dbt__sources.sql

[glsdown]

Hello @robinreeves and apologies for how late this response is.

Thanks for raising the issue. You will see concurrency issues if you are trying to update the table at the same time - in the same way as with any dbt job that you have.

Regarding the note in the README:

Always run the dbt_artifacts models in every dbt invocation which uses the upload_results macro. This ensures that the source models always have the correct fields in case of an update.

This is intended to avoid issues where a new version is released (such as 2.3.0) where we add a new field. If the tables aren't run first, then it will simply error out.

One workaround for you is to make sure that you have fixed your version of dbt artifacts in your packages.yml file using:

packages:
  - package: brooklyn-data/dbt_artifacts
    version: 2.3.0

Then whenever you wish to upgrade the version, make sure to re-run the tables prior to any dbt jobs using the upload_results macro.

dbt run --select dbt_artifacts

This means you don't then need to include the dbt_artifacts run in every invocation.

We have an issue open #294 where we intend to clear this up - so thank you for raising.