Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework sources to be dbt models rather than manually created #188

Merged
merged 19 commits into from
Sep 13, 2022
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/main_test_package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ jobs:
uses: actions/checkout@v2

- name: Install Python packages
run: python -m pip install dbt-snowflake~=1.1.0 sqlfluff-templater-dbt
run: python -m pip install dbt-snowflake~=1.2.0 sqlfluff-templater-dbt

- name: Test database connection
run: dbt debug
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ Pipfile.lock
env.sh

.python_version
.vscode
41 changes: 32 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,27 +36,28 @@ packages:
3. Add an on-run-end hook to your `dbt_project.yml`: `on-run-end: "{{ dbt_artifacts.upload_results(results) }}"`
(We recommend adding a conditional here so that the upload only occurs in your production environment, such as `on-run-end: "{% if target.name == 'prod' %}{{ dbt_artifacts.upload_results(results) }}{% endif %}"`)

4. Create the tables dbt_artifacts uploads to with `dbt run-operation create_dbt_artifacts_tables`
4. If you are using [selectors](https://docs.getdbt.com/reference/node-selection/syntax), be sure to include the `dbt_artifacts` models in your dbt invocation step.

5. Run your project!

> :construction_worker: Always run the dbt_artifacts models in every dbt invocation which uses the `upload_results` macro. This ensures that the source models always have the correct fields in case of an update.

## Configuration

The following configuration can be used to specify where the raw data is uploaded, and where the dbt models are created:
The following configuration can be used to specify where the raw (sources) data is uploaded, and where the dbt models are created:

```yml
vars:
dbt_artifacts_database: your_db # optional, default is your target database
dbt_artifacts_schema: your_schema # optional, default is your target schema
dbt_artifacts_create_schema: true|false # optional, set to false if you don't have privileges to create schema, default is true

models:
...
dbt_artifacts:
+schema: your_destination_schema # optional, default is your target database
+database: your_destination_database # optional, default is your target database
+schema: your_destination_schema # optional, default is your target schema
staging:
+database: your_destination_database # optional, default is your target database
+schema: your_destination_schema # optional, default is your target schema
...
sources:
+database: your_sources_database # optional, default is your target database
+schema: your sources_database # optional, default is your target schema
```

Note that the model materializations are defined in this package's `dbt_project.yml`, so do not set them in your project.
Expand Down Expand Up @@ -92,6 +93,28 @@ vars:
]
```

## Upgrading from 1.x to >=2.0.0
If you were using the following variables:

```yml
vars:
dbt_artifacts_database: your_db
dbt_artifacts_schema: your_schema
```

You must now move these to the following model configs:

```yml
models:
...
dbt_artifacts:
sources:
+database: your_db
+schema: your your_schema
NiallRees marked this conversation as resolved.
Show resolved Hide resolved
```

That's because the raw tables are now managed as dbt models.
NiallRees marked this conversation as resolved.
Show resolved Hide resolved

## Migrating From <1.0.0 to >=1.0.0
To migrate your existing data from the `dbt-artifacts` versions <=0.8.0, a helper macro and guide is provided. This migration uses the old `fct_*` and `dim_*` models' data to populate the new sources. The steps to use the macro are as follows:

Expand Down
5 changes: 4 additions & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: 'dbt_artifacts'
version: '1.2.0'
config-version: 2
require-dbt-version: ">=1.0.0"
require-dbt-version: ">=1.2.0"
jaypeedevlin marked this conversation as resolved.
Show resolved Hide resolved
profile: "dbt_artifacts"

clean-targets: # folders to be removed by `dbt clean`
Expand All @@ -13,3 +13,6 @@ models:
dbt_artifacts:
+materialized: view
+file_format: delta
sources:
+materialized: incremental
+on_schema_change: append_new_columns
45 changes: 0 additions & 45 deletions macros/create_dbt_artifacts_tables.sql

This file was deleted.

86 changes: 0 additions & 86 deletions macros/create_exposures_table_if_not_exists.sql

This file was deleted.

106 changes: 0 additions & 106 deletions macros/create_invocations_table_if_not_exists.sql

This file was deleted.

Loading