Skip to content

Commit

Permalink
3. Append-only insert if no partitions
Browse files Browse the repository at this point in the history
  • Loading branch information
jtcohen6 committed Jan 12, 2021
1 parent 25bae44 commit ecafb59
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 2 deletions.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,8 @@ The following configurations can be supplied to models run with the dbt-spark pl
**Incremental Models**

To use incremental models, specify a `partition_by` clause in your model config. The default incremental strategy used is `insert_overwrite`, which will overwrite the partitions included in your query. Be sure to re-select _all_ of the relevant
data for a partition when using the `insert_overwrite` strategy. If a `partition_by` config is not specified, dbt will overwrite the entire table as an atomic operation, replacing it with new data of the same schema. This is analogous to `truncate` + `insert`.
data for a partition when using the `insert_overwrite` strategy. If a `partition_by` config is not specified, dbt will simply
append new data to the model, without overwriting any existing data.

```
{{ config(
Expand Down
5 changes: 4 additions & 1 deletion dbt/include/spark/macros/materializations/incremental.sql
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
{% macro get_insert_overwrite_sql(source_relation, target_relation) %}

{%- set cols = config.get('partition_by', validator=validation.any[list, basestring]) -%}
{%- set insert = 'insert overwrite' if cols is not none else 'insert into' -%}

{%- set dest_columns = adapter.get_columns_in_relation(target_relation) -%}
{%- set dest_cols_csv = dest_columns | map(attribute='quoted') | join(', ') -%}
insert overwrite table {{ target_relation }}
{{ insert }} table {{ target_relation }}
{{ partition_cols(label="partition") }}
select {{dest_cols_csv}} from {{ source_relation.include(database=false, schema=false) }}

Expand Down

0 comments on commit ecafb59

Please sign in to comment.