Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding multiple unique keys #6438

Merged
merged 19 commits into from
Nov 14, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion website/docs/docs/build/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@
| [schema](/reference/resource-configs/schema) | Specify a custom schema for the snapshot | No | snapshots |
| [alias](/reference/resource-configs/alias) | Specify an alias for the snapshot | No | your_custom_snapshot |
| [strategy](/reference/resource-configs/strategy) | The snapshot strategy to use. Valid values: `timestamp` or `check` | Yes | timestamp |
| [unique_key](/reference/resource-configs/unique_key) | A <Term id="primary-key" /> column or expression for the record | Yes | id |
| [unique_key](/reference/resource-configs/unique_key) | A <Term id="primary-key" /> column(s) (string or array) or expression for the record | Yes | `id` or `[order_id, product_id]` |

Check warning on line 86 in website/docs/docs/build/snapshots.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/docs/build/snapshots.md#L86

[custom.Typos] Oops there's a typo -- did you really mean 'unique_key'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'unique_key'? ", "location": {"path": "website/docs/docs/build/snapshots.md", "range": {"start": {"line": 86, "column": 4}}}, "severity": "WARNING"}
runleonarun marked this conversation as resolved.
Show resolved Hide resolved
| [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] |
| [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at |
| [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True |
Expand Down
52 changes: 21 additions & 31 deletions website/docs/reference/resource-configs/unique_key.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
resource_types: [snapshots]
description: "Unique_key - Read this in-depth guide to learn about configurations in dbt."
description: "Learn more about unique_key configurations in dbt."
datatype: column_name_or_expression
---

Expand All @@ -14,7 +14,7 @@
- name: orders_snapshot
relation: source('my_source', 'my_table')
[config](/reference/snapshot-configs):
unique_key: id
unique_key: order_id

```

Expand Down Expand Up @@ -50,21 +50,23 @@
</File>

## Description

A column name or expression that is unique for the inputs of a snapshot. dbt uses this to match records between a result set and an existing snapshot, so that changes can be captured correctly.

In Versionless and dbt v1.9 and later, [snapshots](/docs/build/snapshots) are defined and configured in YAML files within your `snapshots/` directory. The `unique_key` is specified within the `config` block of your snapshot YAML file.
In Versionless and dbt v1.9 and later, [snapshots](/docs/build/snapshots) are defined and configured in YAML files within your `snapshots/` directory. You can specify one or multiple `unique_key`s within the `config` block of your snapshot YAML file.

Check warning on line 56 in website/docs/reference/resource-configs/unique_key.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/resource-configs/unique_key.md#L56

[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'? ", "location": {"path": "website/docs/reference/resource-configs/unique_key.md", "range": {"start": {"line": 56, "column": 24}}}, "severity": "WARNING"}
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

:::caution
:::caution

Providing a non-unique key will result in unexpected snapshot results. dbt **will not** test the uniqueness of this key, consider [testing](/blog/primary-key-testing#how-to-test-primary-keys-with-dbt) the source data to ensure that this key is indeed unique.

:::

## Default
This is a **required parameter**. No default is provided.

This is a **required parameter**. No default is provided.

## Examples

### Use an `id` column as a unique key

<VersionBlock firstVersion="1.9">
Expand Down Expand Up @@ -114,29 +116,36 @@

</File>

### Use a combination of two columns as a unique key
This configuration accepts a valid column expression. As such, you can concatenate two columns together as a unique key if required. It's a good idea to use a separator (e.g. `'-'`) to ensure uniqueness.

<VersionBlock firstVersion="1.9">

### Use multiple unique keys

You can configure snapshots to use multiple unique keys for primary_key columns.

Check warning on line 123 in website/docs/reference/resource-configs/unique_key.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/resource-configs/unique_key.md#L123

[custom.Typos] Oops there's a typo -- did you really mean 'primary_key'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'primary_key'? ", "location": {"path": "website/docs/reference/resource-configs/unique_key.md", "range": {"start": {"line": 123, "column": 61}}}, "severity": "WARNING"}
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

<File name='snapshots/transaction_items_snapshot.yml'>

```yaml
snapshots:
- name: transaction_items_snapshot
relation: source('erp', 'transactions')
- name: orders_snapshot
relation: source('jaffle_shop', 'orders')
config:
schema: snapshots
unique_key: "transaction_id || '-' || line_item_id"
unique_key:
- order_id
- product_id
strategy: timestamp
updated_at: updated_at

```

</File>
</VersionBlock>

<VersionBlock lastVersion="1.8">

### Use a combination of two columns as a unique key

This configuration accepts a valid column expression. As such, you can concatenate two columns together as a unique key if required. It's a good idea to use a separator (e.g. `'-'`) to ensure uniqueness.

Check warning on line 147 in website/docs/reference/resource-configs/unique_key.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/resource-configs/unique_key.md#L147

[custom.LatinAbbreviations] Avoid Latin abbreviations: 'that is'. Consider using 'idea' instead.
Raw output
{"message": "[custom.LatinAbbreviations] Avoid Latin abbreviations: 'that is'. Consider using 'idea' instead.", "location": {"path": "website/docs/reference/resource-configs/unique_key.md", "range": {"start": {"line": 147, "column": 146}}}, "severity": "WARNING"}

Check warning on line 147 in website/docs/reference/resource-configs/unique_key.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/resource-configs/unique_key.md#L147

[custom.LatinAbbreviations] Avoid Latin abbreviations: 'for example'. Consider using 'e.g' instead.
Raw output
{"message": "[custom.LatinAbbreviations] Avoid Latin abbreviations: 'for example'. Consider using 'e.g' instead.", "location": {"path": "website/docs/reference/resource-configs/unique_key.md", "range": {"start": {"line": 147, "column": 171}}}, "severity": "WARNING"}

Check warning on line 147 in website/docs/reference/resource-configs/unique_key.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/resource-configs/unique_key.md#L147

[custom.Typos] Oops there's a typo -- did you really mean 'e.g.'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'e.g.'? ", "location": {"path": "website/docs/reference/resource-configs/unique_key.md", "range": {"start": {"line": 147, "column": 171}}}, "severity": "WARNING"}
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<File name='snapshots/transaction_items_snapshot.sql'>

```jinja2
Expand All @@ -159,25 +168,9 @@
```

</File>
</VersionBlock>

Though, it's probably a better idea to construct this column in your query and use that as the `unique_key`:

<VersionBlock firstVersion="1.9">

<File name='snapshots/transaction_items_snapshot.yml'>

```yaml
snapshots:
- name: transaction_items_snapshot
relation: {{ ref('transaction_items_ephemeral') }}
config:
schema: snapshots
unique_key: id
strategy: timestamp
updated_at: updated_at
```
</File>

<File name='models/transaction_items_ephemeral.sql'>

Expand All @@ -195,9 +188,6 @@

In this example, we create an ephemeral model `transaction_items_ephemeral` that creates an `id` column that can be used as the `unique_key` our snapshot configuration.

</VersionBlock>

<VersionBlock lastVersion="1.8">
<File name='snapshots/transaction_items_snapshot.sql'>

```jinja2
Expand Down
Loading