[merge after dec 5th] adds new hard_deletes config (#6558)

website/docs/docs/build/snapshots.md

@@ -10,8 +10,7 @@ id: "snapshots"
 * [Snapshot properties](/reference/snapshot-properties)
 * [`snapshot` command](/reference/commands/snapshot)
 
-
-### What are snapshots?
+## What are snapshots?
 Analysts often need to "look back in time" at previous data states in their mutable tables. While some source data systems are built in a way that makes accessing historical data possible, this is not always the case. dbt provides a mechanism, **snapshots**, which records changes to a mutable <Term id="table" /> over time.
 
 Snapshots implement [type-2 Slowly Changing Dimensions](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) over mutable source tables. These Slowly Changing Dimensions (or SCDs) identify how a row in a table changes over time. Imagine you have an `orders` table where the `status` field can be overwritten as the order is processed.
@@ -64,9 +63,9 @@ snapshots:
       [unique_key](/reference/resource-configs/unique_key): column_name_or_expression
       [check_cols](/reference/resource-configs/check_cols): [column_name] | all
       [updated_at](/reference/resource-configs/updated_at): column_name
-      [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes): true | false
       [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): dictionary
       [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): string
+      [hard_deletes](/reference/resource-configs/hard-deletes): ignore | invalidate | new_record 
 ```
 
 </File>
@@ -82,9 +81,9 @@ The following table outlines the configurations available for snapshots:
 | [unique_key](/reference/resource-configs/unique_key) | A <Term id="primary-key" /> column(s) (string or array) or expression for the record | Yes |  `id` or `[order_id, product_id]` |
 | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] |
 | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at |
-| [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True |
 | [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table.| No | string |
 | [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary |
+| [hard_deletes](/reference/resource-configs/hard-deletes) | Specify how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`.| No | string |
 
 
 - In versions prior to v1.9, the `target_schema` (required) and `target_database` (optional) configurations defined a single schema or database to build a snapshot across users and environment. This created problems when testing or developing a snapshot, as there was no clear separation between development and production environments.  In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. Developers can still set a custom location with [`schema`](/reference/resource-configs/schema) and [`database`](/reference/resource-configs/database)  configs, consistent with other resource types.
@@ -216,10 +215,14 @@ When you run the [`dbt snapshot` command](/reference/commands/snapshot):
   - The `dbt_valid_to` column will be updated for any existing records that have changed.
   - The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` or the value configured in `dbt_valid_to_current` (available in dbt Core v1.9+).
 
+<VersionBlock firstVersion="1.9">
+
 #### Note 
 - These column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config.
 - Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table.
-
+- Use the [`hard_deletes`](/reference/resource-configs/hard-deletes) config to track hard deletes by adding a new record when row become "deleted" in source. Supported options are `ignore`, `invalidate`, and `new_record`.
+</VersionBlock>
+
 Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](/reference/dbt-jinja-functions/ref) function.
 
 ## Detecting row changes
@@ -295,7 +298,7 @@ The `check` snapshot strategy can be configured to track changes to _all_ column
 
 :::
 
-**Example Usage**
+**Example usage**
 
 <VersionBlock lastVersion="1.8">
 
@@ -345,15 +348,64 @@ snapshots:
 
 ### Hard deletes (opt-in)
 
+<VersionBlock firstVersion="1.9">
+
+In dbt v1.9 and higher, the [`hard_deletes`](/reference/resource-configs/hard-deletes) config replaces the `invalidate_hard_deletes` config to give you more control on how to handle deleted rows from the source. The `hard_deletes` config is not a separate strategy but an additional opt-in feature that can be used with any snapshot strategy.
+
+The `hard_deletes` config has three options/fields:
+| Field | Description |
+| --------- | ----------- |
+| `ignore` (default) | No action for deleted records. |
+| `invalidate` | Behaves the same as the existing `invalidate_hard_deletes=true`, where deleted records are invalidated by setting `dbt_valid_to`. |
+| `new_record` | Tracks deleted records as new rows using the `dbt_is_deleted` [meta field](#snapshot-meta-fields) when records are deleted.|
+
+import HardDeletes from '/snippets/_hard-deletes.md';
+
+<HardDeletes />
+
+#### Example usage
+
+<File name='snapshots/orders_snapshot.yml'>
+
+```yaml
+snapshots:
+  - name: orders_snapshot_hard_delete
+    relation: source('jaffle_shop', 'orders')
+    config:
+      schema: snapshots
+      unique_key: id
+      strategy: timestamp
+      updated_at: updated_at
+      hard_deletes: new_record  # options are: 'ignore', 'invalidate', or 'new_record'
+```
+
+</File>
+
+In this example, the `hard_deletes: new_record` config will add a new row for deleted records with the `dbt_is_deleted` column set to `True`.
+Any restored records are added as new rows with the `dbt_is_deleted` field set to `False`.
+
+The resulting table will look like this:
+
+| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_is_deleted |
+| -- | ------ | ---------- | -------------- | ------------ | -------------- |
+| 1  | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | False          |
+| 1  | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | False          |
+| 1  | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | 2024-01-01 12:00 | True           |
+| 1  | restored | 2024-01-01 12:00 | 2024-01-01 12:00 |                 | False        |
+
+</VersionBlock>
+
+<VersionBlock lastVersion="1.8">
+
 Rows that are deleted from the source query are not invalidated by default. With the config option `invalidate_hard_deletes`, dbt can track rows that no longer exist. This is done by left joining the snapshot table with the source table, and filtering the rows that are still valid at that point, but no longer can be found in the source table. `dbt_valid_to` will be set to the current snapshot time.
 
 This configuration is not a different strategy as described above, but is an additional opt-in feature. It is not enabled by default since it alters the previous behavior.
 
 For this configuration to work with the `timestamp` strategy, the configured `updated_at` column must be of timestamp type. Otherwise, queries will fail due to mixing data types.
 
-**Example Usage**
+Note, in v1.9 and higher, the [`hard_deletes`](/reference/resource-configs/hard-deletes) config replaces the `invalidate_hard_deletes` config for better control over how to handle deleted rows from the source.
 
-<VersionBlock lastVersion="1.8">
+#### Example usage
 
 <File name='snapshots/orders_snapshot_hard_delete.sql'>
 
@@ -379,38 +431,24 @@ For this configuration to work with the `timestamp` strategy, the configured `up
 
 </VersionBlock>
 
-<VersionBlock firstVersion="1.9">
-
-<File name='snapshots/orders_snapshot.yml'>
-
-```yaml
-snapshots:
-  - name: orders_snapshot_hard_delete
-    relation: source('jaffle_shop', 'orders')
-    config:
-      schema: snapshots
-      unique_key: id
-      strategy: timestamp
-      updated_at: updated_at
-      invalidate_hard_deletes: true
-```
-
-</File>
-
-</VersionBlock>
-
 ## Snapshot meta-fields
 
 Snapshot <Term id="table">tables</Term> will be created as a clone of your source dataset, plus some additional meta-fields*.
 
-In dbt Core v1.9+ (or available sooner in [the "Latest" release track in dbt Cloud](/docs/dbt-versions/cloud-release-tracks)), these column names can be customized to your team or organizational conventions via the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config.
+In dbt Core v1.9+ (or available sooner in [the "Latest" release track in dbt Cloud](/docs/dbt-versions/cloud-release-tracks)):
+- These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config.
+ess)
+- Use the [`dbt_valid_to_current` config](/reference/resource-configs/dbt_valid_to_current) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table.
+- Use the [`hard_deletes`](/reference/resource-configs/hard-deletes) config to track deleted records as new rows with the `dbt_is_deleted` meta field when using the `hard_deletes='new_record'` field.
+
 
 | Field          | Meaning | Usage |
 | -------------- | ------- | ----- |
 | dbt_valid_from | The timestamp when this snapshot row was first inserted | This column can be used to order the different "versions" of a record. |
 | dbt_valid_to   | The timestamp when this row became invalidated. <br /> For current records, this is `NULL` by default <VersionBlock firstVersion="1.9"> or the value specified in `dbt_valid_to_current`.</VersionBlock> | The most recent snapshot record will have `dbt_valid_to` set to `NULL` <VersionBlock firstVersion="1.9"> or the specified value. </VersionBlock> |
 | dbt_scd_id     | A unique key generated for each snapshotted record. | This is used internally by dbt |
 | dbt_updated_at | The updated_at timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt |
+| dbt_is_deleted | A boolean value indicating if the record has been deleted. `True` if deleted, `False` otherwise. | Added when `hard_deletes='new_record'` is configured. This is used internally by dbt |
 
 *The timestamps used for each column are subtly different depending on the strategy you use:
 
@@ -444,6 +482,15 @@ Snapshot results (note that `11:30` is not used anywhere):
 | 1  | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 |
 | 1  | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 |                  | 2024-01-01 11:05 |
 
+Snapshot results with `hard_deletes='new_record'`:
+
+| id | status  | updated_at       | dbt_valid_from   | dbt_valid_to     | dbt_updated_at   | dbt_is_deleted |
+|----|---------|------------------|------------------|------------------|------------------|----------------|
+| 1  | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | False          |
+| 1  | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | 2024-01-01 11:05 | False          |
+| 1  | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 |                  | 2024-01-01 11:20 | True           |
+
+
 </details>
 
 <br/>
@@ -478,6 +525,14 @@ Snapshot results:
 | 1   | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 |
 | 1   | shipped | 2024-01-01 11:30 |                  | 2024-01-01 11:30 |
 
+Snapshot results with `hard_deletes='new_record'`:
+
+| id | status  |  dbt_valid_from   | dbt_valid_to     | dbt_updated_at   | dbt_is_deleted |
+|----|---------|------------------|------------------|------------------|----------------|
+| 1  | pending |  2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | False          |
+| 1  | shipped | 2024-01-01 11:30 | 2024-01-01 11:40 | 2024-01-01 11:30 | False          |
+| 1  | deleted |  2024-01-01 11:40 |                  | 2024-01-01 11:40 | True           |
+
 </details>
 
 ## Configure snapshots in versions 1.8 and earlier
@@ -486,7 +541,7 @@ Snapshot results:
 
 For information about configuring snapshots in dbt versions 1.8 and earlier, select **1.8** from the documentation version picker, and it will appear in this section.
 
-To configure snapshots in versions 1.9 and later, refer to [Configuring snapshots](#configuring-snapshots). The latest versions use a more ergonomic snapshot configuration syntax that also speeds up parsing and compilation.
+To configure snapshots in versions 1.9 and later, refer to [Configuring snapshots](#configuring-snapshots). The latest versions use an updated snapshot configuration syntax that optimizes performance.
 
 </VersionBlock>
 

website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md

@@ -68,6 +68,7 @@ Beginning in dbt Core 1.9, we've streamlined snapshot configuration and added a
 - Standard `schema` and `database` configs supported: Snapshots will now be consistent with other dbt resource types. You can specify where environment-aware snapshots should be stored.
 - Warning for incorrect `updated_at` data type: To ensure data integrity, you'll see a warning if the `updated_at` field specified in the snapshot configuration is not the proper data type or timestamp.
 - Set a custom current indicator for the value of `dbt_valid_to`: Use the [`dbt_valid_to_current` config](/reference/resource-configs/dbt_valid_to_current) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. 
+- Use the [`hard_deletes`](/reference/resource-configs/hard-deletes) configuration to get more control on how to handle deleted rows from the source. Supported methods are `ignore` (default), `invalidate` (replaces legacy `invalidate_hard_deletes=true`), and `new_record`. Setting  `hard_deletes='new_record'` allows you to track hard deletes by adding a new record when row becomes "deleted" in source. 
 
 Read more about [Snapshots meta fields](/docs/build/snapshots#snapshot-meta-fields).
 

website/docs/docs/dbt-versions/release-notes.md

@@ -18,6 +18,10 @@ Release notes are grouped by month for both multi-tenant and virtual private clo
 
 \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability.
 
+## December 2024
+
+- **New**: The [`hard_deletes`](/reference/resource-configs/hard-deletes) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table.
+
 ## November 2024
 - **Enhancement**: Trust signal icons in dbt Explorer are now available for Exposures, providing a quick view of data health while browsing resources. To view trust signal icons, go to dbt Explorer and click **Exposures** under the **Resource** tab. Refer to [Trust signal for resources](/docs/collaborate/explore-projects#trust-signals-for-resources) for more info.
 - **Bug**: Identified and fixed an error with Semantic Layer queries that take longer than 10 minutes to complete.