fivetran-data-model-bot
released this
17 Jul 19:51
·
2 commits
to main
since this release
PR #4 includes the following updates. Please be aware these changes only impact Databricks warehouse users:
🚨 Breaking Changes 🚨
⚠️ Since the following changes result in the table format changing for Databricks users, we recommend running a--full-refresh
after upgrading to this version to avoid possible incremental failures.
- For Databricks All-Purpose clusters, the
salesforce_marketing_cloud__events_enhanced
model will now be materialized using the delta table format (previously parquet).- Delta tables are generally more performant than parquet and are also more widely available for Databricks users. Previously, the parquet file format was causing compilation issues on customers' managed tables.
Documentation
- Added details to the README to highlight the incremental strategies used within the
salesforce_marketing_cloud__events_enhanced
model.
Under the Hood
- The
is_incremental_compatible
macro has been added to the package. This macro will returntrue
if the Databricks runtime being used is an all-purpose cluster or if any other non-Databricks supported destination is being used.- This update was applied as there are other Databricks runtimes (ie. sql warehouse, endpoint, and external runtime) which do not support the
insert_overwrite
incremental strategy used in thesalesforce_marketing_cloud__events_enhanced
model.
- This update was applied as there are other Databricks runtimes (ie. sql warehouse, endpoint, and external runtime) which do not support the
- In addition to the above, for Databricks users the
salesforce_marketing_cloud__events_enhanced
model will now leverage the incremental strategy only if the Databricks runtime is all-purpose. Otherwise, all other Databricks runtimes will not leverage an incremental strategy. - Added validation tests to the
integration_tests
folder to ensure the consistency and integrity of thesalesforce_marketing_cloud__events_enhanced
model for subsequent updates.
Full Changelog: v0.1.0...v0.2.0