Replies: 1 comment
-
It turns out that the bigquery method replaces the entire partition and so expects the full partition to be in the temp table, which it isn't for us. Therefore, we need to use standard incremental for bigquery as well! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
The package is soon going to be tested for and then migrated to the new optimised
incremental
strategy. See the utils PR here: snowplow/dbt-snowplow-utils#117In general the change is quite straight forward. For the models:
snowplow_optimise
setting in the config and set it totrue
incremental strategy
(technically not required but adds benefit for bigquery and lets users now override this in a single place in a variable).There are a few open questions around the best way to manage some of this still, which it would be good to get opinions on.
snowplow__incremental_materialization
in the package or remove this entirely and just hard-code the incremental method for those tables?2) While theincremental strategy
does allow us to use the already optimisedinsert_overwrite
for bigquery, it also adds an extra line to every config and an extra variable. In theory this method is the same across all incremental models where it is called, so the user could set it at a models level in their project if they wanted to. If we removed it and they did not set it, they still get the same benefit as our method is close to the way that method uses partitions anyway.snowplow__disable_upsert_lookback
should be listed in the variables in the project or just in our docs site?Beta Was this translation helpful? Give feedback.
All reactions