Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Microbatch] Leverage static partitioning in dbt-bigquery materialization strategy #1355

Open
Tracked by #10624
MichelleArk opened this issue Sep 26, 2024 · 0 comments
Assignees
Labels
microbatch Issues related to the microbatch incremental strategy

Comments

@MichelleArk
Copy link
Contributor

In #1334, we introduced the microbatch strategy which leverages the dynamic partition insert/overwrite mechanism.

However, static partitioning would be significantly more performant. Benchmarking here:
https://discourse.getdbt.com/t/benchmarking-incremental-strategies-on-bigquery/981

Let's:

  • Leverage static partitioning by setting partitions prior to calling the insert_overwrite_sql as part of the microbatch strategy
  • Build partitions using GENERATE_TIMESTAMP_ARRAY (for 'hour' batch_size) or GENERATE_DATE_ARRAY (for all others) + __dbt_internal_microbatch_event_time_start and __dbt_internal_microbatch_event_time_end jinja context vars.
@MichelleArk MichelleArk self-assigned this Sep 26, 2024
@MichelleArk MichelleArk added the microbatch Issues related to the microbatch incremental strategy label Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
microbatch Issues related to the microbatch incremental strategy
Projects
None yet
Development

No branches or pull requests

1 participant