Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change microbatch lookback default from 0 to 1 #10876

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

QMalcolm
Copy link
Contributor

Resolves #10867

Problem

A lookback value of 0 results in microbatch models consistently having incomplete data.

Solution

Change the lookback default to 1. This guarantees that batches will eventually get completed (assuming the job is run once per batch_size)

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
  • This PR includes type annotations for new and modified functions.

@QMalcolm QMalcolm added the artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking label Oct 17, 2024
@QMalcolm QMalcolm requested review from a team as code owners October 17, 2024 20:52
@QMalcolm QMalcolm requested review from PaulVPham and removed request for a team October 17, 2024 20:52
@cla-bot cla-bot bot added the cla:yes label Oct 17, 2024
Copy link

codecov bot commented Oct 17, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.15%. Comparing base (8be0635) to head (36943f0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10876      +/-   ##
==========================================
- Coverage   89.18%   89.15%   -0.03%     
==========================================
  Files         183      183              
  Lines       23438    23438              
==========================================
- Hits        20903    20896       -7     
- Misses       2535     2542       +7     
Flag Coverage Δ
integration 86.43% <100.00%> (-0.03%) ⬇️
unit 62.13% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Unit Tests 62.13% <100.00%> (ø)
Integration Tests 86.43% <100.00%> (-0.03%) ⬇️

@QMalcolm
Copy link
Contributor Author

Changes to schemas.getdbt.com dbt-labs/schemas.getdbt.com#68

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking cla:yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Defaulting lookback to 0 results in consistently incomplete batches
1 participant