Skip to content

Commit

Permalink
Only pass batches to retry for microbatch model when there was a Part…
Browse files Browse the repository at this point in the history
…ialSuccess

In the previous commit we made it so that retries of microbatch models wouldn't
run in full refresh mode when the microbatch model to retry has batches already
specified from the prior run. This is only problematic when the run being retried
was a full refresh AND all the batches for a given microbatch model failed. In
that case WE DO want to do a full refresh for the given microbatch model. To better
outline the problem, consider the following:

* a microbatch model had a begin of `2020-01-01` and has been running this way for awhile
* the begin config has changed to `2024-01-01` and  dbt run --full-refresh gets run
* every batch for an microbatch model fails
* on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged

To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch
models. Note: microbatch models only have a partially successful status IFF they have both
successful and failed batches.
  • Loading branch information
QMalcolm committed Sep 25, 2024
1 parent 14e8d53 commit a6a1ef8
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion core/dbt/task/retry.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,9 @@ def run(self):
batch_map = {
result.unique_id: result.batch_results.failed
for result in self.previous_results.results
if result.batch_results is not None and len(result.batch_results.failed) > 0
if result.status == NodeStatus.PartialSuccess
and result.batch_results is not None
and len(result.batch_results.failed) > 0
}

class TaskWrapper(self.task_class):
Expand Down

0 comments on commit a6a1ef8

Please sign in to comment.