Skip to content

Commit

Permalink
[fix](load) fix broker load progress due to retry (#42959)
Browse files Browse the repository at this point in the history
## Proposed changes

Currently, when retrying a broker load, it will use a different
`load_id` with the same `job_id`.
The `total_scan_nums` in progress is accumulated by `job_id`.
This will cause the `total_scan_nums` progress to be multiple of the
actual scan nums.

For example, suppose a 10 instance broker load gets retried:

```
initial (0/10) 0% -> retry (0/20) 0% -> finish (10/20) 50%
```
  • Loading branch information
kaijchen authored and Your Name committed Nov 8, 2024
1 parent 4a28d3e commit c9385b4
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import org.apache.doris.analysis.BrokerDesc;
import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.EnvFactory;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.common.Config;
Expand Down Expand Up @@ -217,6 +218,8 @@ public void updateRetryInfo() {
UUID uuid = UUID.randomUUID();
this.loadId = new TUniqueId(uuid.getMostSignificantBits(), uuid.getLeastSignificantBits());
planner.updateLoadId(this.loadId);
// reset progress on each retry, otherwise the finished/total num will be incorrect
Env.getCurrentProgressManager().registerProgressSimple(String.valueOf(callback.getCallbackId()));
}

void settWorkloadGroups(List<TPipelineWorkloadGroup> tWorkloadGroups) {
Expand Down

0 comments on commit c9385b4

Please sign in to comment.