Return errors properly from RepartitionExec #521
Conversation
| Ok(()) | ||
| }); | ||
|
|
||
| // In a separate task, wait for each input to be done |
There was a problem hiding this comment.
This is the actual code change (to check for return value in another task). Otherwise the rest of this PR is tests
| @@ -308,6 +310,45 @@ impl RepartitionExec { | |||
| send_time_nanos: SQLMetric::time_nanos(), | |||
| }) | |||
| } | |||
|
|
|||
| /// Waits for `input_task` which is consuming one of the inputs to | |||
There was a problem hiding this comment.
I wonder if it might be slightly clearer to push the body of the main task into a fallible function, and to then handle propagating any error it returns within the spawned task? i.e. rather than propagating the error through the JoinHandle, make the task that is spawned onto tokio infallible and handle its errors internally??
Edit: I guess the advantage with this approach would be that you could propagate panics as well...
There was a problem hiding this comment.
I agree the approach you describe would be clearer (and avoid needing a separate task) 👍
The reason I did not pull the main body out into its own function was mostly "trying to keep the diff small" (or perhaps my own laziness wanting to avoid having to figure out all the types of the arguments that got captured),
Perhaps that would be a good follow on PR (there is a lot of messiness / duplication for updating counters which I would also kind of like to fix too)
Codecov Report
@@ Coverage Diff @@
## master #521 +/- ##
==========================================
+ Coverage 76.07% 76.09% +0.01%
==========================================
Files 156 156
Lines 26750 26858 +108
==========================================
+ Hits 20351 20438 +87
- Misses 6399 6420 +21
Continue to review full report at Codecov.
|
alamb
force-pushed
the
alamb/repartition_swallows_errors
branch
from
76a1d96
to
fda9122
Compare
|
Rebased |
houqp
added
bug
Which issue does this PR close?
Closes #437
Rationale for this change
Errors should be returned rather than incorrect results (empty stream). This bug threw me for a loop while debugging something in IOx. See more details on #437
Also, since this operator gets added automatically by the repartition operator, it is likely to affect many/most plans in DataFusion
What changes are included in this PR?
Properly propagate errors from input, and tests for same