-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize filter executor in pull-based executor #4421
Conversation
I think the more elegant way is to directly Skip current iterator to the next iterator. For example: trait A {
fn execute() -> Result<Stream>;
}
Struct B {
input: C,
}
Struct C {
input: D
}
Struct D {
}
// All D, B, C have implemented trait A and trait Stream
// Each execute method will call its input's execute method
fn main {
let b = B::new();
let data_stream = b.execute();
while let Some(stream) = data_stream.next().await {
...
}
}
impl Stream for C {
type Item = ..;
fn poll_next(
mut self: std::pin::Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Self::Item>> {
...
if !predicate(value) {
// Skip current iteration
}
}
} But I don't find a proper way to implement it, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like a reasonable change to me. Thanks @xudong963
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice find! 👍
Thanks for your review! |
Benchmark runs are scheduled for baseline = fdc83e8 and contender = 522a2a4. 522a2a4 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
No
Rationale for this change
If Selection operator doesn't produce any rows, we can just pull next
RecordBatch
es from it.What changes are included in this PR?
Add loop for filter executor.
Are these changes tested?
Covered by existing tests
Are there any user-facing changes?
No