Remove qualifiers on pushed down predicates / Fix parquet pruning #689
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
force-pushed
the
alamb/fix_parquet_pruning
branch
from
eb980a5
to
432ce23
Compare
alamb
commented
Jul 6, 2021
| // Remove all qualifiers from the scan as the provider | ||
| // doesn't know (nor should care) how the relation was | ||
| // referred to in the query | ||
| let filters = unnormalize_cols(filters.iter().cloned()); |
There was a problem hiding this comment.
Here is the actual fix for parquet pruning.
| @@ -1762,4 +1794,76 @@ mod tests { | |||
| } | |||
| } | |||
| } | |||
|
|
|||
| #[test] | |||
| fn normalize_cols() { | |||
There was a problem hiding this comment.
I added some additional unit test coverage here for normalize when I was writing the unnormalize versions
datafusion/src/logical_plan/expr.rs
Outdated
| @@ -1120,7 +1120,7 @@ pub fn columnize_expr(e: Expr, input_schema: &DFSchema) -> Expr { | |||
|
|
|||
| /// Recursively call [`Column::normalize`] on all Column expressions | |||
| /// in the `expr` expression tree. | |||
| pub fn normalize_col(e: Expr, schemas: &[&DFSchemaRef]) -> Result<Expr> { | |||
| pub fn normalize_col(expr: Expr, schemas: &[&DFSchemaRef]) -> Result<Expr> { | |||
There was a problem hiding this comment.
Changed to use the name expr to be more consistent with the rest of the codebase
|
FYI @houqp |
Dandandan
reviewed
Jul 6, 2021
Dandandan
approved these changes
Jul 6, 2021
alamb
force-pushed
the
alamb/fix_parquet_pruning
branch
from
20f7472
to
de6de17
Compare
alamb
force-pushed
the
alamb/fix_parquet_pruning
branch
from
de6de17
to
c2971f9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes ##656
Based on #657 (test for parquet pruning) so review that first
Rationale for this change
Predicate pushdown for parquet is currently broken because the filtering logic gets columns that refer to
foo.barbut the table provider only knows about columns (e.g. only aboutbar)What changes are included in this PR?
unnormalize_exprfunction to remove qualifiers from expressionsAre there any user-facing changes?
Pruning works again