feat: Add bloom filter metric to ParquetExec #8772
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
physical-expr
Jefffrey
reviewed
Jan 6, 2024
datafusion/core/src/datasource/physical_plan/parquet/metrics.rs
Outdated
Show resolved
Hide resolved
my-vegetable-has-exploded
changed the title
simonvandel
reviewed
Jan 6, 2024
datafusion/core/src/datasource/physical_plan/parquet/metrics.rs
Outdated
Show resolved
Hide resolved
|
I think the CI failure https://github.com/apache/arrow-datafusion/actions/runs/7430577648/job/20220434682?pr=8772 is unrelated to the changes in this PR (and was fixed on main -- so this branch should pass next time it is merged to main) |
alamb
reviewed
Jan 8, 2024
There was a problem hiding this comment.
This looks great @my-vegetable-has-exploded -- thank you for the contribution
I think we should consider the different name suggested by @simonvandel but otherwise i think this PR is pretty much ready to go!
| format!( | ||
| ", pruning_predicate={} [{}]", | ||
| pre.predicate_expr(), | ||
| pre.literal_guarantees() |
There was a problem hiding this comment.
this is a neat idea to display guarantees 👍
| .as_str(), | ||
| Some(0), | ||
| Some(1), | ||
| // we don't support pruning by statistics for in_list with more than 20 elements currently |
There was a problem hiding this comment.
Also I think it the pruning by bloom filters happens first, so it may not even try to prune by statistics
👍
| @@ -61,7 +61,7 @@ Filter: parquet_table.column1 != Int32(42) | |||
| physical_plan | |||
| CoalesceBatchesExec: target_batch_size=8192 | |||
| --FilterExec: column1@0 != 42 | |||
| ----ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..101], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:101..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:202..303], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:303..403]]}, projection=[column1], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1 | |||
| ----ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..101], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:101..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:202..303], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:303..403]]}, projection=[column1], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1 [column1 not in (42)] | |||
There was a problem hiding this comment.
what do you think about adding a header to the guarantees to it was more explicit for interpreting?
Suggested change
| ----ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..101], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:101..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:202..303], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:303..403]]}, projection=[column1], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1 [column1 not in (42)] | |
| ----ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..101], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:101..202], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:202..303], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:303..403]]}, projection=[column1], predicate=column1@0 != 42, pruning_predicate=column1_min@0 != 42 OR 42 != column1_max@1 required_guarantees=[column1 not in (42)] |
7 tasks
my-vegetable-has-exploded
force-pushed
the
metric-sbbf-tests
branch
from
b88b7ea
to
ecfccdd
Compare
alamb
approved these changes
Jan 9, 2024
There was a problem hiding this comment.
Looks great -- thank you very much @my-vegetable-has-exploded
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #8767,
Closes #8768
Rationale for this change
What changes are included in this PR?
row_groups_prunedmetric into two parts, includingrow_groups_pruned_sbbfandrow_groups_pruned_sbbf.row_group_pruning.rs， by checking value ofrow_groups_pruned_sbbfandrow_groups_pruned_sbbfLiteralGuaranteesofParquetExecAre these changes tested?
Are there any user-facing changes?