[EPIC] Improving Performance

[andygrove]

What is the problem the feature request solves?

This epic is a place to track various ideas around improving query performance.

Some of these ideas apply to upstream DataFusion rather than being Comet-specific.

Planned Work

• Investigate performance of CASE WHEN expression #669
• Investigate performance of decimal math / aggregation #670
• Investigate options for improving performance when reading decimals from Parquet #679

Ideas to Research

These are longer term ideas to explore.

• Avoid transition to Comet in some cases if the overhead of R2C and C2R outweighs the benefit #571
• Use selection vectors instead of copying batches during filter operations (see paper at https://www.pdl.cmu.edu/ftp/Database/ngom-damon2021.pdf)
• Implement StringView / BinaryView
  • Can particulary help in the parquet scan/filter case to avoid string copies. See Andrew Lamb's talk from the June 24 Bay Area DataFusion Meetup for more info.
• Should we implement native versions of RowToColumnar / ColumnarToRow?
• Check that we are removing FilterExec when all filter conditions are successfully pushed down to Parquet, to avoid evaluating the filter predicates twice
• Implement tooling for saving output of query stage to disk so that we can benchmark individual query stages in rust, outside of spark
• Spark will remove dictionary-encoding after a cast (and maybe after other expressions?) but it could be advantageous to retain the dictionary encoding so that upstream native operators can take advantage of this?

Ideas no longer being pursued

• Use JIT for evaluating nested expressions to avoid intermediate arrays (there was a datafusion-jit module, but it was abandonded, so we need to see why)
  • Update: previous experiments in Arrow/DataFusion did not show a speedup from this appraoch and it added a lot of extra code and complexity
• Use mutable vectors during expression evaluation to avoid intermediate arrays (in-place updates are available in arrow-rs)

Related DataFusion issues

• Fuse grouped aggregate and filter operators for improved performance datafusion#5944
• Proposal: Change Accumulator trait to accept RecordBatch / num_rows to allow faster Count datafusion#8067
• Return TableProviderFilterPushDown::Exact when Parquet Pushdown Enabled datafusion#4028
• List of all performance issues in DataFusion

Related Comet issues:

• chore: Investigate impact of small batches on performance #495

Describe the potential solution

No response

Additional context

No response