[Epic] Improved DataFusion Benchmarking

[alamb]

Call to action:

Let's invest more effort in DataFusion benchmarking, both as a mechanism for technical evangelism as well as a guide for actual performance improvements.

Background

We have several examples of performance “comparisons” showing DataFusion not doing well against DuckDB or pola.rs that really was a test of how fast CSV or JSON parsing can go (this blog is one such example) – recent work should make these comparisons much more favorable in the future

It is in the interest of all projects based on DataFusion to focus on their own users and use cases rather than having to explain why they are using supposedly "inferior" technology due to misleading benchmark results (for example recently on ClickBench – see #5276).

Of course not only will improved benchmarking help evangelize DataFusion more, it will also directly help guide the community’s optimization efforts.

Related Tickets

• Review existing datafusion benchmarks and clean them up #5502
• Run DataFusion benchmarks regularly and track performance history over time #5504
• Update benchmarks on clickbench #5276
• Add "first impression" benchmark to bench.sh #6156
• Improve speed of benchmark verification tests #4032
• Report and compare benchmark runs against two branches #5561
• Poor reported performance of DataFusion against DuckDB and Hyper #5942
• Easy DataFusion / DataFusion Benchmarking #6127
• Add a clickbench DataFusion benchmark runner #6128
• Datafusion v19.rc1 scan parquet 20x slower than DuckDB v0.6.1 on 15GB ClickBench data #5404
• Clickbench q32 not working in 32GB RAM #5969
• Add datafusion to the DuckDB flavored h2o.ai benchmark #6584

[comphead]

@alamb 5276 included twice

[jackwener]

@alamb 5276 included twice

Thanks for reminder, I have removed it.

[alamb]

Here is a proposed PR to orchestrate running the benchmarks: #6131