Add support for ClickBench in bench.sh

[alamb]

Which issue does this PR close?

Part of #6994

Rationale for this change

See #6994

The ClickBench benchmarks are both widely used in the industry and focus on grouping / aggregation / filtering: https://github.com/ClickHouse/ClickBench/tree/main/datafusion

What changes are included in this PR?

Add support in bench.sh for downloading the ClickBench dataset (both single file as well as partitioned)

You can download it now via:

./bench.sh data clickbench_1 # single parquet file
./bench.sh data clickbench_partitioned # 100 parquet file partitioned dataset

Example

./bench.sh data clickbench_partitioned
***************************
DataFusion Benchmark Runner and Data Generator
COMMAND: data
BENCHMARK: clickbench_partitioned
DATA_DIR: /home/alamb/arrow-datafusion/benchmarks/data
CARGO_COMMAND: cargo run --profile release-nonlto
***************************
Checking hits_partitioned... downloading with 10 parallel workers.................................................................................................... Done

I also plan to update the runner so it can run the clickbench queries, but will do so as a follow on PR

./bench.sh run clickbench_1 # single parquet file
./bench.sh run clickbench_partitioned # 100 parquet file partitioned dataset

Are these changes tested?

Tested manually

Are there any user-facing changes?

No, this is a development tool

[appletreeisyellow]

Thank you @alamb for adding the benchmark shell for ClickBench! Looks good to me! I have some questions about the echo message below. They are not blocking, just want to understand them

[appletreeisyellow]

1. Do you mean the other way? Since OUTPUT_SIZE is constant in the if statement and dynamic in the else statement

Suggested change

	if test "${OUTPUT_SIZE}" = "14779976446"; then
	echo -n "... found ${OUTPUT_SIZE} bytes ..."
	else
	URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet"
	echo -n "... downloading ${URL} (14GB) ... "
	if test "${OUTPUT_SIZE}" = "14779976446"; then
	echo -n "... downloading ${URL} (14GB) ... "
	else
	URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet"
	echo -n "... found ${OUTPUT_SIZE} bytes ..."

2. Just curious, why the unit is byte in line 353 and KB in line 376? Should they be the same?

[alamb]

Yes, you are right, they should both by bytes (KB was left over from a previous implementation). Nice eyes

[alamb]

Do you mean the other way? Since OUTPUT_SIZE is constant in the if statement and dynamic in the else statement

I think this is the right way -- basically this code is testing if a file with the expected size already exists, and if so it doesn't download it again.

This means that you can run bench.sh download clickbench_1 and be sure you have the most recent data but not have to download all 14GB if it already exists

[appletreeisyellow]

This means that you can run bench.sh download clickbench_1 and be sure you have the most recent data but not have to download all 14GB if it already exists

Ah, got it! Thank you for the explanation

[alamb]

Thank you @alamb for adding the benchmark shell for ClickBench! Looks good to me! I have some questions about the echo message below. They are not blocking, just want to understand them

Thank you very much for the review @appletreeisyellow