ParquetExec::statistics() does not read statistics for many column types (like timstamps, strings, etc)

[alamb]

Describe the bug

While working on #8229 I found another bug that is non obvious, but that can be clearly seen now thanks to #8110 and #8111 from @NGA-TRAN

To Reproduce

❯ copy (values ('foo'), ('bar'), ('baz')) to '/tmp/strings.parquet';
+-------+
| count |
+-------+
| 3     |
+-------+
1 row in set. Query took 0.023 seconds.

And then look at the explain verbose up can see there are no min/max statisics shown:

❯ explain verbose select * from '/tmp/strings.parquet';

|                                                            |                                                                                                                                                                |
| physical_plan_with_stats                                   | ParquetExec: file_groups={1 group: [[private/tmp/strings.parquet]]}, projection=[column1], statistics=[Rows=Exact(3), Bytes=Absent, [(Col[0]: Null=Exact(0))]] |
|                                                            |                                                                                                                                                                |
+------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
80 rows in set. Query took 0.002 seconds.

Expected behavior

I expect there to be min/max values extracted in the statistics for the strings, as there are for integers ((Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(3)))

❯ copy (values (1), (2), (3)) to '/tmp/ints.parquet';
+-------+
| count |
+-------+
| 3     |
+-------+
1 row in set. Query took 0.023 seconds.

❯ explain verbose select * from '/tmp/ints.parquet';
...
                                                                                                               |
| physical_plan                                              | ParquetExec: file_groups={1 group: [[private/tmp/ints.parquet]]}, projection=[column1]                                                                                                              |
|                                                            |                                                                                                                                                                                                     |
| physical_plan_with_stats                                   | ParquetExec: file_groups={1 group: [[private/tmp/ints.parquet]]}, projection=[column1], statistics=[Rows=Exact(3), Bytes=Absent, [(Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(3)) Null=Exact(0))]] |
|                                                            |                                                                                                                                                                                                     |
+------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Additional context

No response

[alamb]

Note that the pruning predicate code does correctly read the statistics for other strings and timestamps, because it uses a different code path

[alamb]

I plan to fix this

[Weijun-H]

Could I pick this ticket up?

[Weijun-H]

In fn summarize_min_max, it cannot handle ByteArray(ValueStatistics<ByteArray>) well. Do we need to convert it to a different type like timestamps, strings, etc 🤔 ?

[alamb]

In fn summarize_min_max, it cannot handle ByteArray(ValueStatistics<ByteArray>) well. Do we need to convert it to a different type like timestamps, strings, etc 🤔 ?

I think there is some subtly related to decimals as well -- the best thing to do is probably to study what the existing code in row_groups does -- I think it is here https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L57

[alamb]

At some point there were multiple code paths to extract statistics in parquet (one for file level and one for row group level) that should likely be combined

[alamb]

I believe we have fixed this with #10453 -- statistics are now correctly extracted

| physical_plan_with_stats                                   | ParquetExec: file_groups={1 group: [[tmp/strings.parquet]]}, projection=[column1], statistics=[Rows=Exact(3), Bytes=Absent, [(Col[0]: Min=Exact(Utf8("bar")) Max=Exact(Utf8("foo")) Null=Exact(0))]] |
|                                                            |                                                                                                                                                                                                      |
| physical_plan_with_schema                                  | ParquetExec: file_groups={1 group: [[tmp/strings.parquet]]}, projection=[column1], schema=[column1:Utf8;N]                                                                                           |
|                                                            |                                                                                                                                                                                                      |
+------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
79 row(s) fetched.
Elapsed 0.007 seconds.