Add size statistics to ParquetMetaData introduced in PARQUET-2261

[etseidl]

Which issue does this PR close?

Closes #5022

Rationale for this change

Implements new page and column chunk size statistics introduced in PARQUET-2261

What changes are included in this PR?

Adds the necessary structures from the updated parquet.thrift, and adds the code necessary to populate them.

Are there any user-facing changes?

No

[etseidl]

This is my first foray into Rust programming, so I'm not sure everything is done as idiomatically as possible. Submitting this now to get early feedback on my approach. I'm also wondering how much testing to add, and whether this should have a configuration parameter to turn generating the statistics off.

[alamb]

I triggered the CI checks and will try and get to review this over the next few days if no one beats me to it

[Samrose-Ahmed]

Hey @etseidl are you still working on this, otherwise I had some code in a fork, I can try to move this along.

[alamb]

It seems I never looked at this PR but I just skimmed it and it looks reasonable to me -- I think the biggest thing it is currently lacking is tests

[etseidl]

Yeah, sorry, I got sidetracked by other work...and unit tests are the bane of my existence 😅. Let me see if I can get some added in the next few days.

[etseidl]

@alamb I've added two tests so far. I still need to test the repetition level histogram, but I think I've pushed test_roundtrip() in writer.rs as far as I can. Can you look at what I've done so far and see if it's acceptable?

I'm also not entirely sure how to test the new statistics in the page indexes. For now my story is if the ColumnMetaData::SizeStatistics are correct, then the per-page stats in the page indexes must also be correct.

[etseidl]

@alamb I think I've addressed most of your concerns. Please take a look when you have a spare moment. I'm going to set this PR back to draft and begin on splitting into at least 2 more PRs (I've already submitted #6045 for the thrift component).

[alamb]

Certainly. I can submit the format changes first, but I'll keep this open for the time being while I address your comments. Should I open new issues for each PR, or is it sufficient to reference #5022 for all three?

I think using one ticket is just fine. No need to make new tickets

[alamb]

Update: what do you think about doing incremental PRs to a feature branch? #6050

[etseidl]

Update: what do you think about doing incremental PRs to a feature branch? #6050

Sounds good to me. We can all use #6050 to discuss deconfliction in the event of conflicts. And we can afford to be a bit more free wheeling too 😄. For instance, I have a branch for changing the offset_index in ParquetMetaData to a Vec<Vec<ParquetOffsetIndex>>. I almost submitted that PR, but held back because the ripple of changes throughout the parquet code was a bit more than I expected. With the feature branch, I'll feel more comfortable going at a slower rate (add the variable length stats gathering, then change the API, then add histograms, then (perhaps) change the column index API to be consistent with the offset index.

Thank you for helping to coordinate all this ❤️

[etseidl]

Superseded by #6105 and others

[alamb]

🎉