Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add size statistics to ParquetMetaData introduced in PARQUET-2261 #5486

Closed
wants to merge 54 commits into from
Closed

Add size statistics to ParquetMetaData introduced in PARQUET-2261 #5486

wants to merge 54 commits into from

Commits on Mar 6, 2024

  1. regen thrift with size statistics added

    @etseidl
    etseidl committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    a7e41c3 View commit details
    Browse the repository at this point in the history

  2. first cut at adding page size statistics

    @etseidl
    etseidl committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    788eef3 View commit details
    Browse the repository at this point in the history

  3. add new stats to chunk metadata test

    @etseidl
    etseidl committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    6296ada View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Merge branch 'apache:master' into size_stats

    @etseidl
    etseidl authored Mar 8, 2024
    Configuration menu
    Copy the full SHA
    84f3d7a View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. fix escapes

    @etseidl
    etseidl committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    0da05a8 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    7301aeb View commit details
    Browse the repository at this point in the history

  3. format

    @etseidl
    etseidl committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    6e5fece View commit details
    Browse the repository at this point in the history

  4. formatting

    @etseidl
    etseidl committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    457eb4a View commit details
    Browse the repository at this point in the history

  5. add escapes

    @etseidl
    etseidl committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    18a5732 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    658512e View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    81c2b2e View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Merge branch 'size_stats' of github.com:etseidl/arrow-rs into size_stats

    @etseidl
    etseidl committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    29dde50 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    84f8512 View commit details
    Browse the repository at this point in the history

  3. add test of SizeStatistics.unencoded_byte_array_data_bytes

    @etseidl
    etseidl committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    9635e5e View commit details
    Browse the repository at this point in the history

  4. test def histogram as well, rename test

    @etseidl
    etseidl committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    c5c07b6 View commit details
    Browse the repository at this point in the history

  5. add an assert

    @etseidl
    etseidl committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    6dd160f View commit details
    Browse the repository at this point in the history

  6. refactor and add test of def histogram with nulls

    @etseidl
    etseidl committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    917b412 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. add test of repetition level histogram

    @etseidl
    etseidl committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    f8961a3 View commit details
    Browse the repository at this point in the history

  2. revert changes to test_roundtrip

    @etseidl
    etseidl committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    73fa099 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. suggestion from review 

    Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
    @etseidl @alamb
    etseidl and alamb authored Jul 1, 2024
    Configuration menu
    Copy the full SHA
    00ca596 View commit details
    Browse the repository at this point in the history

  2. add to documentation as suggested in review

    @etseidl
    etseidl committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6acc500 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. make histograms optional

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    787e3e8 View commit details
    Browse the repository at this point in the history

  2. add histograms to PageIndex

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    46851f4 View commit details
    Browse the repository at this point in the history

  3. use Vec::push()

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    4f8487b View commit details
    Browse the repository at this point in the history

  4. formatting

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    903b06b View commit details
    Browse the repository at this point in the history

  5. check size stats in read metadata

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    fa89836 View commit details
    Browse the repository at this point in the history

  6. check unencoded_byte_array_data_bytes is not set for int cols

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    2800cc7 View commit details
    Browse the repository at this point in the history

  7. rewrite test_byte_array_size_statistics() to not use test_roundtrip()

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    95a0535 View commit details
    Browse the repository at this point in the history

  8. add unencoded_byte_array_data_bytes support in page index

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    fc66a59 View commit details
    Browse the repository at this point in the history

  9. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    542570f View commit details
    Browse the repository at this point in the history

  10. update expected sizes to account for new stats

    @etseidl
    etseidl committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    7be97e5 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. only write SizeStatistics in ColumnMetaData if statistics are enabled

    @etseidl
    etseidl committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    f5ab47b View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. add a little documentation

    @etseidl
    etseidl committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    a008e9e View commit details
    Browse the repository at this point in the history

  2. add ParquetOffsetIndex to avoid double read of OffsetIndex

    @etseidl
    etseidl committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    87ccec2 View commit details
    Browse the repository at this point in the history

  3. cleanup

    @etseidl
    etseidl committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    3eead30 View commit details
    Browse the repository at this point in the history

  4. use less verbose update of variable_length_bytes

    @etseidl
    etseidl committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    ddf40c3 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2024

  1. add some documentation

    @etseidl
    etseidl committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    0ebb72f View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. update to latest thrift (as of 11 Jul 2024) from parquet-format

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    393aea1 View commit details
    Browse the repository at this point in the history

  2. pass None for optional size statistics

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    1c12fb8 View commit details
    Browse the repository at this point in the history

  3. escape HTML tags

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    53cd5fa View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    45f25a8 View commit details
    Browse the repository at this point in the history

  5. don't need to escape brackets in arrays

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    98025cc View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'github/update_parquet_thrift' into size… 

    …_stats
    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    7b59246 View commit details
    Browse the repository at this point in the history

  7. use consistent naming

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    65096dd View commit details
    Browse the repository at this point in the history

  8. suggested doc changes

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    08065ad View commit details
    Browse the repository at this point in the history

  9. more suggested doc changes

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    1cbd4b7 View commit details
    Browse the repository at this point in the history

  10. use more asserts in tests

    @etseidl
    etseidl committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    dce3513 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. move histogram logic into PageMetrics and ColumnMetrics 

    still needs more documentation
    @etseidl
    etseidl committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    f661839 View commit details
    Browse the repository at this point in the history

  2. refactor some to reduce code duplication, finish docs

    @etseidl
    etseidl committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    818a614 View commit details
    Browse the repository at this point in the history

  3. account for new size statistics in heap size calculations

    @etseidl
    etseidl committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    c391dec View commit details
    Browse the repository at this point in the history

  4. add histogram examples to docs

    @etseidl
    etseidl committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    4816a95 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into size_stats

    @etseidl
    etseidl committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e2faf2d View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2024

  1. add some fixmes

    @etseidl
    etseidl committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    d92ae20 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. leave not to self

    @etseidl
    etseidl committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    69dd652 View commit details
    Browse the repository at this point in the history