Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write Bloom filters between row groups instead of the end #5860

Merged
merged 16 commits into from
Jun 21, 2024
Merged

Write Bloom filters between row groups instead of the end #5860

merged 16 commits into from
Jun 21, 2024

Commits on Jun 10, 2024

  1. Add example script to write Parquet files with a Bloom filter

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    759767b View commit details
    Browse the repository at this point in the history

  2. Write Bloom filters between row groups instead of the end 

    This allows Bloom filters to not be saved in memory, which can save
significant space when writing long files
    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    a417f01 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into interleave-bloom

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    5daf96f View commit details
    Browse the repository at this point in the history

  4. Add WriterProperties::bloom_filter_position

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    6effa7f View commit details
    Browse the repository at this point in the history

  5. Mutate the right row group metadata 

    When using BloomFilterPosition::AfterRowGroup this was only writing the Bloom Filter
offset to a temporary clone of the metadata, causing the Bloom Filter to never
be seen by readers
    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    f23759a View commit details
    Browse the repository at this point in the history

  6. Add a test for Bloom Filters written at the end

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    83b475e View commit details
    Browse the repository at this point in the history

  7. Update async writer accordingly

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    3f810b5 View commit details
    Browse the repository at this point in the history

  8. Undo accidental commit

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    ad0c40e View commit details
    Browse the repository at this point in the history

  9. Clippy

    @progval
    progval committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    74c40ee View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Apply suggestions from code review 

    Improve documentation

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
    @progval @alamb
    progval and alamb authored Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f237e8c View commit details
    Browse the repository at this point in the history

  2. Rewrite example with constants as parameters and fewer dependencies

    @progval
    progval committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    d2a7ab8 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into interleave-bloom

    @progval
    progval authored Jun 13, 2024
    Configuration menu
    Copy the full SHA
    b434eea View commit details
    Browse the repository at this point in the history

  4. rustfmt

    @progval
    progval committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    942c6ab View commit details
    Browse the repository at this point in the history

  5. Clippy

    @progval
    progval committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    e4a588d View commit details
    Browse the repository at this point in the history

  6. Fix MSRV

    @progval
    progval committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ed9e576 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Merge remote-tracking branch 'apache/master' into interleave-bloom

    @alamb
    alamb committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    abd81ef View commit details
    Browse the repository at this point in the history