Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stream large files during unpacking #2707

Merged
merged 2 commits into from
Apr 6, 2021
Merged

Stream large files during unpacking #2707

merged 2 commits into from
Apr 6, 2021

Commits on Apr 5, 2021

  1. Stream large files during unpacking 

    Fixes rust-lang#2632, rust-lang#2145, rust-lang#2564

Files over 16M are now written incrementally chunks rather than buffered
in memory in one full linear buffer. This chunk size is not
configurable.

For threaded unpacking, the entire memory buffer will be used to buffer
chunks and a single worker thread will dispatch IO operations from the
buffer, so minimal performance impact should be anticipated (file
size/16M round trips at worst, and most network file systems will
latency hide linear writes).

For immediate unpacking, each chunk is dispatched directly to disk,
which may impact performance as less latency hiding is possible - but
for immediate unpacking clarity of behaviour is the priority.
    @rbtcollins
    rbtcollins committed Apr 5, 2021
    Configuration menu
    Copy the full SHA
    a6e9176 View commit details
    Browse the repository at this point in the history

  2. Fix lints

    @rbtcollins
    rbtcollins committed Apr 5, 2021
    Configuration menu
    Copy the full SHA
    8ad8b74 View commit details
    Browse the repository at this point in the history