Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Constant space addToStoreFromDump and deduplicate code #3801

Merged
merged 15 commits into from
Jul 21, 2020

Commits on Jul 14, 2020

  1. Configuration menu
    Copy the full SHA
    9ec1004 View commit details
    Browse the repository at this point in the history
  2. Crudely make addToStoreFromDump take Source not string

    I just as little beyond the type as possible, so the implementation
    changes this enables can be reviewed separately.
    Ericson2314 committed Jul 14, 2020
    Configuration menu
    Copy the full SHA
    c86fc3a View commit details
    Browse the repository at this point in the history
  3. Dedup LocalStore::addToStore*

    The downsides is that the coroutine has byte-by-byte loop transfer. Will
    fix that next.
    Ericson2314 committed Jul 14, 2020
    Configuration menu
    Copy the full SHA
    9de96ef View commit details
    Browse the repository at this point in the history
  4. LocalStore::addToStoreFromDump copy in chunks

    Rather than copying byte-by-byte, we let the coroutine know how much
    data we would like it to send back to us.
    Ericson2314 committed Jul 14, 2020
    Configuration menu
    Copy the full SHA
    592851f View commit details
    Browse the repository at this point in the history
  5. Fix localhost::addToStore(...Path...)

    We were calculating the nar hash wrong when the file ingestion method
    was flat. I don't think there's anything we can do in that case but dump
    the file again, so that's what I do.
    
    As an optomization, we again could reuse the original dump for just the
    recursive and non-sha256 case, but I rather do that after this fix, and
    after my other PRs which deduplicate this code.
    Ericson2314 committed Jul 14, 2020
    Configuration menu
    Copy the full SHA
    8173e7b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    650c2c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2020

  1. Configuration menu
    Copy the full SHA
    aff69e2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d087cf4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    64b7421 View commit details
    Browse the repository at this point in the history
  4. Get rid of LocalStore::addToStoreCommon

    I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
    a store and then doing nothing too fancy with it.
    
    `LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
    bare-bones sinkToSource of the right dump command.
    Ericson2314 committed Jul 15, 2020
    Configuration menu
    Copy the full SHA
    bc10964 View commit details
    Browse the repository at this point in the history
  5. Revert "LocalStore::addToStoreFromDump copy in chunks"

    This reverts commit 592851f. We don't
    need this extra feature anymore
    Ericson2314 committed Jul 15, 2020
    Configuration menu
    Copy the full SHA
    5602637 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2020

  1. Fix bug in TeeSource

    We use this to simplify `LocalStore::addToStoreFromDump`.
    
    Also, hope I fixed build error with old clang (used in Darwin CI).
    Ericson2314 committed Jul 16, 2020
    Configuration menu
    Copy the full SHA
    3dcca18 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2020

  1. Configuration menu
    Copy the full SHA
    eab945f View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2020

  1. Configuration menu
    Copy the full SHA
    9aae179 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidian…

    …systems/nix into from-dump-stream
    Ericson2314 committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    5a37413 View commit details
    Browse the repository at this point in the history