Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

chunker refactor #665

Open
zelig opened this issue Jun 4, 2018 · 0 comments
Open

chunker refactor #665

zelig opened this issue Jun 4, 2018 · 0 comments

Comments

@zelig
Copy link
Member

zelig commented Jun 4, 2018

part of storage refactor part II: #1039

Chunker is a complex and inefficient code, which is really hard to maintain and optimize. We encountered several times that if a problem is in the chunker we need a lot of time to fix it.
Moreover, for future features (e.g. erasure coding, progress bar), we need to improve the chunker code, so we have to prepare for this.

Some details on what to fix:

  • chunker concurrency control similar to BMT hash
  • bmt and sha3 section/segmentrwriter interface. BMT outstanding features #749
  • segment/sectionwriter interface for hasherstore -> chunker
  • progress bar on upload, save to disk, availabiliy Backend for local storing progress bar #459
  • join API for erasure code functionality
    • n out of m race hook for chunk level erasure coding
  • mock chunk storage and light (payloadless) syncing for network simulations
  • move chunker related files to their own subpackage
    • separate joiner to join.go
    • splitters: tree, pyramid, split.go (new shiny pyramid)
  • benchmarks for splitter
  • refactor benchmark tests to run the same test code for tree and pyramid
@cobordism cobordism added this to the 0.3.1 milestone Jun 7, 2018
@gbalint gbalint removed this from the 0.3.1 milestone Aug 2, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants