Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

share: WriteEDS #1105

Closed
Tracked by #1099
Wondertan opened this issue Sep 15, 2022 · 0 comments · Fixed by #1139
Closed
Tracked by #1099

share: WriteEDS #1105

Wondertan opened this issue Sep 15, 2022 · 0 comments · Fixed by #1139
Assignees
Labels
area:shares Shares and samples

Comments

@Wondertan
Copy link
Member

Wondertan commented Sep 15, 2022

Context

As per #1099, we want to store EDSes as a whole. For this, the CAR format is chosen.

Implementation Details

To write EDS into a stream/file, WriteEDS is introduced. Internally it:

  • Re-imports EDS similarly to
    ipld.ImportShares
    • Using Blockservice with offline
      exchange
      and in-memory Blockstore
    • With NodeVisitor, which saves to the
      Blockstore only NMT Merkle proofs(no shares) NOTE: len(node.Links()) == 2
      • Actual shares are written further in a particular way explained further
  • Creates and writes header CARv1Header
    • Fills up Roots field with EDS.RowRoots/EDS.ColRoots roots converted into CIDs
  • Iterates over shares in quadrant-by-quadrant order via EDS.GetCell
    • Writes the shares in row-by-row order
  • Iterates over in-memory Blockstore and writes NMT Merkle
    proofs stored in it

NOTES:

  • CAR provides a utility to serialize any DAG into the file
    and there is a way to serialize EDS into DAG(share/ipld.ImportShares). This approach is the simplest and traverses
    shares and Merkle Proofs in a depth-first manner packing them in a CAR file. However, this is incompatible with the
    requirement of being able to truncate the CAR file reading out only the first quadrant out of it without NMT proofs,
    so serialization must be different from the utility to support that.
  • Alternatively to WriteEDS, an EDSReader could be introduced to make EDS-to-stream handling more idiomatic
    and efficient in some cases, with the cost of more complex implementation.
// WriteEDS writes the whole EDS into the given io.Writer as CARv1 file.
// All its shares and recomputed NMT proofs.
func WriteEDS(context.Context, *rsmt2d.ExtendedDataSquare, io.Writer) error
@Wondertan Wondertan added the area:shares Shares and samples label Sep 15, 2022
This was referenced Sep 15, 2022
distractedm1nd added a commit that referenced this issue Oct 19, 2022
Closes #1105

Co-authored-by: Viacheslav <viacheslavgonkivskyi@gmail.com>
Co-authored-by: rene <41963722+renaynay@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:shares Shares and samples
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants