Fix/Da Compression Correct Multi-block Back Reference Target #1275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roynalnaruto
added
the
da-compression
roynalnaruto
changed the base branch from
feat/da-compression-witgen
to
feat/da-compr-multi-blob
roynalnaruto
added a commit
that referenced
this pull request
May 22, 2024
* add test * detour zstd dep Signed-off-by: noelwei <fan@scroll.io> * tbt: test for multi-blob * update zstd dep Signed-off-by: noelwei <fan@scroll.io> * chore: remove unused patch * update zstd (scroll repo) Signed-off-by: noelwei <fan@scroll.io> * Correct multi-block back reference target bytes (#1275) Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * minor fix in assignment (block_info and seq_info) * dbg: fix some issues * dbg: fix bitstring table lookup for byte_idx delta * fix: repeated offsets are carried forward into next block * wip test: large-multi-block * update zstd, set windowlog Signed-off-by: noelwei <fan@scroll.io> * fix: is_init continue if nil=1 * refactor bitstring table to separate out bitstrings of different len --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: noelwei <fan@scroll.io> Co-authored-by: Ray Gao <qg2153@columbia.edu>
roynalnaruto
added a commit
that referenced
this pull request
May 24, 2024
* Remove deprecate tests * Remove zstd encoding of literals * Remove raw/rle blocks * Remove block raw/rle scenarios * Remove lstream tag * Remove raw/rle bytes processing * Literals witness row assignment * Remove huffman code component from witness * Add witness rows to sequence section * Add debug flags * Add sequence instruction table witness * Finish recovering original input. sequence decoding done * Add witness rows for sequence FSE tables * Add witness rows for sequence header * Remove debug flags * fmt * Add debug flags * Add debug flags * Add debug flags * Add unit tester * Update Fse construction * Correct fse construction * Correct tag length * Temporarily recover const * Assign literals header rows * Assign columns * Resolve merge compile errors * Modify tag config assignment * Correct block config from witgen * Assign additional seq bitstream fields * Correct tags * configure gadgtes * Add more data fields onto fse * Correct fse assignment * fmt * Remove debug flags * fmt * Fix witness assignment error * fmt * Increase fse fixed capacity * Correct bitstream column assignmnet * Correct assignment for bitstring table * fmt * Correct comment * Remove FseSymbol * Rename FSE section * Recover tests * Correct block header assignment * Correct fse decoder assignment * fmt * Remove constants * Add debug flags * Correct offset increment * Assign padding * Isolate gates * Remove gates * Remove gates * Remove gates * Remove gates * Adjust gates * Recover gates * fix: q_enable fixed column to avoid active gates on unusable rows * wip dbg: non-padded rows except first row * fix: tag transition has been fixed * fix(witgen): tag is_change=true * wip(dbg:gate): continue same tag * fix: compilation (u8 table load) and fhd gate works * fix: fcs OK || unusable_rows=14 * test: fcs is OK * fix(gate): tag=BlockHeader OK * test(lookup): tag=BlockHeader Block_Size OK * test(gate): processing block content OK * fix(gate+lookup): literals header and literals header table * fix(gate): tag=RawBytes OK * fix(gate): tag=SequencesHeader (header decoder fixed order of bits) * fix(fse): sorted state table (partially) * wip(fse sorted states): gates active on unusable rows? * fix(fse): fse table and sorted table OK * witgen: bitstring table OK * Correct skipped bitstream rows * Recover gadgets * Recover gadgets * Recover gadgets * Recover gates * Recover gates * Recover component * Recover constraint * Recover constraints * Recover gates * Recover gates * Recover constraints * Recover constraints * Recover constraints * Recover constraints * fix(lookup): var-bit-packing and other rows of tag=FseCode * restore gate * fix(lookup): bitstream table lookup (start + end) * Correct sequence data init * Correct fse in sequence data decoding * Recover gates * Recover constraints * fix(lookup): interleaved order lookup OK * fix(gate): tag=Null OK * Correct states, symbols, values in bitstream decoding witness * Adjust gates * Adjust fixed table * Remove debug flag * fix(fse_decoder): is_mlt and is_mot expr computation * fse table lookup OK * Correct seq_idx * Recover constraints * Recover gate * fix: constraints updated * chore: degree-overflow fix * Remove is_next_null condition * Add next_nb * Correct tail end bit read idx * fix: update various cases of nil * Correct nil row index * Correct bitstream decoding idx * `SequenceInstructionTable` and `SequenceExecutionConfig` (#1259) * rebase to upstream Signed-off-by: noelwei <fan@scroll.io> * wip of addrtable Signed-off-by: noelwei <fan@scroll.io> * all gates Signed-off-by: noelwei <fan@scroll.io> * change the lookup purpose Signed-off-by: noelwei <fan@scroll.io> * fix according to reviews * purge unused seqvaluetable Signed-off-by: noelwei <fan@scroll.io> * change zero testing to corresponding gadget complete assignment Signed-off-by: noelwei <fan@scroll.io> * trivial fixing Signed-off-by: noelwei <fan@scroll.io> * purge the duplicated works Signed-off-by: noelwei <fan@scroll.io> * unit test (WIP) Signed-off-by: noelwei <fan@scroll.io> * seq exec circuit (WIP) Signed-off-by: noelwei <fan@scroll.io> * seq exec circuit (WIP) Signed-off-by: noelwei <fan@scroll.io> * seq exec circuit (WIP) Signed-off-by: noelwei <fan@scroll.io> * output region: gates and lookups (WIP) Signed-off-by: noelwei <fan@scroll.io> * pass unittest for seqinst table Signed-off-by: noelwei <fan@scroll.io> * seq exec: complete the seq num lookup Signed-off-by: noelwei <fan@scroll.io> * add seq exec info in witgen Signed-off-by: noelwei <fan@scroll.io> * assign and unit tests (WIP) Signed-off-by: noelwei <fan@scroll.io> * refactor for better assignment Signed-off-by: noelwei <fan@scroll.io> * assignments and unit tests (WIP) Signed-off-by: noelwei <fan@scroll.io> * induce debug utilities in AddressRow Signed-off-by: noelwei <fan@scroll.io> * pass first unit test (WIP) Signed-off-by: noelwei <fan@scroll.io> * chore: integrate seq-inst-table and seq-exec-config into decoder-config * refactor to low degree Signed-off-by: noelwei <fan@scroll.io> * more unittest for seq exec (WIP) Signed-off-by: noelwei <fan@scroll.io> * pass unit tests Signed-off-by: noelwei <fan@scroll.io> * add assign entry Signed-off-by: noelwei <fan@scroll.io> * update some witgens, pass decoder's unit test Signed-off-by: noelwei <fan@scroll.io> * integrate seq exec into decoder (WIP) Signed-off-by: noelwei <fan@scroll.io> * temporary disable 3 lookups and unit test pass for the rest Signed-off-by: noelwei <fan@scroll.io> * all of the unit test passed Signed-off-by: noelwei <fan@scroll.io> * trivial updates: head condition in seq exec and exported cells Signed-off-by: noelwei <fan@scroll.io> * clear the warnings Signed-off-by: noelwei <fan@scroll.io> * chore: fmt --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * chore: most clippy issues/warnings resolved * wip(dbg): decode encoded batch data * fix: literals header's size format bits * DA-Compression (Missing Pieces for single encoded batch) (#1263) * Fix repeated match byte slice: * Add batch witgen test * Adjust test * fix: handle predefined table appropriately in table reconstruction * Skip predefined fse tables: * Add copy constraints and assigned exports * wip dbg: expected vs got * fix: rlc_acc and rlc, handle is_reverse approprately * fix decoder rlc (#1266) Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * fix: constraints/lookups OK for batch0000 * fix: literal header assignment fix, batch127 failure * Add tag transition fix for fse * fix: fse table handling of prob=-1 and state_idx/sym_count_acc * fix: fse -> seq data order * fix: handle first real symbol in prob=-1 table cases --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> Co-authored-by: Ho <noel.wei@gmail.com> * commit from @noel2004: f181535 * Dbg (large block) (#1270) * data and test * wip dbg: offset > currently decoded (overflow subtraction) * chore: bitwise op table generic over op and range of lhs/rhs * wip dbg: more blobs (witgen fail) * fix: repeat offset usage (ll ==0) and dont include skipped states * Account for nil row in fse * Fix fse nil row byte skip * Correct is_update_state for nil row in sequence data * fix (fse state increasing, but may not be in u8) and fmt * chore: remove println --------- Co-authored-by: Ray Gao <qg2153@columbia.edu> * Refactor/Cleanup Witgen code (#1271) * Remove debug flags * Remove aux data * fmt and add comments * Recover test * Remove import * Recover test * Recover test * Assign padding * Remove unused variables * Remove initial assignment * Add comments * fmt * fmt * fmt * fmt * Organize comments: * tag_value unused | refactor max_len * more refactor and clippy * fix: handle multi-block witgen --------- Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * chore: refactor zstd config into init fn * fix(tag=sequences_data): columns can change iff conditions are met * Support multi-block (#1273) * add test * detour zstd dep Signed-off-by: noelwei <fan@scroll.io> * tbt: test for multi-blob * update zstd dep Signed-off-by: noelwei <fan@scroll.io> * chore: remove unused patch * update zstd (scroll repo) Signed-off-by: noelwei <fan@scroll.io> * Correct multi-block back reference target bytes (#1275) Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * minor fix in assignment (block_info and seq_info) * dbg: fix some issues * dbg: fix bitstring table lookup for byte_idx delta * fix: repeated offsets are carried forward into next block * wip test: large-multi-block * update zstd, set windowlog Signed-off-by: noelwei <fan@scroll.io> * fix: is_init continue if nil=1 * refactor bitstring table to separate out bitstrings of different len --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: noelwei <fan@scroll.io> Co-authored-by: Ray Gao <qg2153@columbia.edu> * fix: handle padding in blob-data and batch-data configs (integration tests OK) --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> Co-authored-by: Ho <noel.wei@gmail.com> Co-authored-by: noelwei <fan@scroll.io>
lispc
added a commit
that referenced
this pull request
May 24, 2024
* feat: copy existing witgen logic * Separation of `BlobDataConfig` and `BatchDataConfig` (#1214) * blob is the encoded form of batch * fix: bytes_rlc in batch data config is constrained, unit tests OK * Decoder "input region" (#1217) * feat: general gates, tag gates, FHD and FCS * feat: block processing, block header, literals header * fix: degree reduction * feat: add block_idx to decoder's block config * fix: block idx is the index of the block (not the byte index within block) * feat: decode sequences section header * fix: constraints for padded rows * ROM Tables (LLC, MLC, MOC) and Bitstring Accumulation Table (#1222) * feat: add ROM tables for fse code to value (sequence section) * fix: handle probability=0 case (fse) * todo: prob=-1 case not sure how to handle * feat: bitstring accumulation table * print digest of compressed bytes (#1226) * fix: tests should read batches in sorted order (batch000 before batch001 and so on) * Add bitstream decoder config (#1230) * feat: bitstream decoder config in decoder config * add num_sequences to block region * Bitstream Decoding and FSE Table Recovery (#1234) * feat: initial constraints for fse table recovery and bitstream decoding * feat: add check for table_kind (LLT, MOT, MLT) in tag=fseCode * feat: add fse table, related constraints/lookups * fix: degree reduction * chore: some comments * chore: renaming * chore: first row of bitstring table, renaming * chore: degree assertion * Multiple Fixes and Improvements (#1239) * fix: tag_idx == tag_len check on prev row * impr: literals header table to use block_idx * impr: fse table also uses (block_idx, table_kind) identifier * fix: handle is_nil and is_nb0 separately in bitstrream decoder * fix: add q_first=0 to fse table lookup * Sequence Decoding and Execution (#1241) * seq decoder conf | tag=seqData | roms | more todos * feat: sequences data section | more todos * fix: fse table lookup and update-state * feat: checks for nb at init-state * feat: fse and fse sorted states table | support for prob=-1 case * witgen (fse::reconstruct) with prob=-1 case covered * FSE `Predefined_Mode` support for sequences section (#1246) * initial work * tests: fse table reconstruction for default distributions * tests: predefined MLT and MOT added * feat: support predefined mode (compr mode) in decoder config * `FixedTable` instead of multiple ROM tables (#1247) * fix: reduce 2 columns for rom fse order * feat: add the fixed table module * chore(refactor): remove rom tables and use fixed table lookup * fix: pass fixed table by reference * Self-review and missing constraints (misc fixes/refactoring) (#1248) * literals header table (block_idx) and bitstring table (byte_idx) * wip: end of block (start of new block) * wip: more constraints for last rows of fsecode and seqdata * update: last row of tag=FseCode * fix: need a single byte to compare byte value * chore: add a todo * fix: review changes * fix: tag=FseCode last row constraint * chore: enable while adding SeqExecTable * `tag=FseCode` should take into account "variable bit-packing" (#1251) * fix: account for variable bit-packing in fse code section * chore: range starts from 1 (ignore 0) * block fields in tag=BlockHeader * DA Compression Witness Generation and Assignments (#1232) * Remove deprecate tests * Remove zstd encoding of literals * Remove raw/rle blocks * Remove block raw/rle scenarios * Remove lstream tag * Remove raw/rle bytes processing * Literals witness row assignment * Remove huffman code component from witness * Add witness rows to sequence section * Add debug flags * Add sequence instruction table witness * Finish recovering original input. sequence decoding done * Add witness rows for sequence FSE tables * Add witness rows for sequence header * Remove debug flags * fmt * Add debug flags * Add debug flags * Add debug flags * Add unit tester * Update Fse construction * Correct fse construction * Correct tag length * Temporarily recover const * Assign literals header rows * Assign columns * Resolve merge compile errors * Modify tag config assignment * Correct block config from witgen * Assign additional seq bitstream fields * Correct tags * configure gadgtes * Add more data fields onto fse * Correct fse assignment * fmt * Remove debug flags * fmt * Fix witness assignment error * fmt * Increase fse fixed capacity * Correct bitstream column assignmnet * Correct assignment for bitstring table * fmt * Correct comment * Remove FseSymbol * Rename FSE section * Recover tests * Correct block header assignment * Correct fse decoder assignment * fmt * Remove constants * Add debug flags * Correct offset increment * Assign padding * Isolate gates * Remove gates * Remove gates * Remove gates * Remove gates * Adjust gates * Recover gates * fix: q_enable fixed column to avoid active gates on unusable rows * wip dbg: non-padded rows except first row * fix: tag transition has been fixed * fix(witgen): tag is_change=true * wip(dbg:gate): continue same tag * fix: compilation (u8 table load) and fhd gate works * fix: fcs OK || unusable_rows=14 * test: fcs is OK * fix(gate): tag=BlockHeader OK * test(lookup): tag=BlockHeader Block_Size OK * test(gate): processing block content OK * fix(gate+lookup): literals header and literals header table * fix(gate): tag=RawBytes OK * fix(gate): tag=SequencesHeader (header decoder fixed order of bits) * fix(fse): sorted state table (partially) * wip(fse sorted states): gates active on unusable rows? * fix(fse): fse table and sorted table OK * witgen: bitstring table OK * Correct skipped bitstream rows * Recover gadgets * Recover gadgets * Recover gadgets * Recover gates * Recover gates * Recover component * Recover constraint * Recover constraints * Recover gates * Recover gates * Recover constraints * Recover constraints * Recover constraints * Recover constraints * fix(lookup): var-bit-packing and other rows of tag=FseCode * restore gate * fix(lookup): bitstream table lookup (start + end) * Correct sequence data init * Correct fse in sequence data decoding * Recover gates * Recover constraints * fix(lookup): interleaved order lookup OK * fix(gate): tag=Null OK * Correct states, symbols, values in bitstream decoding witness * Adjust gates * Adjust fixed table * Remove debug flag * fix(fse_decoder): is_mlt and is_mot expr computation * fse table lookup OK * Correct seq_idx * Recover constraints * Recover gate * fix: constraints updated * chore: degree-overflow fix * Remove is_next_null condition * Add next_nb * Correct tail end bit read idx * fix: update various cases of nil * Correct nil row index * Correct bitstream decoding idx * `SequenceInstructionTable` and `SequenceExecutionConfig` (#1259) * rebase to upstream Signed-off-by: noelwei <fan@scroll.io> * wip of addrtable Signed-off-by: noelwei <fan@scroll.io> * all gates Signed-off-by: noelwei <fan@scroll.io> * change the lookup purpose Signed-off-by: noelwei <fan@scroll.io> * fix according to reviews * purge unused seqvaluetable Signed-off-by: noelwei <fan@scroll.io> * change zero testing to corresponding gadget complete assignment Signed-off-by: noelwei <fan@scroll.io> * trivial fixing Signed-off-by: noelwei <fan@scroll.io> * purge the duplicated works Signed-off-by: noelwei <fan@scroll.io> * unit test (WIP) Signed-off-by: noelwei <fan@scroll.io> * seq exec circuit (WIP) Signed-off-by: noelwei <fan@scroll.io> * seq exec circuit (WIP) Signed-off-by: noelwei <fan@scroll.io> * seq exec circuit (WIP) Signed-off-by: noelwei <fan@scroll.io> * output region: gates and lookups (WIP) Signed-off-by: noelwei <fan@scroll.io> * pass unittest for seqinst table Signed-off-by: noelwei <fan@scroll.io> * seq exec: complete the seq num lookup Signed-off-by: noelwei <fan@scroll.io> * add seq exec info in witgen Signed-off-by: noelwei <fan@scroll.io> * assign and unit tests (WIP) Signed-off-by: noelwei <fan@scroll.io> * refactor for better assignment Signed-off-by: noelwei <fan@scroll.io> * assignments and unit tests (WIP) Signed-off-by: noelwei <fan@scroll.io> * induce debug utilities in AddressRow Signed-off-by: noelwei <fan@scroll.io> * pass first unit test (WIP) Signed-off-by: noelwei <fan@scroll.io> * chore: integrate seq-inst-table and seq-exec-config into decoder-config * refactor to low degree Signed-off-by: noelwei <fan@scroll.io> * more unittest for seq exec (WIP) Signed-off-by: noelwei <fan@scroll.io> * pass unit tests Signed-off-by: noelwei <fan@scroll.io> * add assign entry Signed-off-by: noelwei <fan@scroll.io> * update some witgens, pass decoder's unit test Signed-off-by: noelwei <fan@scroll.io> * integrate seq exec into decoder (WIP) Signed-off-by: noelwei <fan@scroll.io> * temporary disable 3 lookups and unit test pass for the rest Signed-off-by: noelwei <fan@scroll.io> * all of the unit test passed Signed-off-by: noelwei <fan@scroll.io> * trivial updates: head condition in seq exec and exported cells Signed-off-by: noelwei <fan@scroll.io> * clear the warnings Signed-off-by: noelwei <fan@scroll.io> * chore: fmt --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * chore: most clippy issues/warnings resolved * wip(dbg): decode encoded batch data * fix: literals header's size format bits * DA-Compression (Missing Pieces for single encoded batch) (#1263) * Fix repeated match byte slice: * Add batch witgen test * Adjust test * fix: handle predefined table appropriately in table reconstruction * Skip predefined fse tables: * Add copy constraints and assigned exports * wip dbg: expected vs got * fix: rlc_acc and rlc, handle is_reverse approprately * fix decoder rlc (#1266) Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * fix: constraints/lookups OK for batch0000 * fix: literal header assignment fix, batch127 failure * Add tag transition fix for fse * fix: fse table handling of prob=-1 and state_idx/sym_count_acc * fix: fse -> seq data order * fix: handle first real symbol in prob=-1 table cases --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> Co-authored-by: Ho <noel.wei@gmail.com> * commit from @noel2004: f181535 * Dbg (large block) (#1270) * data and test * wip dbg: offset > currently decoded (overflow subtraction) * chore: bitwise op table generic over op and range of lhs/rhs * wip dbg: more blobs (witgen fail) * fix: repeat offset usage (ll ==0) and dont include skipped states * Account for nil row in fse * Fix fse nil row byte skip * Correct is_update_state for nil row in sequence data * fix (fse state increasing, but may not be in u8) and fmt * chore: remove println --------- Co-authored-by: Ray Gao <qg2153@columbia.edu> * Refactor/Cleanup Witgen code (#1271) * Remove debug flags * Remove aux data * fmt and add comments * Recover test * Remove import * Recover test * Recover test * Assign padding * Remove unused variables * Remove initial assignment * Add comments * fmt * fmt * fmt * fmt * Organize comments: * tag_value unused | refactor max_len * more refactor and clippy * fix: handle multi-block witgen --------- Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * chore: refactor zstd config into init fn * fix(tag=sequences_data): columns can change iff conditions are met * Support multi-block (#1273) * add test * detour zstd dep Signed-off-by: noelwei <fan@scroll.io> * tbt: test for multi-blob * update zstd dep Signed-off-by: noelwei <fan@scroll.io> * chore: remove unused patch * update zstd (scroll repo) Signed-off-by: noelwei <fan@scroll.io> * Correct multi-block back reference target bytes (#1275) Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> * minor fix in assignment (block_info and seq_info) * dbg: fix some issues * dbg: fix bitstring table lookup for byte_idx delta * fix: repeated offsets are carried forward into next block * wip test: large-multi-block * update zstd, set windowlog Signed-off-by: noelwei <fan@scroll.io> * fix: is_init continue if nil=1 * refactor bitstring table to separate out bitstrings of different len --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: noelwei <fan@scroll.io> Co-authored-by: Ray Gao <qg2153@columbia.edu> * fix: handle padding in blob-data and batch-data configs (integration tests OK) --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me> Co-authored-by: Ho <noel.wei@gmail.com> Co-authored-by: noelwei <fan@scroll.io> * fix: compile and test OK, fmt, clippy * fix: dont completely randomise txdata (mock data) --------- Signed-off-by: noelwei <fan@scroll.io> Co-authored-by: Ray Gao <qg2153@columbia.edu> Co-authored-by: Ho <noel.wei@gmail.com> Co-authored-by: noelwei <fan@scroll.io> Co-authored-by: Zhang Zhuo <mycinbrin@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In a multi-block scenario, back reference can extend beyond current block into previously decoded bytes.
This PR adds all decoded bytes to a new block's decoding process and ensures correct back reference in sequence data execution.
Type of change