core/rawdb: freezer batch write #23462
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
core/rawdb: improve append benchmark core/rawdb: add freezer batch core/rawdb: minor benchmark nitpicks core/rawdb: add freezer batch core/rawdb: method docs core/rawdb: resolve freezer batch filenum at write-time core/rawdb: use bytes.Buffer in freezer batch core/rawdb: work in progress -- rlp encoding inside freezer batch core/rawdb: implement write batchs in freezer core/rawdb: improve memory allocations for snappy compression core/rawdb: polish core/rawdb: simplify + fix copy/paste error core/rawdb: error handling for freezertable batches core/rawdb: lint nitpicks core/rawdb: more tests for freezer batching core/rawdb: freezer batch interface core/rawdb: WIP freezer batch interface core/rawdb: check insert count of table batches core/rawdb: use ancient batch in background freeze core/rawdb: fix some issues in background freezer ethdb: document AncientBatch core/rawdb: remove single-item append core/rawdb: use dumpIndexString in batch tests core/rawdb: fix batch test core/rawdb: implement commit during append core/rawdb: add some missing close calls in freezer tests core/rawdb: simplify snappy buffer core/rawdb: change ancient writer interface core/rawdb: fix offset test core/rawdb: track headBytes only in freezerTable core/rawdb: add writeLock core/rawdb: reuse write batch core/rawdb: re-add metrics reporting core/rawdb: recreate the concurrent truncate test of freezer This changes the test slightly to use the 'freezer' object instead of 'freezerTable'. This is necessary because the concurrency handling has moved to the freezer and concurrent append and truncate is no longer allowed on the table object. core/rawdb: fix item count after ModifyAncients and improve the test core/rawdb: add concurrency test for retrieve core/rawdb: allow overriding max table size in newFreezer This makes the concurrency tests fail with -race. core/rawdb: fix race in advanceFile core/rawdb: remove atomic access on freezer.headId It doesn't need to be atomic, all accesses are protected by the lock. core/rawdb: use int64 for writeSize core/rawdb: implement and test all-or-nothing behavior of ModifyAncients core/rawdb: move buffer code to batch file core/rawdb: reimplement batch test on freezer core/rawdb: fix race in AncientSize core, core/rawdb: WIP batch ancient write in sync core/rawdb: delete old batch tests core: fix error handling when inserting side chain receipts This fixes (well, rewrites) the test for the error case of InsertReceiptChain where side chain data is imported and setting the fast block fails because it doesn't match the header chain. This previously relied on the terminateInsert hook. With the new ancient write interface, there is no good place for this hook anymore, so the test now injects an actual error instead of simulating one. core: improve comments core/blockchain, core/rawdb: less iterator usage when deleting leveldb-data core/rawdb: return write error from WriteAncientBlocks It's easier to test this function when it doesn't just exit the process on failure. core/rawdb: add benchmark for WriteAncientBlocks core/rawdb: use os.RemoveAll in test core/rawdb: avoid allocating in benchmark loop core/rawdb: avoid copying difficulty twice The difficulty is already copied by block.Header(), avoid copying it again. core/rawdb: improve WriteAncientBlocks benchmark core: handle block->header validation
holiman
force-pushed
the
batch_freezer_squash
branch
from
70ac648
to
1c473ed
Compare
|
I'm not sure how it happened that I'm listed as the only committer on that squash. I thought it would be a mix icon showing both of us. That's a bit bungled... |
|
Don't worry about that, we can put me back in the final commit using |
6 tasks
holiman
commented
Sep 16, 2021
Comment on lines
-1104
to
-1106
| bc.hc.headerCache.Purge() | ||
| bc.hc.tdCache.Purge() | ||
| bc.hc.numberCache.Purge() |
There was a problem hiding this comment.
@fjl these are now moved into headerchain, inside SetHead
There was a problem hiding this comment.
Yes, but we don't call it in all cases.
yongjun925
pushed a commit
to DODOEX/go-ethereum
that referenced
this pull request
Dec 3, 2022
This change is a rewrite of the freezer code. When writing ancient chain data to the freezer, the previous version first encoded each individual item to a temporary buffer, then wrote the buffer. For small item sizes (for example, in the block hash freezer table), this strategy causes a lot of system calls for writing tiny chunks of data. It also allocated a lot of temporary []byte buffers. In the new version, we instead encode multiple items into a re-useable batch buffer, which is then written to the file all at once. This avoids performing a system call for every inserted item. To make the internal batching work, the ancient database API had to be changed. While integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were also added there. Co-authored-by: Felix Lange <fjl@twurst.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the rebased and squashed version of #22932. Due to the new batched reader, it became pretty difficult to rebase and retain all commits, so I squashed it, which made the whole thing a lot easier.
@fjl you're welcome to do do a more thorough rebase if you want to keep the commit history, or you can rebase and reown it, I don't really care personally.