-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
blockchain, cmd, netsync, main: Add utxocache #1955
Commits on Dec 13, 2023
-
This change is part of the effort to add utxocache support to btcd. sizehelper introduces code for 2 main things: 1: Calculating how many entries to allocate for a map given a size in bytes. 2: Calculating how much a map takes up in memory given the entries were allocated for the map. These functionality are useful for allocating maps so that they'll be allocating below a certain number of bytes. Since go maps will always allocate in powers of B (where B is the bucket size for the given map), it may allocate too much memory. For example, for a map that can store 8GB of entries, the map will grow to be 16GB once the map is full and the caller puts an extra entry onto the map. If we want to give a memory guarantee to the user, we can either: 1: Limit the cache size to fixed sizes (4GB, 8GB, ...). 2: Allocate a slice of maps. The sizehelper code helps with (2).
Configuration menu - View commit details
-
Copy full SHA for e318551 - Browse repository at this point
Copy the full SHA e318551View commit details
Commits on Dec 16, 2023
-
This change is part of the effort to add utxocache support to btcd. mapslice allows the caller to allocate a fixed amount of memory for the utxo cache maps without the mapslice going over that fixed amount of memory. This is useful as we can have variable sizes (1GB, 1.1GB, 2.3GB, etc) while guaranteeing a memory limit.
Configuration menu - View commit details
-
Copy full SHA for 1f2dfa2 - Browse repository at this point
Copy the full SHA 1f2dfa2View commit details -
blockchain: Add tfFresh to txoFlags
This change is part of the effort to add utxocache support to btcd. The fresh flag indicates that the entry is fresh and that the parent view (the database) hasn't yet seen the entry. This is very useful as a performance optimization for the utxo cache as if a fresh entry is spent, we can simply remove it from the cache and don't bother trying to delete it from the database.
Configuration menu - View commit details
-
Copy full SHA for 35c4268 - Browse repository at this point
Copy the full SHA 35c4268View commit details -
blockchain: Add memoryUsage() method on UtxoEntry
This change is part of the effort to add utxocache support to btcd. Getting the memory usage of an entry is very useful for the utxo cache as we need to know how much memory all the cached entries are using to guarantee a cache usage limit for the end user.
Configuration menu - View commit details
-
Copy full SHA for 27cf702 - Browse repository at this point
Copy the full SHA 27cf702View commit details -
blockchain: Add utxoStateConsistency read and write functions
This change is part of the effort to add utxocache support to btcd. The utxoStateConsistency indicates what the last block that the utxo cache got flush at. This is useful for recovery purposes as if the node is unexpectdly shut down, we know which block to start rebuilding the utxo state from.
Configuration menu - View commit details
-
Copy full SHA for fc65744 - Browse repository at this point
Copy the full SHA fc65744View commit details -
blockchain: Refactor dbPutUtxoView
This change is part of the effort to add utxocache support to btcd. dbPutUtxoView handled putting and deleting new/spent utxos from the database. These two functinalities are refactored to their own functions: dbDeleteUtxoEntry and dbPutUtxoEntry. Refactoring these out allows the cache to call these two functions directly instead of having to create a view and saving that view to disk.
Configuration menu - View commit details
-
Copy full SHA for d86e79e - Browse repository at this point
Copy the full SHA d86e79eView commit details -
blockchain: Return early on nil utxo view in dbPutUtxoView
This change is part of the effort to add utxocache support to btcd. connectBlock may have an empty utxoviewpoint as the block verification process may be using the utxo cache directly. In that case, a nil utxo viewpoint will be passed in. Just return early on a nil utxoviewpoint.
Configuration menu - View commit details
-
Copy full SHA for 953d62a - Browse repository at this point
Copy the full SHA 953d62aView commit details -
blockchain: Require utxoBucket in dbFetchUtxoEntry
This change is part of the effort to add utxocache support to btcd. Require the caller to pass in the utxoBucket as the caller may be fetching many utxos in one loop. Having the caller pass it in removes the need for dbFetchUtxoEntry to grab the bucket on every single fetch.
Configuration menu - View commit details
-
Copy full SHA for bcd8f54 - Browse repository at this point
Copy the full SHA bcd8f54View commit details -
blockchain: Refactor fetchInputUtxos
This change is part of the effort to add utxocache support to btcd. fetchInputUtxos had mainly 2 functions: 1: Figure out which outpoints to fetch 2: Call fetchUtxosMain to fetch those outpoints Functionality for (1) is refactored out to fetchInputsToFetch. This is done to allow fetchInputUtxos to use the cache to fetch the outpoints as well in a later commit.
Configuration menu - View commit details
-
Copy full SHA for 053ef33 - Browse repository at this point
Copy the full SHA 053ef33View commit details -
The implemented utxocache implements connectTransactions just like utxoviewpoint and can be used as a drop in replacement for connectTransactions. One thing to note is that unlike the utxoViewpoint, the utxocache immediately deletes the spent entry from the cache. This means that the utxocache is unfit for functions like checkConnectBlock where you expect the entry to still exist but be marked as spent. disconnectTransactions is purposely not implemented as using the cache during reorganizations may leave the utxo state inconsistent if there is an unexpected shutdown. The utxoViewpoint will still have to be used for reorganizations.
Configuration menu - View commit details
-
Copy full SHA for 3c11e48 - Browse repository at this point
Copy the full SHA 3c11e48View commit details -
blockchain, netsync, main, cmd/addblock: Use utxocache
This change is part of the effort to add utxocache support to btcd. utxo cache is now used by the BlockChain struct. By default it's used and the minimum cache is set to 250MiB. The change made helps speed up block/tx validation as the cache allows for much faster lookup of utxos. The initial block download in particular is improved as the db i/o bottleneck is remedied by the cache.
Configuration menu - View commit details
-
Copy full SHA for 16cd44f - Browse repository at this point
Copy the full SHA 16cd44fView commit details -
database/ffldb: make PruneBlocks atomic
PruneBlocks used to delete files immediately before the database transaction finished. By making the prune atomic, we can guarantee that the database flush will happen before the utxo cache is flushed, ensuring that the utxo cache is never in an irrecoverable state.
Configuration menu - View commit details
-
Copy full SHA for d387d16 - Browse repository at this point
Copy the full SHA d387d16View commit details -
blockchain: add flushNeededAfterPrune
flushNeededAfterPrune returns true if the utxocache needs to be flushed after the pruning of the given slice of block hashes. For the utxo cache to be recoverable while pruning is enabled, we need to make sure that there exists blocks since the last utxo cache flush. If there are blocks that are deleted after the last utxo cache flush, the utxo set is irrecoverable. The added method provides a way to tell if a flush is needed.
Configuration menu - View commit details
-
Copy full SHA for dd37dfa - Browse repository at this point
Copy the full SHA dd37dfaView commit details -
blockchain: flush the utxo cache on prune if needed
If the prune will delete block past the last flush hash of the utxocache, the cache will need to be flushed first to avoid a case where the utxocache is irrecoverable. The newly added code adds this flush logic to connectBlock.
Configuration menu - View commit details
-
Copy full SHA for ebc93a3 - Browse repository at this point
Copy the full SHA ebc93a3View commit details -
ffldb: change export_test.go to export.go
The testing function in export_test.go is changed to just export.go so that callers outside the ffldb package will be able to call the function. The main use for this is so that the prune code can be triggered from the blockchain package. This allows testing code to have less than 1.5GB worth of blocks to trigger the prune.
Configuration menu - View commit details
-
Copy full SHA for 84bdd01 - Browse repository at this point
Copy the full SHA 84bdd01View commit details -
Configuration menu - View commit details
-
Copy full SHA for 26b2e9d - Browse repository at this point
Copy the full SHA 26b2e9dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 87a81f1 - Browse repository at this point
Copy the full SHA 87a81f1View commit details