Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

blockchain, cmd, netsync, main: Add utxocache #1955

Merged
merged 17 commits into from
Dec 19, 2023

Commits on Dec 13, 2023

  1. blockchain: Add sizehelper

    This change is part of the effort to add utxocache support to btcd.
    
    sizehelper introduces code for 2 main things:
        1: Calculating how many entries to allocate for a map given a size
           in bytes.
        2: Calculating how much a map takes up in memory given the entries
           were allocated for the map.
    
    These functionality are useful for allocating maps so that they'll be
    allocating below a certain number of bytes.  Since go maps will always
    allocate in powers of B (where B is the bucket size for the given map),
    it may allocate too much memory.  For example, for a map that can store
    8GB of entries, the map will grow to be 16GB once the map is full and
    the caller puts an extra entry onto the map.
    
    If we want to give a memory guarantee to the user, we can either:
        1: Limit the cache size to fixed sizes (4GB, 8GB, ...).
        2: Allocate a slice of maps.
    
    The sizehelper code helps with (2).
    kcalvinalvin committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    e318551 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2023

  1. blockchain: Add mapslice

    This change is part of the effort to add utxocache support to btcd.
    
    mapslice allows the caller to allocate a fixed amount of memory for the
    utxo cache maps without the mapslice going over that fixed amount of
    memory.  This is useful as we can have variable sizes (1GB, 1.1GB, 2.3GB,
    etc) while guaranteeing a memory limit.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    1f2dfa2 View commit details
    Browse the repository at this point in the history
  2. blockchain: Add tfFresh to txoFlags

    This change is part of the effort to add utxocache support to btcd.
    
    The fresh flag indicates that the entry is fresh and that the parent
    view (the database) hasn't yet seen the entry.  This is very useful as
    a performance optimization for the utxo cache as if a fresh entry is
    spent, we can simply remove it from the cache and don't bother trying to
    delete it from the database.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    35c4268 View commit details
    Browse the repository at this point in the history
  3. blockchain: Add memoryUsage() method on UtxoEntry

    This change is part of the effort to add utxocache support to btcd.
    
    Getting the memory usage of an entry is very useful for the utxo cache
    as we need to know how much memory all the cached entries are using to
    guarantee a cache usage limit for the end user.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    27cf702 View commit details
    Browse the repository at this point in the history
  4. blockchain: Add utxoStateConsistency read and write functions

    This change is part of the effort to add utxocache support to btcd.
    
    The utxoStateConsistency indicates what the last block that the utxo
    cache got flush at.  This is useful for recovery purposes as if the node
    is unexpectdly shut down, we know which block to start rebuilding the
    utxo state from.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    fc65744 View commit details
    Browse the repository at this point in the history
  5. blockchain: Refactor dbPutUtxoView

    This change is part of the effort to add utxocache support to btcd.
    
    dbPutUtxoView handled putting and deleting new/spent utxos from the
    database.  These two functinalities are refactored to their own
    functions: dbDeleteUtxoEntry and dbPutUtxoEntry.
    
    Refactoring these out allows the cache to call these two functions
    directly instead of having to create a view and saving that view to
    disk.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    d86e79e View commit details
    Browse the repository at this point in the history
  6. blockchain: Return early on nil utxo view in dbPutUtxoView

    This change is part of the effort to add utxocache support to btcd.
    
    connectBlock may have an empty utxoviewpoint as the block verification
    process may be using the utxo cache directly.  In that case, a nil utxo
    viewpoint will be passed in.  Just return early on a nil utxoviewpoint.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    953d62a View commit details
    Browse the repository at this point in the history
  7. blockchain: Require utxoBucket in dbFetchUtxoEntry

    This change is part of the effort to add utxocache support to btcd.
    
    Require the caller to pass in the utxoBucket as the caller may be
    fetching many utxos in one loop.  Having the caller pass it in removes
    the need for dbFetchUtxoEntry to grab the bucket on every single fetch.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    bcd8f54 View commit details
    Browse the repository at this point in the history
  8. blockchain: Refactor fetchInputUtxos

    This change is part of the effort to add utxocache support to btcd.
    
    fetchInputUtxos had mainly 2 functions:
    1: Figure out which outpoints to fetch
    2: Call fetchUtxosMain to fetch those outpoints
    
    Functionality for (1) is refactored out to fetchInputsToFetch.  This is
    done to allow fetchInputUtxos to use the cache to fetch the outpoints
    as well in a later commit.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    053ef33 View commit details
    Browse the repository at this point in the history
  9. blockchain: Add utxocache

    The implemented utxocache implements connectTransactions just like
    utxoviewpoint and can be used as a drop in replacement for
    connectTransactions.
    
    One thing to note is that unlike the utxoViewpoint, the utxocache
    immediately deletes the spent entry from the cache.  This means that the
    utxocache is unfit for functions like checkConnectBlock where you expect
    the entry to still exist but be marked as spent.
    
    disconnectTransactions is purposely not implemented as using the cache
    during reorganizations may leave the utxo state inconsistent if there is
    an unexpected shutdown.  The utxoViewpoint will still have to be used
    for reorganizations.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    3c11e48 View commit details
    Browse the repository at this point in the history
  10. blockchain, netsync, main, cmd/addblock: Use utxocache

    This change is part of the effort to add utxocache support to btcd.
    
    utxo cache is now used by the BlockChain struct.  By default it's used
    and the minimum cache is set to 250MiB.  The change made helps speed up
    block/tx validation as the cache allows for much faster lookup of utxos.
    The initial block download in particular is improved as the db i/o
    bottleneck is remedied by the cache.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    16cd44f View commit details
    Browse the repository at this point in the history
  11. database/ffldb: make PruneBlocks atomic

    PruneBlocks used to delete files immediately before the database
    transaction finished.  By making the prune atomic, we can guarantee that
    the database flush will happen before the utxo cache is flushed,
    ensuring that the utxo cache is never in an irrecoverable state.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    d387d16 View commit details
    Browse the repository at this point in the history
  12. blockchain: add flushNeededAfterPrune

    flushNeededAfterPrune returns true if the utxocache needs to be flushed
    after the pruning of the given slice of block hashes.  For the utxo
    cache to be recoverable while pruning is enabled, we need to make sure
    that there exists blocks since the last utxo cache flush.  If there are
    blocks that are deleted after the last utxo cache flush, the utxo set is
    irrecoverable.  The added method provides a way to tell if a flush is
    needed.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    dd37dfa View commit details
    Browse the repository at this point in the history
  13. blockchain: flush the utxo cache on prune if needed

    If the prune will delete block past the last flush hash of the
    utxocache, the cache will need to be flushed first to avoid a case
    where the utxocache is irrecoverable.  The newly added code adds this
    flush logic to connectBlock.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    ebc93a3 View commit details
    Browse the repository at this point in the history
  14. ffldb: change export_test.go to export.go

    The testing function in export_test.go is changed to just export.go so
    that callers outside the ffldb package will be able to call the
    function.
    
    The main use for this is so that the prune code can be triggered from
    the blockchain package.  This allows testing code to have less than
    1.5GB worth of blocks to trigger the prune.
    kcalvinalvin committed Dec 16, 2023
    Configuration menu
    Copy the full SHA
    84bdd01 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    26b2e9d View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    87a81f1 View commit details
    Browse the repository at this point in the history