Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moved StorageTransaction to the fuel-core-storage crate #1694

Merged
merged 42 commits into from
Mar 13, 2024

Conversation

xgreenx
Copy link
Collaborator

@xgreenx xgreenx commented Feb 21, 2024

Closes #1589

Overview

The change moves the database transaction logic from the fuel-core to the fuel-core-storage level. The corresponding issue described the reason behind it.

Technical details of implementation

The change splits the KeyValueStore into KeyValueInspect and KeyValueMutate, as well the Blueprint into BlueprintInspect and BlueprintMutate. It allows requiring less restricted constraints for any read-related operations.

One of the main ideas of the change is to allow for the actual storage only to implement KeyValueInspect and Modifiable without the KeyValueMutate. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the Modifiable::commit_changes). This feature is used to track the height of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a separate commit.

The StorageTransaction is a StructuredStorage that uses StorageTransactionInner inside to accumulate modifications. Only StorageStorageInner has a real implementation of the KeyValueMutate(Other types only implement it in tests).

The implementation of the Modifiable for the Database contains a business logic that provides additional safety but limits the usage of the database. The Database now tracks its height and is responsible for its updates. In the commit_changes function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the OnChain database, we are looking for a new Block in the FuelBlocks table).

As was planned in the issue, now the executor has full control over how commits to the storage are done.

All mutation methods now require &mut self - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the Database implements the Clone trait. To be sure that we don't corrupt the state of the database, the commit_changes function implements additional safety checks to be sure that we commit updates per each height only once time.

Side changes:

  • The drop function was moved from Database to RocksDB as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed.
  • The StatisticTable table lives in the off-chain worker.
  • Removed duplication of the Database from the dapp::ConcreteStorage since it is already available from the VM.
  • The executor return only produced Changes instead of the storage transaction, which simplifies the interaction between modules and port definition.
  • The logic related to the iteration over the storage is moved to the fuel-core-storage crate and is now reusable. It provides an interator method that duplicates the logic from MemoryStore on iterating over the BTreeMap and methods like iter_all, iter_all_by_prefix, etc. It was done in a separate revivable commit.
  • The MemoryTransactionView is fully replaced by the StorageTransactionInner.
  • Removed flush method from the Database since it is not needed after Fix for the long database initialization after restart #1664 .

xgreenx and others added 10 commits February 15, 2024 16:51
# Conflicts:
#	crates/fuel-core/src/database/storage.rs
#	crates/fuel-core/src/graphql_api/database.rs
#	crates/fuel-core/src/graphql_api/ports.rs
#	crates/fuel-core/src/graphql_api/storage.rs
#	crates/fuel-core/src/graphql_api/worker_service.rs
#	crates/storage/src/column.rs
# Conflicts:
#	CHANGELOG.md
#	crates/types/src/services/txpool.rs
@xgreenx xgreenx self-assigned this Feb 21, 2024
@xgreenx xgreenx added the no changelog Skip the CI check of the changelog modification label Feb 21, 2024
Base automatically changed from feature/blocks-second-key-extraction to master February 22, 2024 00:33
# Conflicts:
#	crates/fuel-core/src/database/block.rs
#	crates/fuel-core/src/database/storage.rs
#	crates/fuel-core/src/graphql_api/ports.rs
#	crates/fuel-core/src/graphql_api/worker_service.rs
#	crates/storage/src/blueprint.rs
#	crates/storage/src/blueprint/merklized.rs
#	crates/storage/src/blueprint/plain.rs
#	crates/storage/src/blueprint/sparse.rs
#	crates/storage/src/structured_storage.rs
#	crates/storage/src/structured_storage/blocks.rs
Added helper function to create a `StorageTransaction`.
Re-worked `Changes` to use `HashMap<Btree>` for iteration functionality,
The `Database` is responsible for its updating and tracking.
@xgreenx xgreenx requested a review from a team February 23, 2024 11:50
.map(|cow| *cow.root()))
}

fn store_new_block(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved default implementation from the Database here

@xgreenx xgreenx removed the no changelog Skip the CI check of the changelog modification label Feb 23, 2024
# Conflicts:
#	crates/fuel-core/src/database/state.rs
#	crates/fuel-core/src/database/storage.rs
#	crates/fuel-core/src/executor.rs
#	crates/fuel-core/src/graphql_api/worker_service.rs
#	crates/fuel-core/src/schema/dap.rs
#	crates/fuel-core/src/service/genesis/off_chain.rs
#	crates/services/executor/src/executor.rs
#	crates/services/executor/src/ports.rs
#	crates/storage/src/structured_storage.rs
#	crates/storage/src/structured_storage/contracts.rs
# Conflicts:
#	bin/fuel-core/src/cli/snapshot.rs
#	crates/fuel-core/src/combined_database.rs
#	crates/fuel-core/src/database.rs
#	crates/fuel-core/src/database/balances.rs
#	crates/fuel-core/src/database/contracts.rs
#	crates/fuel-core/src/database/state.rs
#	crates/fuel-core/src/database/storage.rs
#	crates/fuel-core/src/service.rs
#	crates/fuel-core/src/service/genesis.rs
#	crates/fuel-core/src/service/genesis/off_chain.rs
#	crates/fuel-core/src/state.rs
#	crates/fuel-core/src/state/in_memory/memory_store.rs
#	crates/fuel-core/src/state/in_memory/transaction.rs
#	crates/fuel-core/src/state/rocks_db.rs
#	crates/services/executor/src/refs/contract.rs
#	crates/storage/src/kv_store.rs
#	crates/storage/src/structured_storage.rs
xgreenx and others added 5 commits March 11, 2024 14:39
# Conflicts:
#	crates/fuel-core/src/database.rs
#	crates/fuel-core/src/service/genesis.rs
#	crates/fuel-core/src/service/genesis/workers.rs
Copy link
Contributor

@bvrooman bvrooman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

47 / 86 files reviewed so far, will continue tomorrow

crates/fuel-core/src/database/balances.rs Show resolved Hide resolved
Copy link
Contributor

@bvrooman bvrooman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 86 files at the time of this comment. A couple of small nits and questions. If you are still interested in doing a group review, that may be helpful for a better understanding of the PR from a high level.

crates/services/importer/src/importer.rs Outdated Show resolved Hide resolved
crates/fuel-core/src/database/state.rs Outdated Show resolved Hide resolved
crates/fuel-core/src/schema/dap.rs Outdated Show resolved Hide resolved
crates/fuel-core/src/database/state.rs Outdated Show resolved Hide resolved
crates/fuel-core/src/state/in_memory/memory_store.rs Outdated Show resolved Hide resolved
crates/services/executor/src/executor.rs Outdated Show resolved Hide resolved
crates/services/relayer/src/storage.rs Outdated Show resolved Hide resolved
crates/storage/src/structured_storage.rs Outdated Show resolved Hide resolved
Dentosal
Dentosal previously approved these changes Mar 13, 2024
Copy link
Member

@Dentosal Dentosal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I think this is finally good to go. Phew.

crates/services/importer/src/importer.rs Outdated Show resolved Hide resolved
crates/services/importer/src/importer.rs Outdated Show resolved Hide resolved
@xgreenx xgreenx merged commit ce72eea into master Mar 13, 2024
31 checks passed
@xgreenx xgreenx deleted the feature/commit-changes branch March 13, 2024 19:36
@xgreenx xgreenx mentioned this pull request Mar 30, 2024
xgreenx added a commit that referenced this pull request Mar 30, 2024
## [Version 0.24.0]

### Added

- [#1786](#1786): Regenesis
now includes off-chain tables.
- [#1716](#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](#1770): Add the new
L1 event type for forced transactions.
- [#1767](#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](#1747): The DA block
height is now included in the genesis state.
- [#1740](#1740): Remove
optional fields from genesis configs
- [#1737](#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after #1664.

- [#1693](#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
#1731
* Regenesis support by @MujkicA in
#1693
* Remove genesis temp tables by @MujkicA in
#1737
* Remove optional fields from configs by @MujkicA in
#1740
* Weekly `cargo update` by @github-actions in
#1745
* Regenesis should also store da block height by @xgreenx in
#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in #1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in #1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in #1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
#1757
* Weekly `cargo update` by @github-actions in
#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
#1769
* Feature/new fti event by @Voxelot in
#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in #1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in #1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in #1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
#1786


**Full Changelog**:
v0.23.0...v0.24.0
@xgreenx xgreenx mentioned this pull request Mar 30, 2024
xgreenx added a commit that referenced this pull request Mar 30, 2024
## Version v0.24.0

### Added

- [#1786](#1786): Regenesis
now includes off-chain tables.
- [#1716](#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](#1770): Add the new
L1 event type for forced transactions.
- [#1767](#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](#1747): The DA block
height is now included in the genesis state.
- [#1740](#1740): Remove
optional fields from genesis configs
- [#1737](#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after #1664.

- [#1693](#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
#1731
* Regenesis support by @MujkicA in
#1693
* Remove genesis temp tables by @MujkicA in
#1737
* Remove optional fields from configs by @MujkicA in
#1740
* Weekly `cargo update` by @github-actions in
#1745
* Regenesis should also store da block height by @xgreenx in
#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in #1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in #1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in #1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
#1757
* Weekly `cargo update` by @github-actions in
#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
#1769
* Feature/new fti event by @Voxelot in
#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in #1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in #1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in #1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
#1786
* Release v0.24.0 by @xgreenx in
#1791
* Moved chain specification into `fuel-core-bin` crate by @xgreenx in
#1792


**Full Changelog**:
v0.23.0...v0.24.1
crypto523 added a commit to crypto523/fuel-core that referenced this pull request Oct 7, 2024
## [Version 0.24.0]

### Added

- [#1786](FuelLabs/fuel-core#1786): Regenesis
now includes off-chain tables.
- [#1716](FuelLabs/fuel-core#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](FuelLabs/fuel-core#1770): Add the new
L1 event type for forced transactions.
- [#1767](FuelLabs/fuel-core#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](FuelLabs/fuel-core#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](FuelLabs/fuel-core#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](FuelLabs/fuel-core#1747): The DA block
height is now included in the genesis state.
- [#1740](FuelLabs/fuel-core#1740): Remove
optional fields from genesis configs
- [#1737](FuelLabs/fuel-core#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](FuelLabs/fuel-core#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](FuelLabs/fuel-core#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](FuelLabs/fuel-core#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](FuelLabs/fuel-core#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](FuelLabs/fuel-core#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](FuelLabs/fuel-core#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](FuelLabs/fuel-core#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](FuelLabs/fuel-core#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](FuelLabs/fuel-core#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](FuelLabs/fuel-core@7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](FuelLabs/fuel-core@5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after FuelLabs/fuel-core#1664.

- [#1693](FuelLabs/fuel-core#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](FuelLabs/fuel-core#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
FuelLabs/fuel-core#1731
* Regenesis support by @MujkicA in
FuelLabs/fuel-core#1693
* Remove genesis temp tables by @MujkicA in
FuelLabs/fuel-core#1737
* Remove optional fields from configs by @MujkicA in
FuelLabs/fuel-core#1740
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1745
* Regenesis should also store da block height by @xgreenx in
FuelLabs/fuel-core#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in FuelLabs/fuel-core#1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in FuelLabs/fuel-core#1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in FuelLabs/fuel-core#1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
FuelLabs/fuel-core#1757
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
FuelLabs/fuel-core#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
FuelLabs/fuel-core#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
FuelLabs/fuel-core#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
FuelLabs/fuel-core#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
FuelLabs/fuel-core#1769
* Feature/new fti event by @Voxelot in
FuelLabs/fuel-core#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
FuelLabs/fuel-core#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in FuelLabs/fuel-core#1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
FuelLabs/fuel-core#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in FuelLabs/fuel-core#1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in FuelLabs/fuel-core#1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
FuelLabs/fuel-core#1786


**Full Changelog**:
FuelLabs/fuel-core@v0.23.0...v0.24.0
crypto523 added a commit to crypto523/fuel-core that referenced this pull request Oct 7, 2024
## Version v0.24.0

### Added

- [#1786](FuelLabs/fuel-core#1786): Regenesis
now includes off-chain tables.
- [#1716](FuelLabs/fuel-core#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](FuelLabs/fuel-core#1770): Add the new
L1 event type for forced transactions.
- [#1767](FuelLabs/fuel-core#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](FuelLabs/fuel-core#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](FuelLabs/fuel-core#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](FuelLabs/fuel-core#1747): The DA block
height is now included in the genesis state.
- [#1740](FuelLabs/fuel-core#1740): Remove
optional fields from genesis configs
- [#1737](FuelLabs/fuel-core#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](FuelLabs/fuel-core#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](FuelLabs/fuel-core#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](FuelLabs/fuel-core#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](FuelLabs/fuel-core#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](FuelLabs/fuel-core#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](FuelLabs/fuel-core#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](FuelLabs/fuel-core#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](FuelLabs/fuel-core#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](FuelLabs/fuel-core#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](FuelLabs/fuel-core@7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](FuelLabs/fuel-core@5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after FuelLabs/fuel-core#1664.

- [#1693](FuelLabs/fuel-core#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](FuelLabs/fuel-core#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
FuelLabs/fuel-core#1731
* Regenesis support by @MujkicA in
FuelLabs/fuel-core#1693
* Remove genesis temp tables by @MujkicA in
FuelLabs/fuel-core#1737
* Remove optional fields from configs by @MujkicA in
FuelLabs/fuel-core#1740
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1745
* Regenesis should also store da block height by @xgreenx in
FuelLabs/fuel-core#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in FuelLabs/fuel-core#1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in FuelLabs/fuel-core#1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in FuelLabs/fuel-core#1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
FuelLabs/fuel-core#1757
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
FuelLabs/fuel-core#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
FuelLabs/fuel-core#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
FuelLabs/fuel-core#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
FuelLabs/fuel-core#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
FuelLabs/fuel-core#1769
* Feature/new fti event by @Voxelot in
FuelLabs/fuel-core#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
FuelLabs/fuel-core#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in FuelLabs/fuel-core#1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
FuelLabs/fuel-core#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in FuelLabs/fuel-core#1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in FuelLabs/fuel-core#1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
FuelLabs/fuel-core#1786
* Release v0.24.0 by @xgreenx in
FuelLabs/fuel-core#1791
* Moved chain specification into `fuel-core-bin` crate by @xgreenx in
FuelLabs/fuel-core#1792


**Full Changelog**:
FuelLabs/fuel-core@v0.23.0...v0.24.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move the DatabaseTransaction logic into StorageTransaction
5 participants