Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: minor cleanup to simulation docs #21777

Merged
merged 4 commits into from
Sep 18, 2024
Merged

docs: minor cleanup to simulation docs #21777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e08f5da
minor cleanup and fixing of links
tac0turtle Sep 17, 2024
04db3be
remove distribution decoders
tac0turtle Sep 17, 2024
51b117e
revert removal of decoders
tac0turtle Sep 17, 2024
1120b96
Update docs/learn/advanced/12-simulation.md
tac0turtle Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions docs/build/building-modules/14-simulator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ and then unmarshals the value from the `KVPair` to the type provided.

You can use the example [here](https://github.com/cosmos/cosmos-sdk/blob/main/x/distribution/simulation/decoder.go) from the distribution module to implement your store decoders.

If the module uses the `collections` package, you can use the example [here](https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/x/bank/module.go#L166) from the Bank module to implement your store decoders.

### Randomized genesis

The simulator tests different scenarios and values for genesis parameters
Expand All @@ -61,33 +63,33 @@ Operations on the simulation are simulated using the full [transaction cycle](..
Shown below is how weights are set:

```go reference
https://github.com/cosmos/cosmos-sdk/blob/release/v0.50.x/x/staking/simulation/operations.go#L19-L86
https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/x/staking/depinject.go#L144-L154
```

As you can see, the weights are predefined in this case. Options exist to override this behavior with different weights. One option is to use `*rand.Rand` to define a random weight for the operation, or you can inject your own predefined weights.

Here is how one can override the above package `simappparams`.

```go reference
https://github.com/cosmos/cosmos-sdk/blob/release/v0.51.x/Makefile#L292-L334
```

The SDK simulations can be executed like normal tests in Go from the shell or within an IDE.
Make sure that you pass the `-tags='sims` parameter to enable them and other params that make sense for your scenario.

```go reference
https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/scripts/build/simulations.mk#L19
```

### Random proposal contents

Randomized governance proposals are also supported on the Cosmos SDK simulator. Each
module must define the governance proposal `Content`s that they expose and register
them to be used on the parameters.
module must register the message to be used for governance proposals.

```go reference
https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/x/staking/depinject.go#L139-L142
```

## Registering simulation functions

Now that all the required functions are defined, we need to integrate them into the module pattern within the `module.go`:

```go reference
https://github.com/cosmos/cosmos-sdk/blob/release/v0.50.x/x/distribution/module.go#L180-L203
https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/x/staking/depinject.go#L127-L154
```

## App Simulator manager
Expand Down Expand Up @@ -133,5 +135,5 @@ The simulations provide deterministic behaviour already. The integration with th
can be done at a high level with the deterministic pseudo random number generator where the fuzzer provides varying numbers.

```go reference
https://github.com/cosmos/cosmos-sdk/blob/release/v0.51.x/Makefile#L352-L355
https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/scripts/build/simulations.mk#L80-L84
```
40 changes: 12 additions & 28 deletions docs/learn/advanced/12-simulation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,37 +7,26 @@ sidebar_position: 1
The Cosmos SDK offers a full fledged simulation framework to fuzz test every
message defined by a module.

On the Cosmos SDK, this functionality is provided by [`SimApp`](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/simapp/app_v2.go), which is a
`Baseapp` application that is used for running the [`simulation`](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/simulation) module.
This module defines all the simulation logic as well as the operations for
randomized parameters like accounts, balances etc.
On the Cosmos SDK, this functionality is provided by [`SimApp`](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/simapp/app_v2.go), which is a `Baseapp` application that is used for running the [`simulation`](https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/simsx/README.md#L1) package. This package defines all the simulation logic as well as the operations for randomized parameters like accounts, balances etc.

## Goals

The blockchain simulator tests how the blockchain application would behave under
real life circumstances by generating and sending randomized messages.
The goal of this is to detect and debug failures that could halt a live chain,
by providing logs and statistics about the operations run by the simulator as
well as exporting the latest application state when a failure was found.
The blockchain simulator tests how the blockchain application would behave under real life circumstances by generating and sending randomized messages. The goal of this is to detect and debug failures that could halt a live chain, by providing logs and statistics about the operations run by the simulator as well as exporting the latest application state when a failure was found.

Its main difference with integration testing is that the simulator app allows
you to pass parameters to customize the chain that's being simulated.
This comes in handy when trying to reproduce bugs that were generated in the
provided operations (randomized or not).
Its main difference with integration testing is that the simulator app allows you to pass parameters to customize the chain that's being simulated. This comes in handy when trying to reproduce bugs that were generated in the provided operations (randomized or not).

## Simulation commands

The simulation app has different commands, each of which tests a different
failure type:

* `AppImportExport`: The simulator exports the initial app state and then it
creates a new app with the exported `genesis.json` as an input, checking for
inconsistencies between the stores.
* `AppImportExport`: The simulator exports the initial app state and then it creates a new app with the exported `genesis.json` as an input, checking for inconsistencies between the stores.
* `AppSimulationAfterImport`: Queues two simulations together. The first one provides the app state (_i.e_ genesis) to the second. Useful to test software upgrades or hard-forks from a live chain.
* `AppStateDeterminism`: Checks that all the nodes return the same values, in the same order.
* `BenchmarkInvariants`: Analysis of the performance of running all modules' invariants (_i.e_ sequentially runs a [benchmark](https://pkg.go.dev/testing/#hdr-Benchmarks) test). An invariant checks for
differences between the values that are on the store and the passive tracker. Eg: total coins held by accounts vs total supply tracker.
* `BenchmarkInvariants`: Analysis of the performance of running all modules' invariants (_i.e_ sequentially runs a [benchmark](https://pkg.go.dev/testing/#hdr-Benchmarks) test). An invariant checks for differences between the values that are on the store and the passive tracker. Eg: total coins held by accounts vs total supply tracker.
* `FullAppSimulation`: General simulation mode. Runs the chain and the specified operations for a given number of blocks. Tests that there're no `panics` on the simulation. It does also run invariant checks on every `Period` but they are not benchmarked.
* `FuzzFullAppSimulation`: Runs general simulation mode with the [go fuzzer](https://go.dev/doc/security/fuzz/) to find panics.
* `AppStateDeterminism`: Runs a few seeds many times to test that the apphash is deterministic across the runs.

Each simulation must receive a set of inputs (_i.e_ flags) such as the number of
blocks that the simulation is run, seed, block size, etc.
Expand All @@ -47,23 +36,18 @@ Check the full list of flags [here](https://github.com/cosmos/cosmos-sdk/blob/v0

In addition to the various inputs and commands, the simulator runs in three modes:

1. Completely random where the initial state, module parameters and simulation
parameters are **pseudo-randomly generated**.
2. From a `genesis.json` file where the initial state and the module parameters are defined.
This mode is helpful for running simulations on a known state such as a live network export where a new (mostly likely breaking) version of the application needs to be tested.
3. From a `params.json` file where the initial state is pseudo-randomly generated but the module and simulation parameters can be provided manually.
This allows for a more controlled and deterministic simulation setup while allowing the state space to still be pseudo-randomly simulated.
The list of available parameters are listed [here](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/simulation/client/cli/flags.go#L59-L78).
1. Completely random where the initial state, module parameters and simulation parameters are **pseudo-randomly generated**.
2. From a `genesis.json` file where the initial state and the module parameters are defined. This mode is helpful for running simulations on a known state such as a live network export where a new (mostly likely breaking) version of the application needs to be tested.
3. From a `params.json` file where the initial state is pseudo-randomly generated but the module and simulation parameters can be provided manually. This allows for a more controlled and deterministic simulation setup while allowing the state space to still be pseudo-randomly simulated. All available parameters are listed [here](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/simulation/client/cli/flags.go#L59-L78).

:::tip
These modes are not mutually exclusive. So you can for example run a randomly
generated genesis state (`1`) with manually generated simulation params (`3`).
These modes are not mutually exclusive. So you can for example run a randomly generated genesis state (`1`) with manually generated simulation params (`3`).
:::

## Usage

This is a general example of how simulations are run. For more specific examples
check the Cosmos SDK [Makefile](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/Makefile#L282-L318).
check the Cosmos SDK [Makefile](https://github.com/cosmos/cosmos-sdk/blob/23cf89cce1882ba9c8280e64735ae200504bfdce/scripts/build/simulations.mk#L1-L104).

```bash
$ go test -mod=readonly github.com/cosmos/cosmos-sdk/simapp \
Expand Down
13 changes: 9 additions & 4 deletions simsx/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@
This package introduces some new helper types to simplify message construction for simulations (sims). The focus is on better dev UX for new message factories.
Technically, they are adapters that build upon the existing sims framework.

#### * [Message factory](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/msg_factory.go)
## [Message factory](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/msg_factory.go)

Simple functions as factories for dedicated sdk.Msgs. They have access to the context, reporter and test data environment. For example:

```go
func MsgSendFactory() simsx.SimMsgFactoryFn[*types.MsgSend] {
return func(ctx context.Context, testData *simsx.ChainDataSource, reporter simsx.SimulationReporter) ([]simsx.SimAccount, *types.MsgSend) {
Expand All @@ -17,26 +18,30 @@ func MsgSendFactory() simsx.SimMsgFactoryFn[*types.MsgSend] {
}
```

#### * [Sims registry](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/registry.go)
## [Sims registry](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/registry.go)

A new helper to register message factories with a default weight value. They can be overwritten by a parameters file as before. The registry is passed to the AppModule type. For example:

```go
func (am AppModule) WeightedOperationsX(weights simsx.WeightSource, reg simsx.Registry) {
reg.Add(weights.Get("msg_send", 100), simulation.MsgSendFactory())
reg.Add(weights.Get("msg_multisend", 10), simulation.MsgMultiSendFactory())
}
```

#### * [Reporter](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/reporter.go)
## [Reporter](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/reporter.go)

The reporter is a flow control structure that can be used in message factories to skip execution at any point. The idea is similar to the testing.T Skip in Go stdlib. Internally, it converts skip, success and failure events to legacy sim messages.
The reporter also provides some capability to print an execution summary.
It is also used to interact with the test data environment to not have errors checked all the time.
Message factories may want to abort early via

```go
if reporter.IsSkipped() {
return nil, nil
}
```

#### * [Test data environment](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/environment.go)
## [Test data environment](https://github.com/cosmos/cosmos-sdk/blob/main/simsx/environment.go)

The test data environment provides simple access to accounts and other test data used in most message factories. It also encapsulates some app internals like bank keeper or address codec.
1 change: 1 addition & 0 deletions types/simulation/collections.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"github.com/cosmos/cosmos-sdk/types/kv"
)

// NewStoreDecoderFuncFromCollectionsSchema returns a function that decodes two kv pairs when the module fully uses collections
func NewStoreDecoderFuncFromCollectionsSchema(schema collections.Schema) func(kvA, kvB kv.Pair) string {
colls := schema.ListCollections()
prefixes := make([][]byte, len(colls))
Expand Down
Loading