Skip to content

Commit

Permalink
feat: integration test helpers (#15556)
Browse files Browse the repository at this point in the history
  • Loading branch information
@julienrbrt
julienrbrt authored Mar 31, 2023
1 parent 43a4db5 commit aeaa301
Show file tree
Hide file tree
Showing 8 changed files with 371 additions and 157 deletions.
30 changes: 16 additions & 14 deletions docs/architecture/adr-059-test-scopes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
## Changelog

* 2022-08-02: Initial Draft
* 2023-03-02: Add precision for integration tests
* 2023-03-23: Add precision for E2E tests

## Status

Expand Down Expand Up @@ -55,13 +57,11 @@ These are almost like integration tests in that they exercise many things togeth
use mocks.

Example 1 journey vs illustrative tests - [depinject's BDD style tests](https://github.com/cosmos/cosmos-sdk/blob/main/depinject/features/bindings.feature), show how we can
rapidly build up many illustrative cases demonstrating behavioral rules without [very much
code](https://github.com/cosmos/cosmos-sdk/blob/main/depinject/binding_test.go) while maintaining high level readability.
rapidly build up many illustrative cases demonstrating behavioral rules without [very much code](https://github.com/cosmos/cosmos-sdk/blob/main/depinject/binding_test.go) while maintaining high level readability.

Example 2 [depinject table driven tests](https://github.com/cosmos/cosmos-sdk/blob/main/depinject/provider_desc_test.go)

Example 3 [Bank keeper tests](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/bank/keeper/keeper_test.go#L94-L105) - A mock implementation of `AccountKeeper` is
supplied to the keeper constructor.
Example 3 [Bank keeper tests](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/bank/keeper/keeper_test.go#L94-L105) - A mock implementation of `AccountKeeper` is supplied to the keeper constructor.

#### Limitations

Expand Down Expand Up @@ -147,6 +147,9 @@ End to end tests exercise the entire system as we understand it in as close an a
to a production environment as is practical. Presently these tests are located at
[tests/e2e](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e) and rely on [testutil/network](https://github.com/cosmos/cosmos-sdk/tree/main/testutil/network) to start up an in-process Tendermint node.

An application should be built as minimally as possible to exercise the desired functionality.
The SDK uses an application will only the required modules for the tests. The application developer is adviced to use its own application for e2e tests.

#### Limitations

In general the limitations of end to end tests are orchestration and compute cost.
Expand All @@ -162,21 +165,21 @@ The scope of e2e tests has been complected with command line interface testing.

We accept these test scopes and identify the following decisions points for each.

| Scope | App Fixture | Mocks? |
| ----------- | ----------- | ------ |
| Unit | None | Yes |
| Integration | depinject | Some |
| Simulation | depinject | No |
| E2E | simapp | No |
| Scope | App Type | Mocks? |
| ----------- | ------------------- | ------ |
| Unit | None | Yes |
| Integration | integration helpers | Some |
| Simulation | minimal app | No |
| E2E | minimal app | No |

The decision above is valid for the SDK. An application developer should test their application with their full application instead of the minimal app.

### Unit Tests

All modules must have mocked unit test coverage.

Illustrative tests should outnumber journeys in unit tests.

~BDD feature tests are recommended when building up illustrative and journey scenarios.~

Unit tests should outnumber integration tests.

Unit tests must not introduce additional dependencies beyond those already present in
Expand All @@ -200,7 +203,7 @@ Integration tests should outnumber e2e tests.

### Simulations

Simulations shall use `depinject`. They are located under `/x/{moduleName}/simulation`.
Simulations shall use a minimal application (usually via app wiring). They are located under `/x/{moduleName}/simulation`.

### E2E Tests

Expand Down Expand Up @@ -233,7 +236,6 @@ demonstrated in [PR#12706](https://github.com/cosmos/cosmos-sdk/pull/12706).

### Neutral

* learning curve for BDD style tests
* some discovery required for e2e transition to dockertest

## Further Discussions
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/building-modules/11-structure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ x/{module_name}
└── README.md
```

* `client/`: The module's CLI client functionality implementation and the module's integration testing suite.
* `client/`: The module's CLI client functionality implementation and the module's CLI testing suite.
* `exported/`: The module's exported types - typically interface types. If a module relies on keepers from another module, it is expected to receive the keepers as interface contracts through the `expected_keepers.go` file (see below) in order to avoid a direct dependency on the module implementing the keepers. However, these interface contracts can define methods that operate on and/or return types that are specific to the module that is implementing the keepers and this is where `exported/` comes into play. The interface types that are defined in `exported/` use canonical types, allowing for the module to receive the keepers as interface contracts through the `expected_keepers.go` file. This pattern allows for code to remain DRY and also alleviates import cycle chaos.
* `keeper/`: The module's `Keeper` and `MsgServer` implementation.
* `module/`: The module's `AppModule` and `AppModuleBasic` implementation.
Expand Down
42 changes: 12 additions & 30 deletions docs/docs/building-modules/16-testing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,34 +57,16 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/x/gov/keeper/keeper_test.g
Integration tests are at the second level of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
In the SDK, we locate our integration tests under [`/tests/integrations`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/integration).

The goal of these integration tests is to test a component with a minimal application (i.e. not `simapp`). The minimal application is defined with the help of [`depinject`](../tooling/02-depinject.md) – the SDK dependency injection framework, and includes all necessary modules to test the component. With the helps of the SDK testing package, we can easily create a minimal application and start the application with a set of genesis transactions: <https://github.com/cosmos/cosmos-sdk/blob/main/testutil/sims/app_helpers.go>.
The goal of these integration tests is to test how a component interacts with other dependencies. Compared to unit tests, integration tests do not mock dependencies. Instead, they use the direct dependencies of the component. This differs as well from end-to-end tests, which test the component with a full application.

### Example

Here, we will walkthrough the integration tests of the `x/distribution` module. The `x/distribution` module has, in addition to keeper unit tests, integration tests that test the `x/distribution` module with a minimal application. This is expected as you may want to test the `x/distribution` module with actual application logic, instead of only mocked dependencies.

For creating a minimal application, we use [`simtestutil.Setup`](https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/testutil/sims/app_helpers.go#L95-L99) and an [`AppConfig`](../tooling/02-depinject.md) of the `x/distribution` minimal dependencies.

For instance, the `AppConfig` of `x/distribution` is defined as:

* https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/x/distribution/testutil/app_config.go

This is a stripped down version of the `simapp` `AppConfig`:

* https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/simapp/app_config.go

:::note
You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/x/slashing/app_test.go#L54-L62). There no difference between those two ways, use whichever you prefer.
:::
Integration tests interact with the tested module via the defined `Msg` and `Query` services. The result of the test can be verified by checking the state of the application, by checking the emitted events or the response. It is adviced to combine two of these methods to verify the result of the test.

```go reference
https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/tests/integration/distribution/keeper/keeper_test.go#L28-L33
```
The SDK provides small helpers for quickly setting up an integration tests. These helpers can be found at <https://github.com/cosmos/cosmos-sdk/blob/main/testutil/integration>.

Now the types are injected and we can use them for our tests:
### Example

```go reference
https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/tests/integration/distribution/keeper/keeper_test.go#L21-L53
https://github.com/cosmos/cosmos-sdk/blob/29e22b3bdb05353555c8e0b269311bbff7b8deca/testutil/integration/example_test.go#L22-L89
```

## Deterministic and Regression tests
Expand All @@ -106,6 +88,10 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/tests/integration/bank/kee

Simulations uses as well a minimal application, built with [`depinject`](../tooling/02-depinject.md):

:::note
You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/x/slashing/app_test.go#L54-L62). There is no difference between those two ways, use whichever you prefer.
:::

Following is an example for `x/gov/` simulations:

```go reference
Expand All @@ -121,6 +107,7 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.47.0-rc1/x/gov/simulation/operation
End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e).

<!-- @julienrbrt: makes more sense to use an app wired app to have 0 simapp dependencies -->
For that, the SDK is using `simapp` but you should use your own application (`appd`).
Here are some examples:

Expand All @@ -132,11 +119,6 @@ Here are some examples:
The SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/architecture/adr-059-test-scopes.html). This page will eventually be updated with better examples.
:::

## Summary
## Learn More

| Scope | App Fixture | Mocks? |
| ----------- | ----------- | ------ |
| Unit | None | Yes |
| Integration | `depinject` | Some |
| Simulation | `depinject` | No |
| E2E | `appd` | No |
Learn more about testing scope in [ADR-59](https://docs.cosmos.network/main/architecture/adr-059-test-scopes.html).
Loading

0 comments on commit aeaa301

Please sign in to comment.