Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add invariant testing #760

Merged
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
book
out
cache
.idea
.idea
.vscode
2 changes: 1 addition & 1 deletion src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
- [Advanced Testing](./forge/advanced-testing.md)
- [Fuzz Testing](./forge/fuzz-testing.md)
- [Differential Testing](./forge/differential-ffi-testing.md)
<!-- - [Invariant Testing]() !-->
- [Invariant Testing](./forge/invariant-testing.md)
<!-- - [Symbolic Testing]() !-->
<!-- - [Table Testing]() !-->
<!-- - [Mutation Testing]() !-->
Expand Down
4 changes: 2 additions & 2 deletions src/forge/advanced-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@

Forge comes with a number of advanced testing methods:

- [Fuzz Testing](./fuzz-testing.md)
- [Differential Testing](./differential-ffi-testing.md)
- [Fuzz Testing](./fuzz-testing.md)
- [Invariant Testing](./invariant-testing.md)
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved

In the future, Forge will also support these:

- [Invariant Testing](#)
- [Symbolic Execution](#)
- [Mutation Testing](#)

Expand Down
303 changes: 303 additions & 0 deletions src/forge/invariant-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,303 @@
# Invariant Testing
## Overview

Invariant testing allows for a set of mathematical invariants to be tested against randomized sequences of pre-defined function calls from pre-defined contracts. After each function call is performed, all defined invariants are asserted.

Invariant testing is a powerful tool to expose incorrect logic in protocols. Due to the fact that function call sequences are randomized and have fuzzed inputs, invariant testing can expose false assumptions and incorrect logic in edge cases and highly complex protocol states.

Invariant testing campaigns have two dimensions, `runs` and `depth`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Invariant testing campaigns have two dimensions, `runs` and `depth`.
Invariant testing campaigns have two dimensions, `runs` and `depth`:

- `runs`: Number of times that a sequence of function calls is generated and run.
- `depth`: Number of function calls made in a given `run`. All defined invariants are asserted after each function call is made.
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved

## Defining Invariants

Invariants are mathematical expressions that should always hold true over the course of a fuzzing campaign. A good invariant testing suite should have as many invariants as possible, and can have different testing suites for different protocol states.
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved

There are different ways to assert invariants:

<table>
<tr><th>Type</th><th>Explanation</th><th>Example</th></tr>

<tr>

<td>Direct assertions</td>
<td>Directly query a protocol smart contracts and assert values are expected.</td>
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved
<td>

```solidity
assertGe(
token.totalAssets(),
token.totalSupply()
)
```
</td>

</tr>

<tr>

<td>Ghost variable assertions</td>
<td>Query a protocol smart contract and compare it against a value that has been persisted in the test environment (ghost variable).</td>
<td>

```solidity
assertEq(
token.totalSupply(),
sumBalanceOf
)
```
</td>

</tr>

<tr>

<td>Naive implementation assertions</td>
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved
<td>Query a protocol smart contract function and compare it against a naive and typically highly gas-inefficient implementation of the same desired logic.</td>
<td>

```solidity
assertEq(
pool.outstandingInterest(),
test.naiveInterest()
)
```
</td>

</tr>
</table>

### Conditional Invariants

Invariants must hold for the course over the course of a given fuzzing campaign, but that doesn't mean they must hold true in every situation. There is the possibility for certain invariants to be introduced/removed in a given scenario (e.g., during a liquidation). For this a dedicated testing contract should be used.
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved

It is not recommended to introduce conditional logic into invariant assertions because they have the possibility of introducing false positives because of an incorrect code path. For example:

```solidity
function invariant_example() external {
if (protocolCondition) return;

assertEq(val1, val2);
}
```

In this situation, if `protocolCondition == true`, the invariant is not asserted at all. Sometimes this can be desired behavior, but it can cause issues if the `protocolCondition` is true for the whole fuzzing campaign unexpectedly, or there is a logic error in the condition itself. For this reason its better to try and define an alternative invariant for that condition as well, for example:

```solidity
function invariant_example() external {
if (protocolCondition) {
assertLe(val1, val2);
return;
};

assertEq(val1, val2);
}
```

Another approach to handle different invariants across protocol states is to utilize dedicated invariant testing contracts for different scenarios. These scenarios can be bootstrapped using the `setUp` function, but it is more powerful to leverage target contracts, which are outlined in the next section.

## Target Contracts

Target contracts are the set of contracts that will be called over the course of a given invariant test fuzzing campaign. This set of contracts defaults to all contracts that were deployed in the `setUp` function, but can be customized to allow for more advanced invariant testing.

### Function Call Probability Distribution

Functions from these contracts will be called at random with fuzzed inputs. The probability of a function being called is broken down by contract and then by function.

For example:

```
targetContract1: 50%
├─ function1: 50% (25%)
└─ function2: 50% (25%)

targetContract2: 50%
├─ function1: 25% (12.5%)
├─ function2: 25% (12.5%)
├─ function3: 25% (12.5%)
└─ function4: 25% (12.5%)
```

This is something to be mindful of when designing target contracts, as target contracts with less functions will have each function called more often due to this probability distribution.

### Target Contract Setup

Target contracts can be set up using the following three methods:
1. Contracts that are deployed in the `setUp` function are automatically added to the set of target contracts.
2. Contracts that are added to the the `targetContracts` array are added to the set of target contracts. (TODO: check if this is possible during invariant campaigns yet)
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved
3. Contracts that are deployed in the `setUp` can be **removed** from the target contracts if they are added to the `excludeContracts` array.
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved

## Target Contract Patterns

When running invariant testing, especially against contracts with more complex logic, it is important to consider how the target contracts are used.

### Open Testing

The default configuration for target contracts is set to all contracts that are deployed during the setup. For smaller modules and more arithmetic contracts, this works well. For example:

```solidity
contract ExampleContract1 {

uint256 val1;
uint256 val2;
uint256 val3;

function addToA(uint256 amount) external {
val1 += amount;
val3 += amount;
}

function addToB(uint256 amount) external {
val2 += amount;
val3 += amount;
}

}
```

This contract could be deployed and tested using the default target contract pattern:

```solidity
contract InvariantExample1 {

ExampleContract1 foo;

function setUp() external {
foo = new ExampleContract1();
}

function invariant_A() external {
assertEq(foo.val1() + foo.val2(), foo.val3());
}

function invariant_B() external {
assertGe(foo.val1() + foo.val2(), foo.val1());
}

}
```

This setup will call `foo.addToA()` and `foo.addToB()` with a 50%-50% probability distribution with fuzzed inputs. Inevitably, the inputs will start to cause overflows and the function calls will start reverting. Since the default configuration in invariant testing is `fail_on_revert = true`, this will not cause the tests to fail. The invariants will hold throughout the rest of the fuzzing campaign and the result is that the test will pass. The output will look something like this:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent documentation @lucas-manuel, thank you! I believe the default configuration here is fail_on_revert = false as per the config README which makes more sense in the context of this sentence and the final handler section.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


```
[PASS] invariant_A() (runs: 50, calls: 10000, reverts: 5533)
[PASS] invariant_B() (runs: 50, calls: 10000, reverts: 5533)
```

### Bounded Testing

For more complex and integrated protocols, more sophisticated target contract usage is required to achieve the desired results. For example, a ERC-4626 based contract that accepts deposits of another ERC-20 token:

```solidity
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.7;
lucas-manuel marked this conversation as resolved.
Show resolved Hide resolved

interface IERC20Like {

function balanceOf(address owner_) external view returns (uint256 balance_);

function transferFrom(
address owner_,
address recipient_,
uint256 amount_
) external returns (bool success_);

}

contract Basic4626Deposit {

/**********************************************************************************************/
/*** Storage ***/
/**********************************************************************************************/

address public immutable asset;

string public name;
string public symbol;

uint8 public immutable decimals;

uint256 public totalSupply;

mapping(address => uint256) public balanceOf;

/**********************************************************************************************/
/*** Constructor ***/
/**********************************************************************************************/

constructor(address asset_, string memory name_, string memory symbol_, uint8 decimals_) {
asset = asset_;
name = name_;
symbol = symbol_;
decimals = decimals_;
}

/**********************************************************************************************/
/*** External Functions ***/
/**********************************************************************************************/

function deposit(uint256 assets_, address receiver_) external returns (uint256 shares_) {
shares_ = convertToShares(assets_);

require(receiver_ != address(0), "ZERO_RECEIVER");
require(shares_ != uint256(0), "ZERO_SHARES");
require(assets_ != uint256(0), "ZERO_ASSETS");

totalSupply += shares_;

// Cannot overflow because totalSupply would first overflow in the statement above.
unchecked { balanceOf[receiver_] += shares_; }

require(
IERC20Like(asset).transferFrom(msg.sender, address(this), assets_),
"TRANSFER_FROM"
);
}

function transfer(address recipient_, uint256 amount_) external returns (bool success_) {
balanceOf[msg.sender] -= amount_;

// Cannot overflow because minting prevents overflow of totalSupply,
// and sum of user balances == totalSupply.
unchecked { balanceOf[recipient_] += amount_; }

return true;
}

/**********************************************************************************************/
/*** Public View Functions ***/
/**********************************************************************************************/

function convertToAssets(uint256 shares_) public view returns (uint256 assets_) {
uint256 supply_ = totalSupply; // Cache to stack.

assets_ = supply_ == 0 ? shares_ : (shares_ * totalAssets()) / supply_;
}

function convertToShares(uint256 assets_) public view returns (uint256 shares_) {
uint256 supply_ = totalSupply; // Cache to stack.

shares_ = supply_ == 0 ? assets_ : (assets_ * supply_) / totalAssets();
}

function totalAssets() public view returns (uint256 assets_) {
assets_ = IERC20Like(asset).balanceOf(address(this));
}

}

```

This contract's `deposit` function requires that the caller has a non-zero balance of the ERC-20 `asset`. In the open invariant testing approach, `deposit()` and `transfer()` would be called with a 50-50% distribution, but they would revert on every call. This would cause the invariant tests to "pass", but in reality no state was manipulated in the desired contract at all. This is where target contracts can be leveraged. When a contract requires some additional logic in order to function properly, it can be added in a dedicated contract called a `Handler`.

```solidity
function deposit(uint256 assets) public virtual {
asset.mint(address(this), assets);

asset.approve(address(token), assets);

uint256 shares = token.deposit(assets, address(this));
}
```

This contract will provide the necessary setup before a function call is made in order to ensure it is successful.