Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: coverage for libraries #7510

Merged
merged 10 commits into from
Mar 29, 2024
Merged

fix: coverage for libraries #7510

merged 10 commits into from
Mar 29, 2024

Conversation

klkvr
Copy link
Member

@klkvr klkvr commented Mar 28, 2024

Motivation

Closes #7054
Closes #2567
Closes #4854
Closes #4654

Solution

Currently we're collecting anchors via the following steps:

  1. Walk the AST for each contract definition to find coverage items contained in it along with its base contracts and potentially libraries.
  2. Collect mapping from contract_id to coverage items contained in it itself or its base contracts/libraries.
  3. Try to find anchors for each coverage item found in step 2.

This approach requires us to be able to reliably find libraries used by a contract which turned out to be non-trivial.

One possible approach could be to reuse some of #6936 code to walk through typed AST and find all contracts/libs used by a given contract. However, this might be breaking due to typed AST not working for some solc versions and this approach is also not perfect.

Instead, we now just collect all coverage items and then create a mapping from source_id to all coverage items given source contains.

After that, when looking for anchors, we only try finding anchors for items from source ids appearing in source map. This allows us to get rid of dependency resolution through AST.

Example

I was mainly testing those on Uniswap/v4-core contracts.

Report via latest nightly: report_nightly.txt
Report via this PR: report_pr.txt

diff:

9,11c9,11
< | src/libraries/BitMath.sol                  | 0.00% (0/47)      | 0.00% (0/57)       | 0.00% (0/36)     | 0.00% (0/2)      |
< | src/libraries/FullMath.sol                 | 0.00% (0/29)      | 0.00% (0/33)       | 0.00% (0/8)      | 0.00% (0/2)      |
< | src/libraries/Hooks.sol                    | 0.00% (0/41)      | 0.00% (0/76)       | 0.00% (0/28)     | 0.00% (0/14)     |
---
> | src/libraries/BitMath.sol                  | 100.00% (47/47)   | 100.00% (57/57)    | 100.00% (36/36)  | 100.00% (2/2)    |
> | src/libraries/FullMath.sol                 | 100.00% (29/29)   | 100.00% (33/33)    | 100.00% (8/8)    | 100.00% (2/2)    |
> | src/libraries/Hooks.sol                    | 100.00% (41/41)   | 98.68% (75/76)     | 96.43% (27/28)   | 100.00% (14/14)  |
14,23c14,23
< | src/libraries/Pool.sol                     | 0.00% (0/130)     | 0.00% (0/141)      | 0.00% (0/86)     | 0.00% (0/13)     |
< | src/libraries/PoolGetters.sol              | 0.00% (0/2)       | 0.00% (0/2)        | 100.00% (0/0)    | 0.00% (0/2)      |
< | src/libraries/Position.sol                 | 0.00% (0/12)      | 0.00% (0/14)       | 0.00% (0/6)      | 0.00% (0/2)      |
< | src/libraries/SafeCast.sol                 | 0.00% (0/8)       | 0.00% (0/14)       | 0.00% (0/8)      | 0.00% (0/4)      |
< | src/libraries/SqrtPriceMath.sol            | 0.00% (0/32)      | 0.00% (0/66)       | 0.00% (0/24)     | 0.00% (0/8)      |
< | src/libraries/SwapFeeLibrary.sol           | 0.00% (0/5)       | 0.00% (0/9)        | 0.00% (0/4)      | 0.00% (0/3)      |
< | src/libraries/SwapMath.sol                 | 0.00% (0/23)      | 0.00% (0/30)       | 0.00% (0/12)     | 0.00% (0/1)      |
< | src/libraries/TickBitmap.sol               | 0.00% (0/19)      | 0.00% (0/34)       | 0.00% (0/6)      | 0.00% (0/3)      |
< | src/libraries/TickMath.sol                 | 97.87% (92/94)    | 92.57% (137/148)   | 82.61% (38/46)   | 50.00% (2/4)     |
< | src/libraries/UnsafeMath.sol               | 0.00% (0/1)       | 0.00% (0/1)        | 100.00% (0/0)    | 0.00% (0/1)      |
---
> | src/libraries/Pool.sol                     | 96.92% (126/130)  | 94.33% (133/141)   | 89.53% (77/86)   | 100.00% (13/13)  |
> | src/libraries/PoolGetters.sol              | 100.00% (2/2)     | 100.00% (2/2)      | 100.00% (0/0)    | 100.00% (2/2)    |
> | src/libraries/Position.sol                 | 100.00% (12/12)   | 100.00% (14/14)    | 100.00% (6/6)    | 100.00% (2/2)    |
> | src/libraries/SafeCast.sol                 | 100.00% (8/8)     | 100.00% (14/14)    | 100.00% (8/8)    | 100.00% (4/4)    |
> | src/libraries/SqrtPriceMath.sol            | 100.00% (32/32)   | 98.48% (65/66)     | 83.33% (20/24)   | 100.00% (8/8)    |
> | src/libraries/SwapFeeLibrary.sol           | 100.00% (5/5)     | 100.00% (9/9)      | 100.00% (4/4)    | 100.00% (3/3)    |
> | src/libraries/SwapMath.sol                 | 100.00% (23/23)   | 100.00% (30/30)    | 100.00% (12/12)  | 100.00% (1/1)    |
> | src/libraries/TickBitmap.sol               | 100.00% (19/19)   | 100.00% (34/34)    | 100.00% (6/6)    | 100.00% (3/3)    |
> | src/libraries/TickMath.sol                 | 97.87% (92/94)    | 97.30% (144/148)   | 97.83% (45/46)   | 50.00% (2/4)     |
> | src/libraries/UnsafeMath.sol               | 100.00% (1/1)     | 100.00% (1/1)      | 100.00% (0/0)    | 100.00% (1/1)    |
51,53c51,53
< | src/types/BalanceDelta.sol                 | 0.00% (0/2)       | 0.00% (0/2)        | 100.00% (0/0)    | 0.00% (0/2)      |
< | src/types/Currency.sol                     | 0.00% (0/15)      | 0.00% (0/24)       | 0.00% (0/10)     | 0.00% (0/6)      |
< | src/types/PoolId.sol                       | 0.00% (0/1)       | 0.00% (0/2)        | 100.00% (0/0)    | 0.00% (0/1)      |
---
> | src/types/BalanceDelta.sol                 | 100.00% (2/2)     | 100.00% (2/2)      | 100.00% (0/0)    | 100.00% (2/2)    |
> | src/types/Currency.sol                     | 100.00% (15/15)   | 91.67% (22/24)     | 80.00% (8/10)    | 100.00% (6/6)    |
> | src/types/PoolId.sol                       | 100.00% (1/1)     | 100.00% (2/2)      | 100.00% (0/0)    | 100.00% (1/1)    |
55c55
< | test/utils/Deployers.sol                   | 0.00% (0/50)      | 0.00% (0/63)       | 0.00% (0/10)     | 0.00% (0/11)     |
---
> | test/utils/Deployers.sol                   | 96.00% (48/50)    | 93.65% (59/63)     | 80.00% (8/10)    | 90.91% (10/11)   |
58,59c58,59
< | test/utils/SortTokens.sol                  | 0.00% (0/3)       | 0.00% (0/5)        | 0.00% (0/2)      | 0.00% (0/1)      |
< | Total                                      | 43.42% (554/1276) | 42.89% (745/1737)  | 22.24% (153/688) | 40.35% (115/285) |
---
> | test/utils/SortTokens.sol                  | 66.67% (2/3)      | 80.00% (4/5)       | 50.00% (1/2)     | 100.00% (1/1)    |
> | Total                                      | 75.78% (967/1276) | 75.30% (1308/1737) | 55.38% (381/688) | 66.67% (190/285) |

cc @onbjerg

@klkvr klkvr changed the title fix: coverage for external libraries fix: coverage for libraries Mar 28, 2024
Copy link
Member

@onbjerg onbjerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great overall, really stoked, an initial q tho

// end up here again
if items.is_empty() || is_test {
self.contract_items.insert(contract_id.clone(), Vec::new());
let ContractVisitor { items, .. } = ContractVisitor::new(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

w/o the if !self.contract_items.contains_key(contract_id) { check, will this not lead to re-analyzing contracts? and possibly duping coverage items?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're walking only ContractDefinition nodes from self.contracts, so it should be fine as long as there is no duplicating ContractId keys

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think thats a fair assumption

@onbjerg onbjerg added T-bug Type: bug Cmd-forge-coverage Command: forge coverage labels Mar 28, 2024
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love to see it.

only a few style nits, defer to @onbjerg

is there a few to add a few sanity tests for this? probably difficult?

crates/evm/coverage/src/analysis.rs Outdated Show resolved Hide resolved
crates/forge/bin/cmd/coverage.rs Outdated Show resolved Hide resolved
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, pending @onbjerg

.contract_items

// Build helper mapping used by `find_anchors`
let mut items_by_source_id = HashMap::new();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let mut items_by_source_id = HashMap::new();
let mut items_by_source_id = HashMap::with_capacity(source_analysis.items.len());

let mut items_by_source_id = HashMap::new();

for (item_id, item) in source_analysis.items.iter().enumerate() {
items_by_source_id.entry(item.loc.source_id).or_insert_with(Vec::new).push(item_id);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be or_default() I believe

@klkvr klkvr merged commit 452956f into master Mar 29, 2024
19 checks passed
@klkvr klkvr deleted the klkvr/coverage-libraries branch March 29, 2024 13:34
ZhangZhuoSJTU pushed a commit to MEDGA-eth/foundry that referenced this pull request Mar 29, 2024
* fix: coverage for internal libraries

* optimize

* optimize

* doc

* rm tests

* clippy

* clippy + fmt

* clean up

* for loop

* review fixes
@frontier159
Copy link
Contributor

This sounds great, thanks @klkvr

It seems to have a regression for different --report arguments though:

  1. If --report summary is explicitly used, it shows zero coverage for everything (should be the same as not specifying according to the --help)
  2. If --report lcov is used, it shows a lot less coverage than before the change

FWIW, I run via this flow to get a html representation of the coverage and branches:

forge coverage --report summary --report lcov
lcov --remove lcov.info 'contracts/test/*' 'test/*' --output-file lcov.info --rc lcov_branch_coverage=1
genhtml lcov.info -o report --branch-coverage --legend

@klkvr
Copy link
Member Author

klkvr commented Mar 31, 2024

@frontier159 Thank you for reporting and really sorry for breaking those. Any chance you could share a repository on which this can be reproduced?

@frontier159
Copy link
Contributor

frontier159 commented Mar 31, 2024

@klkvr it seems after a forge clean it works, but then running a second time gives near zero coverage.

You can try this repo:

  1. clone with submodules
  2. cd apps/protocol
  3. nvm use
  4. yarn
  5. yarn forge-coverage-html

After a yarn clean && forge clean gives the correct coverage. But then running yarn forge-coverage-html a second time (without a clean) gives almost zero coverage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Cmd-forge-coverage Command: forge coverage T-bug Type: bug
Projects
None yet
4 participants