feat(test): only compile files needed for tests #7334

klkvr · 2024-03-07T18:54:13Z

Motivation

Compile contracts requesting only abi output and filter out abis which match given filters or just start with test/invariant

This allowed to move some checks to be performed earlier ("No tests match the provided pattern")

This might be breaking for some codebases using vm.getDeployedCode as we won't recompile contracts which are not appearing as dependecy of test contract, however if contract being fetched through vm.getDeployedCode is imported from test file, then it will still be recompiled and I believe that it's a common pattern used, for example, in optimism codebase (https://github.com/ethereum-optimism/optimism/blob/53573e0ea6a807a125784cc5c7df07cbb4dbe3bc/packages/contracts-bedrock/scripts/Deployer.sol#L25)

Closes #7045

cc @mds1

mds1 · 2024-03-07T19:47:23Z

Awesome, will test this!

This might be breaking for some codebases using vm.getDeployedCode as we won't recompile contracts which are not appearing as dependecy of test contract, however if contract being fetched through vm.getDeployedCode is imported from test file, then it will still be recompiled

Good callout. Is a potential fix here for forge to error if it sees a contract name in getDeployedCode that is not seen as an import? This would have to be optional though, as such a safety check is not always applicable (e.g. vyper contracts compiled in a prior step). Having that safety check as opt-out would probably be the conservative default

klkvr · 2024-03-07T19:54:27Z

Good callout. Is a potential fix here for forge to error if it sees a contract name in getDeployedCode that is not seen as an import? This would have to be optional though, as such a safety check is not always applicable (e.g. vyper contracts compiled in a prior step). Having that safety check as opt-out would probably be the conservative default

Oh, that's a nice idea. Yeah, we can manage the list of contracts which were compiled before current test run and only allow getDeployedCode invocations for freshly compiled/cached contracts.

I don't think that this will cause issues with Vyper: afaik people are using ffi to compile and get bytecode of Vyper contract (like here, for example: https://github.com/pcaversaccio/snekmate/blob/main/lib/utils/VyperDeployer.sol#L53-L80)

mds1 · 2024-03-07T20:03:57Z

I still think there are cases where it might cause issues, like seaport, where they use one profile to compile, then another to test: https://github.com/ProjectOpenSea/seaport?tab=readme-ov-file#foundry-tests (cc @emo-eth to confirm since you used to work on seaport)

mattsse

+1 on @mds1 suggestion

this is now always invoked?
the overhead should be minimal?
perhaps we can skip this if there's no filter?

crates/forge/bin/cmd/test/mod.rs

mattsse · 2024-03-09T11:26:15Z

crates/forge/bin/cmd/test/mod.rs

+        project.solc_config.settings.output_selection.0 = BTreeMap::from([(
+            "*".to_string(),
+            BTreeMap::from([("*".to_string(), vec!["abi".to_string()])]),
+        )]);


I think there's an easier way to do this, if not we should make this easier in foundry-compilers

klkvr · 2024-03-09T13:39:34Z

After #87 we'll spend additional time only obtaining abi for dirty sources. If there's no filters we will filеer out contracts which don't have any invariant/test functions which reduces the full compilation time and is reasonable imo

This additional step results in ~4-5% build time increase for normal builds and in <1% increase for via-ir builds, so it should worth it

klkvr · 2024-03-10T09:15:00Z

I've added logic for runtime validation of getCode and getDeployedCode inputs. We are now collecting ArtifactIds after compilation and find match across them depending on user input.

This should make those cheatcodes impl more stable as we won't make any assumptions on artifact location.

I've also added ability to pass artifacts in the format of vm.getCode("src/Counter.sol:Counter:0.X.Y") which should resolve and provide a clear API for #7045.

This check can be disabled with unchecked_cheatcode_artifacts = true config flag (perhaps there is a better name for this?)

Ref foundry-rs/foundry#7334 (comment) with that helper it will be `OutputSelection::common_output_selection(["abi".to_string()])`

onbjerg

looks good to me, remember to close #7045 (and other issues if you find some) with this pr

klkvr · 2024-03-12T16:37:19Z

Some checks are failing after compilers update, working on fixing those

mattsse

amazing

mds1 · 2024-03-15T21:46:04Z

Testing on optimism codebase using a test suite that is pretty self contained and does not require most of the codebase:

Before this PR

Using 9f6bb3b

$ time forge t --mt testFuzz_handleCall_initiation_succeeds                    
[⠒] Compiling...
[⠑] Compiling 1 files with 0.5.17
[⠘] Compiling 16 files with 0.8.19
[⠃] Compiling 122 files with 0.8.23
[⠊] Compiling 347 files with 0.8.15
[⠘] Solc 0.5.17 finished in 48.71ms
[⠒] Solc 0.8.19 finished in 723.05ms
[⠒] Solc 0.8.23 finished in 4.66s
[⠒] Solc 0.8.15 finished in 163.61s
Compiler run successful!

Ran 1 test for test/L1/DelayedVetoable.t.sol:DelayedVetoable_HandleCall_Test
[PASS] testFuzz_handleCall_initiation_succeeds(bytes) (runs: 64, μ: 41080, ~: 40741)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 4.80ms (4.25ms CPU time)

Ran 1 test suite in 142.81ms (4.80ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
forge t --mt testFuzz_handleCall_initiation_succeeds  168.29s user 2.35s system 102% cpu 2:46.42 total

After this PR

Using e758bd2, looks like it could not find the test

time forge t --mt testFuzz_handleCall_initiation_succeeds
No tests match the provided pattern:
        match-test: `testFuzz_handleCall_initiation_succeeds`
Error: 
No tests to run
forge t --mt testFuzz_handleCall_initiation_succeeds  0.04s user 0.07s system 65% cpu 0.182 total

klkvr · 2024-03-15T23:32:31Z

@mds1 can't really reproduce this failure locally, getting:

$ time forge t --mt testFuzz_handleCall_initiation_succeeds
[⠒] Compiling...
[⠆] Compiling 23 files with 0.8.15
[⠔] Solc 0.8.15 finished in 2.24s
Compiler run successful!

Ran 1 test for test/L1/DelayedVetoable.t.sol:DelayedVetoable_HandleCall_Test
[PASS] testFuzz_handleCall_initiation_succeeds(bytes) (runs: 64, μ: 41133, ~: 40746)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 23.72ms (20.42ms CPU time)

Ran 1 test suite in 163.19ms (23.72ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
forge t --mt   7.58s user 0.54s system 114% cpu 7.092 total

can you check if issue still persists with purged cache and share logs with RUST_LOG=foundry_compilers if so?

mds1 · 2024-03-17T22:01:36Z

Confirmed it is working now 👍

time forge t --mt testFuzz_handleCall_initiation_succeeds 
[⠒] Compiling...
[⠢] Compiling 23 files with 0.8.15
[⠆] Solc 0.8.15 finished in 2.11s
Compiler run successful!

Ran 1 test for test/L1/DelayedVetoable.t.sol:DelayedVetoable_HandleCall_Test
[PASS] testFuzz_handleCall_initiation_succeeds(bytes) (runs: 64, μ: 41089, ~: 40736)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 5.68ms (4.36ms CPU time)

Ran 1 test suite in 137.69ms (5.68ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
forge t --mt testFuzz_handleCall_initiation_succeeds  6.54s user 0.45s system 111% cpu 6.296 total

* feat(forge test): only compile files needed for tests * remove comment * clippy * update fixtures * getCode + getDeployedCode updates * fixes * fix path matching * clippy * add config flag * fix * docs * fmt * patch compilers * fix Cargo.toml * update patch * update patch * doc * rm space * cargo cheats * new output selection fn * log compiler errors on failure * fixes

* chore: make cast use an alloy provider * move initial methods to alloy * feat(`foundry-common`): NameOrAddress ENS util (#7122) * feat(foundry-common): NameOrAddress ENS util * chore: rename err * chore: remove from impl for str * chore: unrelated fix from alloy upgrade * nit * feat(`cast`): Move non `tx` methods to alloy (#7129) * chore: add alloy contract * feat(cast): migrate most methods to alloy * chore: leave todo for converting a tx envelope into an rpc tx * fix: use proper type for storage * readd decodetx for now * chore: extend txbuilder to build an alloy tx request * feat: migrate most methods bar send/decode raw tx * fix: include tx data * simplify txbuilder * chore: simplify back access_list * chore: remove unnecesary conversion * fmt * doctests * fmt * do not use trait * Update crates/cast/bin/main.rs Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> * cleanup builder * clippy * fix doc comments --------- Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> * DocumentMut * wip * wip * wip: bump alloy * wip * wip * wip * [wip] migrate to alloy providers and signers (#7425) wip * fix wallets after alloy bump * clean up deps * use serde on consensus types * update TypedTransaction for anvil * make anvil compile * wip: make script compile * fix script * make forge compile * fix: anvil tests * bump alloy * fix tests * fix tx builder * fix cargo.toml * fix cargo.toml * fix script gas price logic * remove ethers from anvil * clippy * rm all_derives * deps * fmt * fix tests * configure clippy * clippy * add feature * fix cargo deny * fix persist * fix doctests * fmt * fix clap * review fixes * fmt * bump alloy * Update cargo.toml * fmt * fixes * ethers clean-up * fix(fmt): fix indent closing parenthesis enclosed in { } (#7557) * fix(fmt): fix indent closing parenthesis enclosed in { } * Fix testdata bad formatting * feat(test): only compile files needed for tests (#7334) * feat(forge test): only compile files needed for tests * remove comment * clippy * update fixtures * getCode + getDeployedCode updates * fixes * fix path matching * clippy * add config flag * fix * docs * fmt * patch compilers * fix Cargo.toml * update patch * update patch * doc * rm space * cargo cheats * new output selection fn * log compiler errors on failure * fixes * fix: do not flood dictionary with data dependent on fuzz inputs (#7552) * fix dictionary * clippy + fmt * fix * Feat: Index cheatcode for Strings (#7539) * feat: index cheatcode * some nits to make it work * nit: use as_str() * final changes * chore: reviewed changes * chore: reduce logs in tests (#7566) * fix(script): decode custom error in script fail message (#7563) * clippy * bump alloy * AnyNetwork * bump alloy * add comment * clippy * bump alloy * fixes * refactor cast logs to use alloy (#7594) * refactor cast logs to use alloy * fmt * make clippy happy * cleanup * doc nits --------- Co-authored-by: evalir <hi@enriqueortiz.dev> --------- Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> Co-authored-by: Arsenii Kulikov <klkvrr@gmail.com> Co-authored-by: grandizzy <38490174+grandizzy@users.noreply.github.com> Co-authored-by: Krishang <93703995+kamuik16@users.noreply.github.com> Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com> Co-authored-by: bernard-wagner <bernard-wagner@users.noreply.github.com>

klkvr requested review from DaniPopes, Evalir and mattsse as code owners March 7, 2024 18:54

mattsse requested changes Mar 9, 2024

View reviewed changes

klkvr force-pushed the klkvr/selective-compilation-test branch from 8da2d53 to dadce48 Compare March 10, 2024 08:40

klkvr mentioned this pull request Mar 10, 2024

helper for OutputSelection foundry-rs/compilers#89

Merged

mattsse pushed a commit to foundry-rs/compilers that referenced this pull request Mar 10, 2024

helper for OutputSelection (#89)

7f670d5

Ref foundry-rs/foundry#7334 (comment) with that helper it will be `OutputSelection::common_output_selection(["abi".to_string()])`

onbjerg approved these changes Mar 12, 2024

View reviewed changes

klkvr added 13 commits March 12, 2024 16:59

feat(forge test): only compile files needed for tests

cdf724e

remove comment

66301ca

clippy

67d7f3c

update fixtures

465557b

getCode + getDeployedCode updates

f368431

fixes

51f2c94

fix path matching

5fd3b46

clippy

edad2eb

add config flag

4687bde

fix

04e6bc1

docs

f04ead2

fmt

afcf743

patch compilers

5c4e204

klkvr force-pushed the klkvr/selective-compilation-test branch from ec2063d to 5c4e204 Compare March 12, 2024 13:02

fix Cargo.toml

229bf5c

klkvr mentioned this pull request Mar 12, 2024

refactor: caching logic foundry-rs/compilers#90

Merged

update patch

93f2f3b

klkvr added 5 commits March 13, 2024 05:16

update patch

9581f68

doc

a5d661f

rm space

b3f61dd

cargo cheats

e8f40e5

new output selection fn

aa0b303

mattsse approved these changes Mar 13, 2024

View reviewed changes

log compiler errors on failure

e758bd2

klkvr added 3 commits March 30, 2024 19:53

Merge branch 'master' into klkvr/selective-compilation-test

4dc6196

Merge branch 'master' into klkvr/selective-compilation-test

4ec4319

fixes

6dd972f

klkvr merged commit c10f32a into foundry-rs:master Apr 4, 2024
20 checks passed

klkvr mentioned this pull request Apr 5, 2024

vm.getCode doesn't find artifacts by name #7569

Closed

2 tasks

sakulstra mentioned this pull request Apr 5, 2024

forge no longer finds test #7578

Closed

2 tasks

klkvr mentioned this pull request Apr 5, 2024

fix: --match-path is broken #7579

Merged

This was referenced Apr 8, 2024

fix: smarter getCode validation #7597

Merged

feat(forge): compile only the test files matched via --match-test #5041

Closed

deployCode/getCode no longer works with different profiles #7607

Closed

This was referenced Apr 26, 2024

Reset to upstream phylaxsystems/phoundry#12

Merged

feat/forge bin as lib phylaxsystems/phoundry#13

Merged

klkvr mentioned this pull request Jun 2, 2024

Forge compiles all test contracts when specified to only use one contract. #2066

Closed

2 tasks

adu-web3 mentioned this pull request Jul 4, 2024

chore: use latest version of foundry tools ExocoreNetwork/exocore-contracts#43

Merged

2 tasks

meetmangukiya mentioned this pull request Sep 11, 2024

bug: getCode/getDeployedCode wont find a valid artifact #8842

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(test): only compile files needed for tests #7334

feat(test): only compile files needed for tests #7334

klkvr commented Mar 7, 2024 •

edited

Loading

mds1 commented Mar 7, 2024

klkvr commented Mar 7, 2024

mds1 commented Mar 7, 2024

mattsse left a comment

mattsse Mar 9, 2024

klkvr commented Mar 9, 2024 •

edited

Loading

klkvr commented Mar 10, 2024 •

edited

Loading

onbjerg left a comment

klkvr commented Mar 12, 2024

mattsse left a comment

mds1 commented Mar 15, 2024

klkvr commented Mar 15, 2024

mds1 commented Mar 17, 2024

feat(test): only compile files needed for tests #7334

feat(test): only compile files needed for tests #7334

Conversation

klkvr commented Mar 7, 2024 • edited Loading

Motivation

mds1 commented Mar 7, 2024

klkvr commented Mar 7, 2024

mds1 commented Mar 7, 2024

mattsse left a comment

Choose a reason for hiding this comment

mattsse Mar 9, 2024

Choose a reason for hiding this comment

klkvr commented Mar 9, 2024 • edited Loading

klkvr commented Mar 10, 2024 • edited Loading

onbjerg left a comment

Choose a reason for hiding this comment

klkvr commented Mar 12, 2024

mattsse left a comment

Choose a reason for hiding this comment

mds1 commented Mar 15, 2024

Before this PR

After this PR

klkvr commented Mar 15, 2024

mds1 commented Mar 17, 2024

klkvr commented Mar 7, 2024 •

edited

Loading

klkvr commented Mar 9, 2024 •

edited

Loading

klkvr commented Mar 10, 2024 •

edited

Loading