feat(test): add fuzz tests failure persistence

[grandizzy]

Motivation

closes #2551

Solution

• use proptest FileFailurePersistence to create Fuzz TestRunner with (for invariant tests custom implementation is needed as we already replay and shrink the sequence, will be done with feat: fuzz corpus saving and replay in standard format #2552 )
• default persistence file is ~/.foundry/cache/fuzz/failures
• both dir and file are configurable to make switching between different working directories / files handy
• output dir can be changed by setting failure_persist_dir in toml
• files can be customized by passing --fuzz-input-file arg, by setting failure_persist_file in toml file or per test by using forge-config: default.fuzz.failure-persist-file inline config

TBD:
had to remove Copy from config as is required for file and dir options...
couple of tests failing in CI but passing locally

[mds1]

Should we persist the data in the project's local ./cache folder instead? I haven't looked at exactly what's in the file, but if the fuzz data is unique to the project we probably should use the local cache. cc @klkvr

[klkvr]

Yeah, this should definitely be somewhere in project's root. Let's write to ./{cache_path}/fuzz by default.

[grandizzy]

oh, right, I pushed a commit to write in project root dir

[grandizzy]

Should we persist the data in the project's local ./cache folder instead? I haven't looked at exactly what's in the file, but if the fuzz data is unique to the project we probably should use the local cache. cc @klkvr

yeah, the fuzz data is unique per fuzzed test

[mds1]

Am I correct that this means that by default, each run overwrites the previous? If so I think that's good, to prevent the cache from silently getting too large. We should also make sure an env var args are supported so it's easy to set a one-off name or one-off input file without modifying the foundry.toml

[grandizzy]

Re file persistence:
when Proptest finds a failing test case it writes the case in file and then subsequent runs will first replay those test cases before generating new inputs. So the file is not overwritten and that ensures there's no regression.
Re file size - I don't think this can grow too much considering that

• once a test fails you have to fix the case in order to get new inputs and possible failures / new lines written in persisted file
• one can use the inline config to redirect failure cases in different files

Re env var: there's FOUNDRY_FUZZ_FAILURE_PERSIST_FILE that can be used to specify the input file

[klkvr]

Shouldn't it be also possible to enable this for invariants by generating first call + seed for next calls by invariant strategy?

[mattsse]

ty!

pending @klkvr

test failures should be fixed on master

[grandizzy]

Shouldn't it be also possible to enable this for invariants by generating first call + seed for next calls by invariant strategy?

Yeah, I got to the conclusion that the best way is to have #2552 implemented for invariants, reasons being

• proptest's TestRunner generates new seed for each failed case and in invariant case this doesn't help as we need same seed for a sequence https://github.com/proptest-rs/proptest/blob/master/proptest/src/test_runner/runner.rs#L610-L611
• another problem is that it writes into the file only if test run results in an error and in case of invariants there's an entire sequence for failing the test https://github.com/proptest-rs/proptest/blob/master/proptest/src/test_runner/runner.rs#L621-L638
  I initially looked into implementing the FailurePersistence trait but from what I can tell the entire TestRunner run_in_process_with_replay logic needs to be custom implemented and I think we want to avoid this...
• would be great to have something (easily custommizable) to generate and replay a standard corpus (like suggested in feat: fuzz corpus saving and replay in standard format #2552 (comment)). I think this is also easier to get done as we already have the sequence of calls replayed and shrinked when the test fails (without the need to pass to proptest for replaying) and from here we can implement different traits like writting the corpus and / or generating solidity regression tests