Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

custom stopping criteria for tests #256

Open
ghilesZ opened this issue Jul 22, 2022 · 1 comment
Open

custom stopping criteria for tests #256

ghilesZ opened this issue Jul 22, 2022 · 1 comment

Comments

@ghilesZ
Copy link
Contributor

ghilesZ commented Jul 22, 2022

Hi,

I can't find a way in the documentation to stop a test when a given criterion is met.
Is there an way to do that using the current API?

Ideally i would like to have an additional parameter to the function Test.make*, let say a predicate ?until: () -> bool, that would be called at each iteration, an if it returns true, stop the testing. This could be useful in many cases, eg, if the function under test has a behavior that depends on an external environment, or if we want to test a function until its coverage is more than a threshold etc.

The Test.make function would have as a signature something like:

val make : 
  ?if_assumptions_fail:([ `Fatal | `Warning ] * float) ->
  ?count:int ->
  ?long_factor:int ->
  ?max_gen:int ->
  ?max_fail:int ->
  ?small:( 'a -> int ) ->
  ?retries:int ->
  ?name:string ->
  ?until:(unit -> bool)
  'a arbitrary ->
  ( 'a -> bool ) ->
  t

I am willing to participate to the development of such a feature if you think it's a good idea.

@jmid
Copy link
Collaborator

jmid commented Aug 18, 2022

Here's a couple of half-baked ideas to achieve something like that:

  • By raising a suitable exception upon meeting the criteria and wrapping a try-catch of around Test.check_* to catch the test error
  • By expressing the criteria as a precondition with assume (or ==> - but beware it evaluates both sides eagerly in CBV OCaml). This approach will have the drawback of repeatedly trying to satisfy the criteria before finally giving up.

Interface-wise, rather than an impure function unit -> bool I suppose one could also extend QCheck with a dedicated exception Stop_criteria_met and a handler and message for that particular exception in the test driver loop. That should be mostly free for QCheck users who don't use the feature.

From 10.000 ft, If I'm interested in maximizing coverage, rather than extending a blackbox PBT framework, I would probably recommend a "graybox" PBT framework such as Crowbar: https://github.com/stedolan/crowbar. It has a similar QuickCheck-like interface, but underneath the hood the AFL fuzz engine takes coverage into account when generating input test data.
As such, I would prefer to better understand a concrete QCheck use case first 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants