Manually add concrete examples to test

[paldepind]

Consider the following scenario: Through usage, a bug is discovered in a function for some concrete input. In order to test the bug and potentially shrink the input a property based test is created. But, it turns out that none of the random values that the property is tried with encounters the bug. For instance, let's say that a function fails when given a prime number but none of the random numbers generated happens to be prime? What does one do in such a case?

It seems to me that it would be nice if it was possible to specify some concrete examples for assert that it would then always try. Something like this:

fc.assert(
  fc.property(fc.tuple(fc.nat(), fc.string()), ...),
  {
    examples: [
      [ 28374, "foobarbaz" ]
    ]
  }
);

The above is just a way to combine the random generated input with some hardcoded input that is known to trigger or have triggered an error.

What do you think about the above? I just ran in to a problem similar to the above. As of right now, I've just duplicated the test with the values I know doesn't work hardcoded into it.

[dubzzz]

Hi,

There is indeed a need for such feature. When I first introduced fast-check during a meetup, some people told me about this nice example feature. Basically the Python framework Hypothesis has this feature.

I would definitely be interested in having such feature in fast-check. It should not be a big deal to add it.

The points we need to discuss are: How should it interact with other settings of assert? Should we be able to shrink it?

1. For instance, given the example above, do we consider that the framework runs 2 hard-coded examples followed by 98 random (2+98=100, the default number of tests).
   me: yes

2. Concerning the shrink: it might be really a challenge as we do not know how the example value was computed.
   me: not for a first version

_. Can't we think of something letting the user referencing past seed+path as examples. Just an idea, but it might be a problem if we want to change something in a shrinker or generator.
me: not ideal for enhancements of shrink...

[paldepind]

For instance, given the example above, do we consider that the framework runs 2 hard-coded examples followed by 98 random (2+98=100, the default number of tests).
me: yes

That sounds fine to me. The difference between the two isn't that significant and both would be fine for me personally.

Concerning the shrink: it might be really a challenge as we do not know how the example value was computed.
me: not for a first version

I think using the default shrinking mechanism would be very useful. Shrinking is actually half the reason why I think this would be useful.

When I discover a bug in a project what I usually is to create a test that exposes the bug. In the test, I often use the concrete data that made the error occur in practice. The data that caused the bug may be arbitrary and it may be easy to shrink. If the example got shrunken that would in many cases make the bug easier to find and fix.

[dubzzz]

I think I will first try with a version with no shrink capabilities and then try to implement one having this capability.

Shrinking such entry is indeed a very complex stuff. Basically whenever destructuring the object is hard, there is a challenge. Here are some examples where it can get really challenging to start from an example and somehow shrink it:

• Your arbitrary is a fc.oneof. How, given only an example, can you find back the arbitrary that might have produced this entry?

• Your arbitrary is the result of a map operation. In this case, without the knowledge of an unmapper function there is no way to get back to the original value, shrink it and reapply the mapper. This could be solved by adding an optional unmapper parameter to map.

• Your arbitrary is the result of a chain. In this case I am not sure the unmapper trick can work.

Anyway, I have the very same need for another feature I am currently working on: adding model based testing capabilities to fast-check in order to check complex systems and ui. So I think I will quickly have to tackle this issue in order to have at least a partial coverage.

[dubzzz]

@paldepind I just commited a first version of the concrete examples. I will try to add the support for shrinking in coming versions. For the moment, the examples are executed first followed by the generated values.

If you want to use it quickly, I can release a new minor in coming days ;)

[paldepind]

I can definitely see how shrinking can be really difficult 😅 My own use case at hand was just two random numbers so I didn't think of how complex supporting that in the general case would be.

adding model based testing capabilities to fast-check in order to check complex systems and ui.

That sounds really exciting.

I just commited a first version of the concrete examples. I will try to add the support for shrinking in coming versions. For the moment, the examples are executed first followed by the generated values.

Amazing 🎉 I can't wait to try it myself.

I have to say, I've enjoyed using fast-check thus far. Thank you and props for the really great work you've been doing on this impressive library 👍

[dubzzz]

The feature of concrete examples has been released with fast-check@1.4.0.

Looking forward for your feedbacks on it.

I will work on improving it by adding shrinker to it later. This feature is really a good idea :)

[dubzzz]

@paldepind Is the feature OK for your needs? Can we close this issue?

[paldepind]

Yes. I've been using it in my projects and it has been working great 👍 Thanks a lot for adding the feature.

[dubzzz]

Can I close the issue?

[dubzzz]

@paldepind Closing the issue for the moment, no shrink on examples planned for the moment but I keep the idea in mind. Please do not hesitate to re-open if you really need it

[dubzzz]

@paldepind Next minor release will include shrinking capabilities on examples 🥳

See #2391