Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hspec's around doesn't wrap shrunk quickcheck tests #413

Open
ChickenProp opened this issue Jul 19, 2024 · 0 comments
Open

hspec's around doesn't wrap shrunk quickcheck tests #413

ChickenProp opened this issue Jul 19, 2024 · 0 comments

Comments

@ChickenProp
Copy link

At first I thought this was a bug in hspec, I opened an issue at hspec/hspec#899. But as things currently stand I'm not sure how hspec could fix it without changes to quickcheck. So I'm posting here too to try to get more eyes.

Short version is that hspec has an around function that lets you wrap every run of a quickcheck test. But if the test fails and shrinks, it doesn't wrap the shrunk attempts. For example:

  let hook act = do
        putStrLn "before"
        act
        putStrLn "after"
  around_ hook $ it "xxx" $ do
    let shr = \case
          2 -> [1, 0]
          _ -> []
    forAllShrink getSize shr $ \someInt -> ioProperty $ do
      putStrLn $ "Testing " <> show someInt
      someInt `shouldNotBe` 2
before     
Testing 0
after
before
Testing 1
after
before
Testing 2
after
Testing 1
Testing 0

We have two successful tests that get wrapped, a failing test that gets wrapped, then two shrinks of the failing test that don't get wrapped.

The way this works is that hspec examples can be of type a -> Property, and hooks can provide the a. a -> Property is essentially a -> QCGen -> Int -> Rose Result, where the root Result is the result of the first test and children are the results of any shrinks. (Even if shrinking is unnecessary, every shrink is in the tree, though not actually evaluated.) But that type can't do what we want, since we have a single a to construct the whole tree. Currently it essentially calls hook $ \a -> reduceRose (r a), which means the function passed to the hook

  • Does the IO necessary to get the root Result of the tree
  • Does the IO necessary to get the IO necessary to get any immediate child Results of the tree
  • Does not get any child Results of the tree.

Which gives us the behavior observed, that the hook wraps the initial test but not any shrinks.

I don't see any way to get around this with the types as they are. Switching to QCGen -> Int -> Rose (a -> IO Result) might work, but it feels like that would need large changes to both quickcheck and hspec, and I can't rule out that it would fail for some other reason.

Any suggestions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant