Effect benchmarks

[janmasrovira]

Overview

This pr implements a simple benchmark suite to compare the efficiency of effectful-core and polysemy.

I've implemented the suite with the help of tasty-bench. It is a simple benchmarking library that has minimal dependencies and it can be run with a default main using the same cli options as our tasty test suite.

How to run

stack run juvixbench

If you only want to run a particular benchmark:

stack run juvixbench -- -p "/Output/"

Results

The results show that effectful is the clear winner, in some cases it is extremely close to the raw version.

State

This benchmark adds the first 2 ^ 22 first naturals:

countRaw :: Natural -> Natural
countRaw = go 0
  where
    go :: Natural -> Natural -> Natural
    go acc = \case
      0 -> acc
      m -> go (acc + m) (pred m)

Results:

   State
      Eff State (Static): OK
        25.2 ms ± 2.4 ms
      Sem State:          OK
        2.526 s ± 5.1 ms
      Raw State:          OK
        22.3 ms ± 1.5 ms

Output

This benchmark collects the first 2 ^ 21 naturals in a list and adds them.

countdownRaw :: Natural -> Natural
countdownRaw = sum' . reverse . go []
  where
    go :: [Natural] -> Natural -> [Natural]
    go acc = \case
      0 -> acc
      m -> go (m : acc) (pred m)

Results:

      Eff Output (Dynamic): OK
        693  ms ±  61 ms
      Eff Accum (Static):   OK
        553  ms ±  36 ms
      Sem Output:           OK
        2.606 s ±  91 ms
      Raw Output:           OK
        604  ms ±  26 ms

Reader (First Order)

Repeats a constant in a list and adds it. The effects based version ask the constant value in each iteration.

countRaw :: Natural -> Natural
countRaw = sum' . go []
  where
    go :: [Natural] -> Natural -> [Natural]
    go acc = \case
      0 -> acc
      m -> go (c : acc) (pred m)

Results:

    Reader (First order)
      Eff Reader (Static): OK
        103  ms ± 6.9 ms
      Sem Reader:          OK
        328  ms ±  31 ms
      Raw Reader:          OK
        106  ms ± 1.9 ms

Reader (Higher Order)

Adds the first 2 ^ 21 naturals. The effects based version use local (from the Reader) effect to pass down the argument that counts the iterations.

countRaw :: Natural -> Natural
countRaw = sum' . go []
  where
    go :: [Natural] -> Natural -> [Natural]
    go acc = \case
      0 -> acc
      m -> go (m : acc) (pred m)

Results:

    Reader (Higher order)
      Eff Reader (Static): OK
        720  ms ±  56 ms
      Sem Reader:          OK
        2.094 s ± 182 ms
      Raw Reader:          OK
        154  ms ± 2.2 ms

Embed IO

Opens a temporary file and appends a character to it a number of times.

countRaw :: Natural -> IO ()
countRaw n =
  withSystemTempFile "tmp" $ \_ h -> go h n
  where
    go :: Handle -> Natural -> IO ()
    go h = \case
      0 -> return ()
      a -> hPutChar h c >> go h (pred a)

Results:

   Embed IO
      Raw IO:       OK
        464  ms ±  12 ms
      Eff RIO:      OK
        487  ms ± 3.5 ms
      Sem Embed IO: OK
        582  ms ±  33 ms