Run benchmark against develop branch during PR

[Friendseeker]

Issue

zinc-benchmark result varies greatly between different CI runs. Therefore, to identify how a PR affects zinc performance, cannot simply read zinc-benchmark CI results from latest develop commit and PR commit to cross-compare two results.

Changes

We can run zinc-benchmark against develop branch in the same CI run. This way, both results are from the same machine, hence can be cross-compared.

Validating the PR

The CI for this PR shows the following two benchmark results

[info] Benchmark                                                (_tempDir)  (zincEnabled)    Mode  Cnt           Score          Error   Units
[info] HotScalacBenchmark.action                         /tmp/sbt_70981d01           true  sample    5       11999.065 ±      239.119   ms/op
[info] HotScalacBenchmark.action:gc.alloc.rate           /tmp/sbt_70981d01           true  sample    5         250.219 ±        5.020  MB/sec
[info] HotScalacBenchmark.action:gc.alloc.rate.norm      /tmp/sbt_70981d01           true  sample    5  3153949251.200 ± 10577976.123    B/op
[info] HotScalacBenchmark.action:gc.count                /tmp/sbt_70981d01           true  sample    5         105.000                 counts
[info] HotScalacBenchmark.action:gc.time                 /tmp/sbt_70981d01           true  sample    5        3105.000                     ms
[info] HotScalacBenchmark.action:p0.00                   /tmp/sbt_70981d01           true  sample            11895.046                  ms/op
[info] HotScalacBenchmark.action:p0.50                   /tmp/sbt_70981d01           true  sample            12012.487                  ms/op
[info] HotScalacBenchmark.action:p0.90                   /tmp/sbt_70981d01           true  sample            12062.818                  ms/op
[info] HotScalacBenchmark.action:p0.95                   /tmp/sbt_70981d01           true  sample            12062.818                  ms/op
[info] HotScalacBenchmark.action:p0.99                   /tmp/sbt_70981d01           true  sample            12062.818                  ms/op
[info] HotScalacBenchmark.action:p0.999                  /tmp/sbt_70981d01           true  sample            12062.818                  ms/op
[info] HotScalacBenchmark.action:p0.9999                 /tmp/sbt_70981d01           true  sample            12062.818                  ms/op
[info] HotScalacBenchmark.action:p1.00                   /tmp/sbt_70981d01           true  sample            12062.818                  ms/op
[info] HotShapelessBenchmark.action                      /tmp/sbt_70981d01           true  sample   10        5920.680 ±      119.988   ms/op
[info] HotShapelessBenchmark.action:gc.alloc.rate        /tmp/sbt_70981d01           true  sample    5         260.354 ±        3.556  MB/sec
[info] HotShapelessBenchmark.action:gc.alloc.rate.norm   /tmp/sbt_70981d01           true  sample    5  1618077778.400 ±  6072595.654    B/op
[info] HotShapelessBenchmark.action:gc.count             /tmp/sbt_70981d01           true  sample    5         113.000                 counts
[info] HotShapelessBenchmark.action:gc.time              /tmp/sbt_70981d01           true  sample    5        3574.000                     ms
[info] HotShapelessBenchmark.action:p0.00                /tmp/sbt_70981d01           true  sample             5771.362                  ms/op
[info] HotShapelessBenchmark.action:p0.50                /tmp/sbt_70981d01           true  sample             5909.774                  ms/op
[info] HotShapelessBenchmark.action:p0.90                /tmp/sbt_70981d01           true  sample             6036.442                  ms/op
[info] HotShapelessBenchmark.action:p0.95                /tmp/sbt_70981d01           true  sample             6039.798                  ms/op
[info] HotShapelessBenchmark.action:p0.99                /tmp/sbt_70981d01           true  sample             6039.798                  ms/op
[info] HotShapelessBenchmark.action:p0.999               /tmp/sbt_70981d01           true  sample             6039.798                  ms/op
[info] HotShapelessBenchmark.action:p0.9999              /tmp/sbt_70981d01           true  sample             6039.798                  ms/op
[info] HotShapelessBenchmark.action:p1.00                /tmp/sbt_70981d01           true  sample             6039.798                  ms/op
[info] ColdScalacBenchmark.action                        /tmp/sbt_70981d01           true      ss    5       40940.788 ±     1686.444   ms/op
[info] ColdScalacBenchmark.action:gc.alloc.rate          /tmp/sbt_70981d01           true      ss    5          82.731 ±        3.546  MB/sec
[info] ColdScalacBenchmark.action:gc.alloc.rate.norm     /tmp/sbt_70981d01           true      ss    5  3610734454.400 ± 27419144.854    B/op
[info] ColdScalacBenchmark.action:gc.count               /tmp/sbt_70981d01           true      ss    5         286.000                 counts
[info] ColdScalacBenchmark.action:gc.time                /tmp/sbt_70981d01           true      ss    5        4829.000                     ms
[info] ColdShapelessBenchmark.action                     /tmp/sbt_70981d01           true      ss    5       28315.024 ±     1038.779   ms/op
[info] ColdShapelessBenchmark.action:gc.alloc.rate       /tmp/sbt_70981d01           true      ss    5          65.592 ±        2.766  MB/sec
[info] ColdShapelessBenchmark.action:gc.alloc.rate.norm  /tmp/sbt_70981d01           true      ss    5  1986163083.200 ± 24376219.027    B/op
[info] ColdShapelessBenchmark.action:gc.count            /tmp/sbt_70981d01           true      ss    5         244.000                 counts
[info] ColdShapelessBenchmark.action:gc.time             /tmp/sbt_70981d01           true      ss    5        3238.000                     ms
[success] Total time: 794 s (13:14), completed Dec 31, 2023, 2:08:11 AM

[info] Benchmark                                                (_tempDir)  (zincEnabled)    Mode  Cnt           Score          Error   Units
[info] HotScalacBenchmark.action                         /tmp/sbt_b7ada20e           true  sample    5       12509.092 ±      294.635   ms/op
[info] HotScalacBenchmark.action:gc.alloc.rate           /tmp/sbt_b7ada20e           true  sample    5         240.157 ±        6.191  MB/sec
[info] HotScalacBenchmark.action:gc.alloc.rate.norm      /tmp/sbt_b7ada20e           true  sample    5  3154012417.600 ±  2988068.443    B/op
[info] HotScalacBenchmark.action:gc.count                /tmp/sbt_b7ada20e           true  sample    5         100.000                 counts
[info] HotScalacBenchmark.action:gc.time                 /tmp/sbt_b7ada20e           true  sample    5        3001.000                     ms
[info] HotScalacBenchmark.action:p0.00                   /tmp/sbt_b7ada20e           true  sample            12448.694                  ms/op
[info] HotScalacBenchmark.action:p0.50                   /tmp/sbt_b7ada20e           true  sample            12465.471                  ms/op
[info] HotScalacBenchmark.action:p0.90                   /tmp/sbt_b7ada20e           true  sample            12633.244                  ms/op
[info] HotScalacBenchmark.action:p0.95                   /tmp/sbt_b7ada20e           true  sample            12633.244                  ms/op
[info] HotScalacBenchmark.action:p0.99                   /tmp/sbt_b7ada20e           true  sample            12633.244                  ms/op
[info] HotScalacBenchmark.action:p0.999                  /tmp/sbt_b7ada20e           true  sample            12633.244                  ms/op
[info] HotScalacBenchmark.action:p0.9999                 /tmp/sbt_b7ada20e           true  sample            12633.244                  ms/op
[info] HotScalacBenchmark.action:p1.00                   /tmp/sbt_b7ada20e           true  sample            12633.244                  ms/op
[info] HotShapelessBenchmark.action                      /tmp/sbt_b7ada20e           true  sample   10        6049.864 ±       70.941   ms/op
[info] HotShapelessBenchmark.action:gc.alloc.rate        /tmp/sbt_b7ada20e           true  sample    5         253.837 ±        5.629  MB/sec
[info] HotShapelessBenchmark.action:gc.alloc.rate.norm   /tmp/sbt_b7ada20e           true  sample    5  1612303818.400 ±  6708296.678    B/op
[info] HotShapelessBenchmark.action:gc.count             /tmp/sbt_b7ada20e           true  sample    5         109.000                 counts
[info] HotShapelessBenchmark.action:gc.time              /tmp/sbt_b7ada20e           true  sample    5        3474.000                     ms
[info] HotShapelessBenchmark.action:p0.00                /tmp/sbt_b7ada20e           true  sample             5955.912                  ms/op
[info] HotShapelessBenchmark.action:p0.50                /tmp/sbt_b7ada20e           true  sample             6052.381                  ms/op
[info] HotShapelessBenchmark.action:p0.90                /tmp/sbt_b7ada20e           true  sample             6121.167                  ms/op
[info] HotShapelessBenchmark.action:p0.95                /tmp/sbt_b7ada20e           true  sample             6123.684                  ms/op
[info] HotShapelessBenchmark.action:p0.99                /tmp/sbt_b7ada20e           true  sample             6123.684                  ms/op
[info] HotShapelessBenchmark.action:p0.999               /tmp/sbt_b7ada20e           true  sample             6123.684                  ms/op
[info] HotShapelessBenchmark.action:p0.9999              /tmp/sbt_b7ada20e           true  sample             6123.684                  ms/op
[info] HotShapelessBenchmark.action:p1.00                /tmp/sbt_b7ada20e           true  sample             6123.684                  ms/op
[info] ColdScalacBenchmark.action                        /tmp/sbt_b7ada20e           true      ss    5       40602.136 ±     1572.176   ms/op
[info] ColdScalacBenchmark.action:gc.alloc.rate          /tmp/sbt_b7ada20e           true      ss    5          83.292 ±        3.151  MB/sec
[info] ColdScalacBenchmark.action:gc.alloc.rate.norm     /tmp/sbt_b7ada20e           true      ss    5  3604907076.800 ± 11699684.661    B/op
[info] ColdScalacBenchmark.action:gc.count               /tmp/sbt_b7ada20e           true      ss    5         289.000                 counts
[info] ColdScalacBenchmark.action:gc.time                /tmp/sbt_b7ada20e           true      ss    5        4700.000                     ms
[info] ColdShapelessBenchmark.action                     /tmp/sbt_b7ada20e           true      ss    5       28378.696 ±     1437.995   ms/op
[info] ColdShapelessBenchmark.action:gc.alloc.rate       /tmp/sbt_b7ada20e           true      ss    5          65.365 ±        3.185  MB/sec
[info] ColdShapelessBenchmark.action:gc.alloc.rate.norm  /tmp/sbt_b7ada20e           true      ss    5  1985263536.000 ±  9961427.713    B/op
[info] ColdShapelessBenchmark.action:gc.count            /tmp/sbt_b7ada20e           true      ss    5         205.000                 counts
[info] ColdShapelessBenchmark.action:gc.time             /tmp/sbt_b7ada20e           true      ss    5        3120.000                     ms
[success] Total time: 822 s (13:42), completed Dec 31, 2023, 2:24:41 AM

The two runs have very similar number, as desired.

[eed3si9n]

Nice!