Update to new Regression Performance Testing on 'SSoelvsten/BDD-Benchmark'

[SSoelvsten]

Finally, my CI is independent of the example/ folder. This allows following up on #555 .

[github-actions]

🟢 Regression Test (QBF 'breakthrough/3x5_11_bwnib')

'origin/ci/update-regression' is a change in performance of 1.09% (stdev: 1.04%).

...	origin/main	origin/ci/update-regression
Mean	13009.00	12867.40
Standard Deviation	135.33	45.52

Number of samples: 5

[github-actions]

🟡 Regression Test (QBF 'breakthrough/3x4_19_bwnib')

'origin/ci/update-regression' is a change in performance of -0.23% (stdev: 0.62%).

...	origin/main	origin/ci/update-regression
Mean	46847.20	46955.20
Standard Deviation	288.40	277.09

Number of samples: 5

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (fe56517) 96.951% compared to head (147dc2f) 96.952%.

❗ Current head 147dc2f differs from pull request most recent head a65f421. Consider uploading reports for the commit a65f421 to get more accurate results

Additional details and impacted files

@@              Coverage Diff              @@
##              main      #582       +/-   ##
=============================================
+ Coverage   96.951%   96.952%   +0.001%     
=============================================
  Files           84        84               
  Lines         5871      5872        +1     
=============================================
+ Hits          5692      5693        +1     
  Misses         179       179               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

🔴 Regression Test (QBF 'breakthrough/3x5_11_bwnib')

'origin/ci/update-regression' is a change in performance of -1.15% (stdev: 0.80%).

...	origin/main	origin/ci/update-regression
Mean	12467.40	12611.00
Standard Deviation	96.27	101.11

Number of samples: 5

[github-actions]

🔴 Regression Test (Picotrav 'adder')

'origin/ci/update-regression' is a change in performance of -14.23% (stdev: 10.95%).

...	origin/main	origin/ci/update-regression
Mean	9264.40	10582.80
Standard Deviation	75.46	1158.89

Number of samples: 5

[github-actions]

🟢 Regression Test (QBF 'ep/8x8_7_e-8-1_p-3-4_bwnib')

'origin/ci/update-regression' is a change in performance of 0.65% (stdev: 3.41%).

...	origin/main	origin/ci/update-regression
Mean	16970.20	16860.40
Standard Deviation	579.42	40.38

Number of samples: 5

[github-actions]

🟢 Regression Test (QBF 'breakthrough/3x5_11_bwnib')

'origin/ci/update-regression' is a change in performance of 0.51% (stdev: 1.84%).

...	origin/main	origin/ci/update-regression
Mean	12893.00	12826.80
Standard Deviation	236.84	42.38

Number of samples: 5

[github-actions]

🟢 Regression Test (QBF 'ep/8x8_7_e-8-1_p-3-4_bwnib')

'origin/ci/update-regression' is a change in performance of 1.92% (stdev: 3.49%).

...	origin/main	origin/ci/update-regression
Mean	17017.20	16690.00
Standard Deviation	593.39	63.87

Number of samples: 5

[github-actions]

🟢 Regression Test (Picotrav 'adder')

'origin/ci/update-regression' is a change in performance of 2.54% (stdev: 7.89%).

...	origin/main	origin/ci/update-regression
Mean	9545.70	9303.40
Standard Deviation	753.37	35.49

Number of samples: 10

[github-actions]

🟢 Regression Test (QBF 'domineering/5x5_13_bwnib')

'origin/ci/update-regression' is a change in performance of 1.28% (stdev: 3.21%).

...	origin/main	origin/ci/update-regression
Mean	42322.40	41780.80
Standard Deviation	1357.49	58.57

Number of samples: 5

[github-actions]

🟢 Regression Test (QBF 'breakthrough/3x4_19_bwnib')

'origin/ci/update-regression' is a change in performance of 0.06% (stdev: 0.65%).

...	origin/main	origin/ci/update-regression
Mean	46859.00	46832.60
Standard Deviation	302.84	271.70

Number of samples: 5

[github-actions]

🟢 Regression Test (QBF 'breakthrough/3x5_11_bwnib')

'origin/ci/update-regression' is a change in performance of 0.52% (stdev: 1.38%).

...	origin/main	origin/ci/update-regression
Mean	12610.80	12545.80
Standard Deviation	173.59	47.77

Number of samples: 5

[github-actions]

🟢 Regression Test (QBF 'ep/8x8_7_e-8-1_p-3-4_bwnib')

'origin/ci/update-regression' is a change in performance of 1.26% (stdev: 3.65%).

...	origin/main	origin/ci/update-regression
Mean	16686.40	16475.80
Standard Deviation	609.79	77.33

Number of samples: 5

[github-actions]

🟡 Regression Test (Picotrav 'adder')

'origin/ci/update-regression' is a change in performance of -2.65% (stdev: 11.39%).

...	origin/main	origin/ci/update-regression
Mean	9605.70	9860.00
Standard Deviation	834.52	1122.77

Number of samples: 10

[github-actions]

🟢 Regression Test (QBF 'domineering/5x5_13_bwnib')

'origin/ci/update-regression' is a change in performance of 1.31% (stdev: 3.14%).

...	origin/main	origin/ci/update-regression
Mean	42493.80	41938.20
Standard Deviation	1332.39	67.97

Number of samples: 5

[github-actions]

🟡 Regression Test (QBF 'breakthrough/3x4_19_bwnib')

'origin/ci/update-regression' is a change in performance of -0.01% (stdev: 0.98%).

...	origin/main	origin/ci/update-regression
Mean	46727.60	46734.60
Standard Deviation	459.90	191.80

Number of samples: 5

[github-actions]

🟡 Regression Test (Picotrav 'arbiter')

'origin/ci/update-regression' is a change in performance of -0.18% (stdev: 0.56%).

...	origin/main	origin/ci/update-regression
Mean	62767.90	62882.30
Standard Deviation	348.72	185.13

Number of samples: 10

[github-actions]

🟡 Regression Test (Picotrav 'mem_ctrl')

'origin/ci/update-regression' is a change in performance of -0.09% (stdev: 0.50%).

...	origin/main	origin/ci/update-regression
Mean	111453.43	111549.29
Standard Deviation	551.73	151.70

Number of samples: 7