Image-Based Unit Tests Fail on Mac

[jwedel]

Expected behavior

Unit tests should be green.

Current behavior

Pulling the latest master and run pnpm test I get this:

[karma] Chrome 126.0.0.0 (Mac OS 10.15.7) Time scale tests auto /base/test/fixtures/scale.timeseries/data-timestamps.js FAILED
[karma]         Fixture test failed:
[karma]           Difference: 1166px / 0.44%
[karma]           Threshold: 1%
[karma]           Tolerance: 0.15%
[karma]         Error: Fixture test failed:
[karma]           Difference: 1166px / 0.44%
[karma]           Threshold: 1%
[karma]           Tolerance: 0.15%
[karma]             at <Jasmine>
[karma]             at node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:1086:25 <- test/index.js:1091:27
[karma]             at image.onload (node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:527:5 <- test/index.js:532:7)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7) Time scale tests auto /base/test/fixtures/scale.timeseries/data-timestamps.js FAILED
[karma]         Fixture test failed:
[karma]           Difference: 1166px / 0.44%
[karma]           Threshold: 1%
[karma]           Tolerance: 0.15%
[karma]         Error: Fixture test failed:
[karma]           Difference: 1166px / 0.44%
[karma]           Threshold: 1%
[karma]           Tolerance: 0.15%
[karma]             at <Jasmine>
[karma]             at node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:1086:25 <- test/index.js:1091:27
[karma]             at image.onload (node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:527:5 <- test/index.js:532:7)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7): Randomized with seed 00514
[karma] 
[karma] Firefox 115.0 (Mac OS 10.15): Executed 50 of 1649 (1 FAILED) (2.982 secs / 2.85 secs)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7): Executed 1009 of 1649 (1 FAILED) (9.784 secs / 8.73 secs)
[karma] TOTAL: 2 FAILED, 1057 SUCCESS

We also use image-based tests for our Chart.js use case and had similar issues.

The reason was, that each OS uses different fonts. So on Windows, Linux and Mac, the Chart.Js fonts for scale lables etc. look slightly different.

Reproducible sample

---

Optional extra steps/info to reproduce

No response

Possible solution

We actually "solved" this by making Linux the default and starting the tests in a Linux docker container on other environments to get the same result.

Allowing an error threshold >0% for Chart.js is not a good idea as this causes false positives when e.g. a 1px line in a line chart has the wrong color.

Context

No response

chart.js version

v4.4.3

Browser name and version

No response

Link to your project

No response