Fix UI-tests running locally

[brichet]

Description

When running UI tests locally, it often fails because the screenshots do not match.

It seems to be mostly due to the font used locally and in the CI.

Reproduce

Run the ui-tests locally.

Expected behavior

The tests to pass.

Proposed solution

Remove the snapshot containing text as much as possible

[dlqqq]

I've run into this same issue when maintaining Jupyter AI. Because all of our CI runners are configured to use Linux, I cannot update the snapshots or reproduce any E2E test failures locally.

Are there any objections to only running E2E tests on macOS? This would essentially invert the problem: macOS contributors would be able to update snapshots locally, while Linux contributors would have to rely on the GitHub bot. One might argue that we should have tests for multiple platforms, but then nobody will be able to update snapshots or run E2E tests locally. Choosing macOS seems like the "lesser evil" here.

[brichet]

I'm using on Linux, so this may not be the only issue here.
Maybe there is a way to set up the tests to use the same font locally and in the CI ?

[krassowski]

Are there any objections to only running E2E tests on macOS?

macOS cannot be run on virtual machines unless on a physical mac devices which cost $$$ (even more so in developing countries than what you can imagine, for example see https://news.ycombinator.com/item?id=30610454). That is a steep contribution barrier. On the other hand, macOS users can run Linux as VM for free.

Maybe there is a way to set up the tests to use the same font locally and in the CI ?

Yes. There have been previous attempts at it, either directly by forcing a specific font, or indirectly by forcing a specific docker image (which is even more reproducible in general).

See:

• Enforcing fonts for galata jupyterlab#11682
• Use docker on CI to speed it up jupyterlab#15916

If someone could pick up one or the other this would benefit all users of galata.

More broadly, a recent release of Ubuntu changed implementation of fonts (the same font with the same name now renders differently), see jupyterlab/jupyterlab#16820 (comment) so even hard-coding a font name does not guarantee reproducibility across OS versions.