Fix flaky tests

[colin-grant-work]

Our CI fails a lot for reasons unrelated to the PR being tested. Usually a few flaky tests are the culprits. We recently disabled the TypeScript code lens test because it was so flaky and had such a long timeout, but there are still a few tests that regularly fail for no particular reason.

• tests for closeEditorOnDelete
• tests from the launch-preferences.spec.js suite. This one in particular runs dozens of permutations of the same test with little informational value, but every run has a chance to miss a timeout.

These failures are pretty annoying for the developer, since we either have to just rerun the tests and hope for green or rebase or make some meaningless change and hope for green, and that eats up time and causes PR's to languish while the CI works itself out.

[colin-grant-work]

I'd also add here continued misgivings about the new Playwright tests (@JonasHelming, @planger). À propos a failure in CI, I went to run the tests locally (MacOS) and got numerous failures. I switched to master and still got numerous failures; I did a thorough git clean and deleted ~/.theia and got one fewer failures, but still many:

[playwright]   Slow test file: ../../tests/theia-quick-command.test.ts (4m)
[playwright]   Slow test file: ../../tests/theia-explorer-view.test.ts (2m)
[playwright]   Slow test file: ../../tests/theia-preference-view.test.ts (2m)
[playwright]   Slow test file: ../../tests/theia-problems-view.test.ts (2m)
[playwright]   Consider splitting slow test files to speed up parallel execution
[playwright] 
[playwright]   10 failed
[playwright]     ../../tests/theia-explorer-view.test.ts:46:9 › Theia Explorer View › should be possible to close and reopen it 
[playwright]     ../../tests/theia-explorer-view.test.ts:56:9 › Theia Explorer View › should show one folder named "sampleFolder" and one file named "sample.txt" 
[playwright]     ../../tests/theia-main-menu.test.ts:50:9 › Theia Main Menu › should return menu item by name 'New File' 
[playwright]     ../../tests/theia-preference-view.test.ts:30:9 › Preference View › should be visible and active after being opened 
[playwright]     ../../tests/theia-preference-view.test.ts:37:9 › Preference View › should be able to read, set, and reset String preferences 
[playwright]     ../../tests/theia-problems-view.test.ts:30:9 › Theia Problems View › should be visible and active after being opened 
[playwright]     ../../tests/theia-problems-view.test.ts:37:9 › Theia Problems View › should be opened at the bottom and have the title 'Problems' 
[playwright]     ../../tests/theia-quick-command.test.ts:34:9 › Theia Quick Command › should show quick command palette 
[playwright]     ../../tests/theia-quick-command.test.ts:39:9 › Theia Quick Command › should trigger 'About' command after typing 
[playwright]     ../../tests/theia-quick-command.test.ts:53:9 › Theia Quick Command › should trigger 'Toggle Explorer View' command after typing 
[playwright]   10 skipped
[playwright]   28 passed (11m)

The runtime was quite long - at least the reported 11 min - and tests that can't be run in a variety of environments are not very productive.

[planger]

I think there are two separate issues:

1. The Theia Text Editor > should close without saving also fails in the CI but has only been observed with the Monaco PR so far. I can reproduce this test failure locally, but only with this PR, not on master. Locally it was caused by the Auto-save issue as described above.
2. playwright: tests fail on MacOS #10793 happens on master too, but can so far only be reproduced on MacOS and not in the CI

Or did I misunderstand that?

[planger]

@vince-fugnitto : I believe there are two issues:

1. playwright: tests fail on MacOS #10793 (failing on Mac) also happening on master.
2. One specific test failing on the electron uplift, because the auto save behavior seems to have changed (which is actually an interesting catch)

You were faster @JonasHelming :)

[colin-grant-work]

Re: the Monaco uplift, it is true that the default auto save has changed. VSCode's default differs depending on whether it's running in the browser (in which case autosave is active by default) or in Electron (inactive by default), and I replicated that on the Monaco branch.

An unrelated, and surprising, failure is here: https://github.com/eclipse-theia/theia/runs/5326900058?check_suite_focus=true

I modified the rendering of keybindings, but tests related to the problems widget are failing.

[planger]

An unrelated, and surprising, failure is here: https://github.com/eclipse-theia/theia/runs/5326900058?check_suite_focus=true

This one is indeed surprising. I tried locally but can't reproduce it. I ran the tests 10 times locally with your branch and they didn't fail once. 🤷

[planger]

Re: the Monaco uplift, it is true that the default auto save has changed. VSCode's default differs depending on whether it's running in the browser (in which case autosave is active by default) or in Electron (inactive by default), and I replicated that on the Monaco branch.

We could explicitly set the auto-save preference before the tests to ensure both tests would run consistently for browser and electron (even though we only run them in the browser for now). If you'd agree, I'm happy to create an issue and schedule it for being addressed by us.

[JonasHelming]

@planger Do you think this is actually solved in the meantime?

[planger]

Yes, those particular flaky tests have definitely been fixed in the meantime and also in general the builds are pretty stable again thanks to several changes of the past couple of months.
In the past weeks I only saw theia-explorer-view.test.ts failing once or twice across all test runs. Aside from those everything was constantly green over quite some time. We'll continue observing, but at the moment I'd say there is no particular flakiness.

Edit: PS: This comment is in the context of Playwright

[JonasHelming]

@colin-grant-work Are you fine with setting this to "answered"?