Feature: Track, Monitor and Fix Flaky Tests in the CI

[Sahejkm]

Recently we are seeing flakyness in running e2e tests in the CI, we should have some monitoring / tracking in place for tests which failed in CI in the past runs

ex: #6250 1st run failed https://github.com/facebook/lexical/actions/runs/9371922530, 2nd run passed https://github.com/facebook/lexical/actions/runs/9372503730

1. FLAKY TEST: Links › Can handle pressing Enter at the beginning of a Link | Env: Linux/Chrome/Collab | PR: [Lexical] Skip newly added flaky "Resize merged cells height/width" tests for linux/collab/firefox mode #6250 | PR Upgrade vite version to address vulnerabilities detected by the Open … #6284

2. FLAKY TEST: History › Can type two paragraphs of text and correctly undo and redo | Env: mac (18.18.0, webkit, plain-text, modern-events) | PR: [lexical-eslint-plugin] Fix: Use cjs compatible export from built version #6252 [Lexical] Fix missing meta copyright headers in files  #6258

3. FLAKY TEST: Nested List › remove list breaks when selection in empty nested list item 2 linux (18.18.0, chromium, rich-text, modern-events) PR: [lexical-utils] Fix: mergeRegister should call cleanup functions in reverse order #6260 | [Lexical][Meta] Add dependencies needed to setup lexical-website in m… #6293

• Tracking: Automaticall track CI failures due to flaky test or manually add comments to tests with flaky runs in PRs if this tracking can't be automated
• Try fix flaky tests 1st or else worst case add fixme for specific Environment if the issue is recurring

[etrepum]

I think the hard part here is mostly deciding where these flaky test reports should go, there isn't really a github feature for it so something else would have to be deployed.

[potatowagon]

I think the hard part here is mostly deciding where these flaky test reports should go, there isn't really a github feature for it so something else would have to be deployed.

untested top of mind thought: i could perhaps use github artifacts to upload html reports, and keep the retention such that the total storage used is not significant enough to cross the free storage limit (500MB free). one playright html report is about 500KB

the html report would make debugging tests easier tho looking at the file it is minified html so parsing it in its raw state dont provide the data (test status) we are looking for.

for an overview of the tests' flaky run history across the repo, still have to research how to do that

[etrepum]

Yeah I was thinking about the aggregate, that's the actionable part. People only look at artifacts when something goes wrong in a particular test run, and those expire. We upload the playwright videos as artifacts on every e2e test failure and I suspect those videos almost never get looked at (they are so low res it's not really helpful anyway IMO). I guess a third party service like airtable, supabase, etc. would be fine but you'd need to add a secret to the repo to make it work and ideally expose at least a read-only view of the aggregate report to other maintainers (or publish it on the website somewhere out of the way).

[potatowagon]

4. https://github.com/facebook/lexical/actions/runs/9409582384/job/25919783165

[Sahejkm]

5. https://github.com/facebook/lexical/actions/runs/9444245507/job/26009440131?pr=6284

[etrepum]

https://github.com/facebook/lexical/actions/runs/9457203892/job/26050551481?pr=6291

[potatowagon]

closing as this is tracked under discussions in flaky test tracker