Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(e2e): Major refactor and stabilization of e2e tests #7581

Merged
merged 115 commits into from
Aug 7, 2024
Merged

Conversation

unlikelyzero
Copy link
Collaborator

@unlikelyzero unlikelyzero commented Mar 12, 2024

Closes #7307
Closes #7797
Closes #7763

Describe your changes:

  • Upgrades the eslint-playwright-plugin package to v1.5.2
  • Updates the default rules for linting e2e tests to be more strict
  • Removes our custom test fixture for overriding the clock, in favor of Playwright's new clock API. Refactors @clock tests to use the clock API.
  • Various a11y fixes and improvements
  • Refactor of many (not all) raw locators
  • Refactor all instances of page.click() to use Locator.click() instead
  • Refactor all instances of page.dragAndDrop() to use Locator.dragTo(Locator) instead
  • ... and many more!

All Submissions:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Is this a notable change that will require a special callout in the release notes? For example, will this break compatibility with existing APIs or projects that consume these plugins?

Author Checklist

  • Changes address original issue?
  • Tests included and/or updated with changes?
  • Has this been smoke tested?
  • Have you associated this PR with a type: label? Note: this is not necessarily the same as the original issue.
  • Have you associated a milestone with this PR? Note: leave blank if unsure.
  • Is this a breaking change to be called out in the release notes?
  • Testing instructions included in associated issue OR is this a dependency/testcase change?

Reviewer Checklist

  • Changes appear to address issue?
  • Reviewer has tested changes by following the provided instructions?
  • Changes appear not to be breaking changes?
  • Appropriate automated tests included?
  • Code style and in-line documentation are appropriate?

@unlikelyzero unlikelyzero added the type:maintenance tests, chores, or project maintenance label Mar 12, 2024
@unlikelyzero unlikelyzero added this to the Target:4.0.0 milestone Mar 12, 2024
@unlikelyzero unlikelyzero requested a review from ozyx March 12, 2024 02:01
Copy link

codecov bot commented Mar 12, 2024

Codecov Report

Attention: Patch coverage is 85.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 56.84%. Comparing base (e3fcbe1) to head (93f2833).
Report is 1 commits behind head on master.

Files Patch % Lines
src/api/annotation/AnnotationAPI.js 85.00% 3 Missing ⚠️
src/utils/clock/DefaultClock.js 66.66% 2 Missing ⚠️
src/utils/visibility/VisibilityObserver.js 83.33% 2 Missing ⚠️
src/api/composition/CompositionProvider.js 50.00% 1 Missing ⚠️
src/plugins/plot/legend/PlotLegend.vue 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7581      +/-   ##
==========================================
- Coverage   57.15%   56.84%   -0.31%     
==========================================
  Files         674      674              
  Lines       27276    27283       +7     
  Branches     2668     2669       +1     
==========================================
- Hits        15589    15509      -80     
- Misses      11347    11435      +88     
+ Partials      340      339       -1     
Flag Coverage Δ
e2e-ci 61.02% <84.90%> (?)
e2e-full 23.52% <22.64%> (-18.46%) ⬇️
e2e-stable ?
unit 49.38% <67.79%> (-0.05%) ⬇️
Files Coverage Δ
example/exampleUser/ExampleUserProvider.js 83.60% <ø> (ø)
example/generator/SinewaveStalenessProvider.js 100.00% <ø> (ø)
openmct.js 100.00% <100.00%> (ø)
src/MCT.js 96.93% <ø> (ø)
src/api/Editor.js 90.90% <ø> (ø)
src/api/actions/ActionCollection.js 83.58% <ø> (ø)
src/api/actions/ActionsAPI.js 96.15% <100.00%> (ø)
src/api/composition/CompositionAPI.js 100.00% <ø> (ø)
src/api/composition/CompositionCollection.js 93.58% <ø> (ø)
src/api/composition/DefaultCompositionProvider.js 95.45% <ø> (ø)
... and 111 more

... and 7 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3fcbe1...93f2833. Read the comment docs.

ozyx
ozyx previously requested changes Mar 12, 2024
Copy link
Contributor

@ozyx ozyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got some comments.

Do you think we should keep some of the repeat offenders as warnings for now and fix them over time?

.eslintrc.cjs Show resolved Hide resolved
.eslintrc.cjs Show resolved Hide resolved
Comment on lines +401 to +407
1. Use [user-facing locators](https://playwright.dev/docs/best-practices#use-locators) (Now a eslint rule!)

```js
page.getByRole('button', { name: 'Create' } )
```
Instead of
```js
page.locator('.c-create-button')
```
Note: `page.locator()` can be used in performance tests as xk6-browser does not yet support the new `page.getBy` pattern and css lookups can be [1.5x faster](https://serpapi.com/blog/css-selectors-faster-than-getbyrole-playwright/)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion:

  1. Prefer using user-facing locators for better readability and maintainability:
// Recommended
page.getByRole('button', { name: 'Create' })

Instead of the less intuitive:

// Not recommended
page.locator('.c-create-button')

Note: In performance tests, page.locator() may still be used as xk6-browser lacks support for the page.getBy pattern. Additionally, CSS selectors can be up to 1.5x faster than role-based selectors.

e2e/tests/performance/tagging.perf.spec.js Show resolved Hide resolved
@@ -152,7 +152,7 @@ test.describe('ExportAsJSON Disabled Actions', () => {
test.describe('ExportAsJSON ProgressBar @couchdb', () => {
let folder;
test.beforeEach(async ({ page }) => {
await page.goto('./', { waitUntil: 'networkidle' });
await page.goto('./', { waitUntil: 'domcontentloaded' });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i worry about this breaking the couchdb tests

.eslintrc.cjs Outdated Show resolved Hide resolved
src/ui/layout/MctTree.vue Outdated Show resolved Hide resolved
@unlikelyzero unlikelyzero added the pr:e2e:flakefinder Runs the tests 10 times label Aug 7, 2024
Copy link
Collaborator Author

@unlikelyzero unlikelyzero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 thing to fix and we need to talk about the Aria label on gauges

await page.getByLabel('Edit Object').click();

// Expand the 'My Items' folder in the left tree
page.click('button[title="Show selected item in tree"]');
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫡

@@ -281,29 +281,3 @@ test.describe('Global Time Conductor', () => {
// select an option and verify the offsets are updated correctly
});
});

test.describe('Time Conductor History', () => {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this totally gone?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's no longer relevant because you can't set milliseconds in the time conductor anymore. The time conductor history menu is still there, though. Maybe we should keep the suite

src/plugins/gauge/components/GaugeComponent.vue Outdated Show resolved Hide resolved

return `${this.expanded ? 'Collapse' : 'Expand'}${name}${type}`;
return `${this.expanded ? 'Collapse' : 'Expand'}${name} Plot Series Options`;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@ozyx ozyx changed the title [e2e] Update playwright eslint plugin [e2e] Major refactor and stabilization of e2e tests Aug 7, 2024
@ozyx ozyx changed the title [e2e] Major refactor and stabilization of e2e tests test(e2e): Major refactor and stabilization of e2e tests Aug 7, 2024
@ozyx ozyx added the pr:e2e:couchdb npm run test:e2e:couchdb label Aug 7, 2024
@github-actions github-actions bot removed the pr:e2e:couchdb npm run test:e2e:couchdb label Aug 7, 2024
@ozyx ozyx added the pr:e2e:couchdb npm run test:e2e:couchdb label Aug 7, 2024
@github-actions github-actions bot removed the pr:e2e:couchdb npm run test:e2e:couchdb label Aug 7, 2024
Copy link
Contributor

@ozyx ozyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👀 😏

@ozyx ozyx added the pr:e2e:couchdb npm run test:e2e:couchdb label Aug 7, 2024
@ozyx ozyx enabled auto-merge (squash) August 7, 2024 21:35
@github-actions github-actions bot removed the pr:e2e:couchdb npm run test:e2e:couchdb label Aug 7, 2024
@ozyx ozyx merged commit 0413e77 into master Aug 7, 2024
21 checks passed
@ozyx ozyx deleted the eslint_update branch August 7, 2024 21:36
@akhenry
Copy link
Contributor

akhenry commented Oct 2, 2024

This PR touches runtime code and needs test instructions in the associated issue(s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
notable_change A change which should be noted in the changelog pr:e2e:flakefinder Runs the tests 10 times type:maintenance tests, chores, or project maintenance
Projects
None yet
3 participants