Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up test jail doc & skip current flaky tests #6774

Merged
merged 5 commits into from
Nov 30, 2023
Merged

Set up test jail doc & skip current flaky tests #6774

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5bb2a87
Set up test jail doc with current inmates
DejayJD Nov 22, 2023
a81ea0b
Tweak the doc
DejayJD Nov 22, 2023
b87f73f
Put more tests in flaky jail 😭
DejayJD Nov 22, 2023
366eaa1
Update packages/probers/cypress/e2e/flaky-test-jail.md
DejayJD Nov 27, 2023
68e6016
add ticket for test jail
DejayJD Nov 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added packages/probers/cypress/e2e/flaky test meme.jpeg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
34 changes: 34 additions & 0 deletions packages/probers/cypress/e2e/flaky-test-jail.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Flaky Test Jail

![flaky test meme](./flaky%20test%20meme.jpeg)

A list of flaky/misbehaving tests that we are currently skipping in order to keep a green baseline

## Current Inmates

#### [Upload Track](./uploadTrack.cy.ts)
Copy link
Contributor Author

@DejayJD DejayJD Nov 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make a client team ticket to fix the upload track suite?


This entire suite is in jail.

The upload flow seems to generally take too long in CI and results in timeouts and consistent failures.

#### [Sign Up](./signUp.cy.ts) - [C-3403](https://linear.app/audius/issue/C-3403/get-signup-e2e-green)

`"should create an account (mobile/desktop)"` are in jail because the new sign up flow is still a WIP and this test + the code needs more work before going green again

`"can navigate to sign-up from ...xyz (mobile/desktop)"` are in jail due to a bug
with the sign up modal where it just shows the waves background and no content [separate bug ticket](https://linear.app/audius/issue/C-3401/sign-up-sometimes-shows-waves-and-no-content)

<!-- Template

#### [Test Name](./link-to-test-file.cy.ts) - [ticket-number](ticket-link)

Description of which tests and why they're in flaky test jail

-->

## Test Jail Process

- Create an appropriate ticket for addressing the test
- Update this doc with the flaky tests and a description
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is necessary if we have the ticket tbh, or if we do want a description, the comment could have it

- Add a TODO: comment by the test with the ticket number for searchability
16 changes: 10 additions & 6 deletions packages/probers/cypress/e2e/signUp.cy.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ describe('Sign Up', () => {
})

context('desktop', () => {
it('can navigate to signup from trending', () => {
// TODO: IN TEST JAIL [C-3403] - Failing due to "waves bg but no content" issue
it.skip('can navigate to signup from trending', () => {
cy.visit('trending')
cy.findByText(/have an account\?/i).should('exist')
cy.findByRole('link', { name: /sign up/i }).click()
Expand All @@ -54,7 +55,8 @@ describe('Sign Up', () => {
assertOnSignUpPage()
})

it('can navigate to sign-up from sign-in', () => {
// TODO: IN TEST JAIL [C-3403] - Failing due to "waves bg but no content" issue
it.skip('can navigate to sign-up from sign-in', () => {
cy.visit('signin')
cy.findByRole('link', { name: /create an account/i }).click()

Expand All @@ -67,7 +69,7 @@ describe('Sign Up', () => {
assertOnSignUpPage()
})

// TODO: Un-skip when full account creation flow is set up
// TODO: IN TEST JAIL [C-3403] turn back on when full account creation flow is set up
it.skip('should create an account', () => {
const testUser = generateTestUser()
const { email, password, handle, name } = testUser
Expand Down Expand Up @@ -178,7 +180,8 @@ describe('Sign Up', () => {
cy.viewport('iphone-x')
})

it('can navigate to signup from trending', () => {
// TODO: IN TEST JAIL [C-3403] - Failing due to "waves bg but no content" issue
it.skip('can navigate to signup from trending', () => {
cy.visit('trending')
cy.findByRole('link', { name: /sign up/i }).click()
assertOnSignUpPage()
Expand All @@ -189,7 +192,8 @@ describe('Sign Up', () => {
assertOnSignUpPage()
})

it('can navigate to sign-up from sign-in', () => {
// TODO: IN TEST JAIL [C-3403] - Failing due to "waves bg but no content" issue
it.skip('can navigate to sign-up from sign-in', () => {
cy.visit('signin')
cy.findByRole('link', { name: /create an account/i }).click()

Expand All @@ -204,7 +208,7 @@ describe('Sign Up', () => {
assertOnSignUpPage()
})

// TODO: Un-skip when full account creation flow is set up
// TODO: IN TEST JAIL [C-3403] turn back on when full account creation flow is set up
it.skip('should create an account', () => {
const testUser = generateTestUser()
const { email, password, handle } = testUser
Expand Down
7 changes: 5 additions & 2 deletions packages/probers/cypress/e2e/uploadTrack.cy.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@ const completeUpload = () => {
}).should('exist')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that only one upload test was running due to a it.only. I'm assuming that was intentional? But I guess it might have been an accidental push.
Either way that test was still flaky so I skipped the whole suite for now

}

describe('Upload', () => {
// TODO [C-3420]: This suite is in test jail due to upload flows taking too long and timing out
describe.skip('Upload', () => {
beforeEach(() => {
localStorage.setItem('HAS_REQUESTED_BROWSER_PUSH_PERMISSION', 'true')
})
Expand Down Expand Up @@ -285,7 +286,9 @@ describe('Upload', () => {
completeUpload()

cy.findByRole('link', { name: /visit track page/i }).click()
cy.findByRole('heading', { name: /pay-gated track/i, level: 1 }).should('exist')
cy.findByRole('heading', { name: /pay-gated track/i, level: 1 }).should(
'exist'
)
cy.findByRole('button', { name: /preview/i }).should('exist')
cy.findByText(/premium track/i).should('exist')
cy.findByText(/users can unlock/i).should('exist')
Expand Down