Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove cloning and user-reset from e2e tests #10418

Merged
merged 3 commits into from
Mar 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/e2e-tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Our old E2E test suite re-recorded those examples every time the test suite ran.
The tests themselves are written using a set of "page object"-style helper functions that abstract specific pieces of DOM interaction. This allows the tests themselves to be written in a more readable style and focus on the sequence of operations:

```ts
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test(`Test stepping forward through breakpoints when rewound before the first one.`, async ({
pageWithMeta: { page, recordingId },
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/comments-01.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { openDevToolsTab, startTest } from "../helpers";
import { E2E_USER_1 } from "../helpers/authentication";
import { addSourceCodeComment, deleteComment, editComment } from "../helpers/comments";
import { openSource } from "../helpers/source-explorer-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "authenticated_comments.html", testUsers: [E2E_USER_1] });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/comments-02.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import {
replyToComment,
} from "../helpers/comments";
import { openSource } from "../helpers/source-explorer-panel";
import test, { Page } from "../testFixtureCloneRecording";
import test, { Page } from "../testFixture";

const url = "authenticated_comments.html";

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/comments-03.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import {
} from "../helpers/comments";
import { openNetworkPanel } from "../helpers/network-panel";
import { openSource } from "../helpers/source-explorer-panel";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "authenticated_comments.html", testUsers: [E2E_USER_1] });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/logpoints-01.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import {
import { openSource } from "../helpers/source-explorer-panel";
import { addLogpoint, editLogPoint, removeAllLogpoints } from "../helpers/source-panel";
import { waitForRecordingToFinishIndexing } from "../helpers/utils";
import test, { Page, expect } from "../testFixtureCloneRecording";
import test, { Page, expect } from "../testFixture";

const url = "authenticated_logpoints.html";
const lineNumber = 14;
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/passport-01.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { isPassportItemCompleted, showPassport } from "../helpers/passport";
import { enablePassport } from "../helpers/settings";
import { addLogpoint, removeAllLogpoints } from "../helpers/source-panel";
import { waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "doc_rr_console.html", testUsers: [E2E_USER_1] });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/passport-02.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { openReactDevtoolsPanel } from "../helpers/new-react-devtools-panel";
import { isPassportItemCompleted } from "../helpers/passport";
import { enablePassport } from "../helpers/settings";
import { waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "cra/dist/index.html", testUsers: [E2E_USER_1] });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/passport-03.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { enablePassport } from "../helpers/settings";
import { addLogpoint, editLogPoint, removeAllLogpoints } from "../helpers/source-panel";
import { enterFocusMode, saveFocusRange } from "../helpers/timeline";
import { clearTextArea, getCommandKey, waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "doc_rr_console.html", testUsers: [E2E_USER_1] });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/authenticated/passport-04.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { addSourceCodeComment, deleteComment } from "../helpers/comments";
import { isPassportItemCompleted } from "../helpers/passport";
import { enablePassport } from "../helpers/settings";
import { waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "doc_rr_console.html", testUsers: [E2E_USER_1] });

Expand Down
64 changes: 0 additions & 64 deletions packages/e2e-tests/helpers/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -268,67 +268,3 @@ export async function resetTestUser(email: string, testScope: string) {
},
});
}

export async function cloneTestRecording(recordingId: string): Promise<string> {
if (!process.env.AUTHENTICATED_TESTS_WORKSPACE_API_KEY) {
throw new Error(
"AUTHENTICATED_TESTS_WORKSPACE_API_KEY must be set in order to clone test recordings."
);
}

const variables = { recordingId, secret: process.env.AUTOMATED_TEST_SECRET };

var startTime = performance.now();
const clonedRecording = await axios({
url: config.graphqlUrl,
method: "POST",
headers: {
Authorization: `Bearer ${process.env.AUTHENTICATED_TESTS_WORKSPACE_API_KEY}`,
},
data: {
query: `
mutation cloneTestRecording($recordingId: UUID!, $secret: String!) {
cloneTestRecording(input: { recordingId: $recordingId, secret: $secret }) {
recordingId
}
}
`,
variables,
},
});
var endTime = performance.now();
const timeInSecs = Math.round((endTime - startTime) / 1000);
if (timeInSecs > 10) {
console.warn(`Cloning took ${timeInSecs} seconds.`);
}

return clonedRecording.data.data.cloneTestRecording.recordingId;
}

export async function deleteTestRecording(recordingId: string) {
if (!process.env.AUTHENTICATED_TESTS_WORKSPACE_API_KEY) {
throw new Error(
"AUTHENTICATED_TESTS_WORKSPACE_API_KEY must be set in order to delete test recordings."
);
}

const variables = { recordingId, secret: process.env.AUTOMATED_TEST_SECRET };

return axios({
url: config.graphqlUrl,
method: "POST",
headers: {
Authorization: `Bearer ${process.env.AUTHENTICATED_TESTS_WORKSPACE_API_KEY}`,
},
data: {
query: `
mutation deleteTestRecording($recordingId: UUID!, $secret: String!) {
deleteTestRecording(input: { recordingId: $recordingId, secret: $secret }) {
success
}
}
`,
variables,
},
});
}
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import crypto from "crypto";
import { Page, test as base } from "@playwright/test";
import axios from "axios";

import exampleRecordings from "./examples.json";
import { TestRecordingKey } from "./helpers";
import { TestUser } from "./helpers/authentication";
import { cloneTestRecording, deleteTestRecording, resetTestUser } from "./helpers/utils";
import { resetTestUser } from "./helpers/utils";
import { loadRecording } from "./scripts/loadRecording";

type TestIsolatedRecordingFixture = {
Expand All @@ -23,30 +23,16 @@ const testWithCloneRecording = base.extend<TestIsolatedRecordingFixture>({
exampleKey: undefined,
testUsers: undefined,
pageWithMeta: async ({ page, exampleKey, testUsers }, use) => {
const exampleRecordings = require("./examples.json");
Copy link
Contributor Author

@bvaughn bvaughn Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why were we using require here before?

Using import gives up TSC hints for the JSON structure.

Tracing through Git blame, it looks like Jason wrote it this way two years ago. My guess is that our tooling didn't support import at the time, or that he didn't realize you could use import 🤷🏼 Either way, I think this is a good change to help avoid typos.

if (!exampleRecordings[exampleKey]) {
throw new Error(`Invalid recording: ${exampleKey}`);
}

const testScope = crypto.randomUUID();
let newRecordingId: string | undefined = undefined;
try {
try {
const { recording } = exampleRecordings[exampleKey];
console.log("Cloning recording");
newRecordingId = await cloneTestRecording(recording);
} catch (err: any) {
if (axios.isAxiosError(err)) {
console.error("Axios error cloning recording: ", {
errors: err.response?.data?.errors,
details: err.toJSON(),
});
}
throw err;
}

let recordingId = exampleRecordings[exampleKey]?.recording;
try {
try {
await loadRecording(newRecordingId);
await loadRecording(recordingId);
} catch (e) {
console.warn("Error processing recording; ignoring.");
}
Expand All @@ -59,7 +45,7 @@ const testWithCloneRecording = base.extend<TestIsolatedRecordingFixture>({
// Run test.
await use({
page,
recordingId: newRecordingId,
recordingId: recordingId,
testScope,
});
} finally {
Expand All @@ -69,15 +55,13 @@ const testWithCloneRecording = base.extend<TestIsolatedRecordingFixture>({
console.error("Error stopping JS coverage: ", err);
}

if (newRecordingId) {
await deleteTestRecording(newRecordingId);
}
for (const user of testUsers ?? []) {
await resetTestUser(user.email, testScope);
}
}
},
});

export default testWithCloneRecording;
export { expect } from "@playwright/test";
export type { Page } from "@playwright/test";
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-01.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { openDevToolsTab, startTest } from "../helpers";
import { executeAndVerifyTerminalExpression } from "../helpers/console-panel";
import { resumeToLine, rewindToLine } from "../helpers/pause-information-panel";
import { addBreakpoint } from "../helpers/source-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_basic.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-02.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { openDevToolsTab, startTest } from "../helpers";
import { executeAndVerifyTerminalExpression } from "../helpers/console-panel";
import { rewindToLine } from "../helpers/pause-information-panel";
import { addBreakpoint } from "../helpers/source-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_basic.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-03.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { openDevToolsTab, startTest } from "../helpers";
import { executeAndVerifyTerminalExpression } from "../helpers/console-panel";
import { resumeToLine, rewind, rewindToLine } from "../helpers/pause-information-panel";
import { addBreakpoint, removeBreakpoint } from "../helpers/source-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_basic.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-04.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { openDevToolsTab, startTest } from "../helpers";
import { resumeToLine, rewindToLine } from "../helpers/pause-information-panel";
import { addBreakpoint } from "../helpers/source-panel";
import test, { Page } from "../testFixtureCloneRecording";
import test, { Page } from "../testFixture";

test.use({ exampleKey: "doc_control_flow.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-05.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import {
rewindToLine,
} from "../helpers/pause-information-panel";
import { addBreakpoint, removeBreakpoint } from "../helpers/source-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_debugger_statements.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-06.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { openDevToolsTab, startTest } from "../helpers";
import { findConsoleMessage } from "../helpers/console-panel";
import { addLogpoint } from "../helpers/source-panel";
import test, { Page, expect } from "../testFixtureCloneRecording";
import test, { Page, expect } from "../testFixture";

async function checkMessageLocation(page: Page, text: string, location: string) {
const message = await findConsoleMessage(page, text, "log-point");
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-07.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import {
waitForLogpoint,
} from "../helpers/source-panel";
import { debugPrint } from "../helpers/utils";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_navigate.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/breakpoints-08.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import {
togglePoint,
} from "../helpers/pause-information-panel";
import { addBreakpoint, removeBreakpoint } from "../helpers/source-panel";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "doc_navigate.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console-expressions-01.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import {
warpToMessage,
} from "../helpers/console-panel";
import { selectFrame, verifyFramesCount } from "../helpers/pause-information-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_async.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console_async_eval.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import {
} from "../helpers/console-panel";
import { selectFrame, verifyFramesCount } from "../helpers/pause-information-panel";
import { addLogpoint } from "../helpers/source-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_async.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console_dock.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import {
verifyToolboxLayout,
verifyToolboxLayoutOptions,
} from "../helpers/layout";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_basic.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console_errors.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import {
toggleSideFilters,
verifyConsoleMessage,
} from "../helpers/console-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_exceptions_bundle.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console_eval.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import {
openConsolePanel,
warpToMessage,
} from "../helpers/console-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_basic.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console_warp-01.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import {
rewindToLine,
} from "../helpers/pause-information-panel";
import { addBreakpoint } from "../helpers/source-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_error.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/console_warp-02.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import {
warpToMessage,
} from "../helpers/console-panel";
import { reverseStepOverToLine, stepOverToLine } from "../helpers/pause-information-panel";
import test from "../testFixtureCloneRecording";
import test from "../testFixture";

test.use({ exampleKey: "doc_rr_logs.html" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/cypress-01_basic-panel.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import {
openCypressTestPanel,
} from "../helpers/testsuites";
import { waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "flake/adding-spec.ts" });

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import {
} from "../helpers/testsuites";
import { getTimelineCurrentHoverPercent, getTimelineCurrentPercent } from "../helpers/timeline";
import { waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "flake/adding-spec.ts" });

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import {
} from "../helpers/testsuites";
import { waitForTimelineAdvanced } from "../helpers/timeline";
import { delay, getByTestName, waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "flake/adding-spec.ts" });

Expand Down
2 changes: 1 addition & 1 deletion packages/e2e-tests/tests/cypress-04_menu-commands.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import {
} from "../helpers/testsuites";
import { getTimelineCurrentPercent, waitForTimelineAdvanced } from "../helpers/timeline";
import { getByTestName, waitFor } from "../helpers/utils";
import test, { expect } from "../testFixtureCloneRecording";
import test, { expect } from "../testFixture";

test.use({ exampleKey: "cypress-realworld/bankaccounts.spec.js" });

Expand Down
Loading
Loading