ci: e2e tests on windows #750

dpilch · 2023-10-23T15:25:14Z

Description of changes

Run E2E tests on Windows.

The Windows bash terminal on CodeBuild appears to be non-interactive. This is causing issues with selecting a region when running amplify configure. I've set all the Windows E2E tests to run us-east-1 for now and we can fix that later when you have some more time.

There are two disabled tests for windows only:

amplify pull on an ios project.
Cypress run after generating graphql statements.

Test 1 is not very valuable. It is possible to do this with the current CLI, but realistically no customer would do this because XCode is not available on windows.

Test 2 is much more valuable. This asserts that the generated graphql statements work in a running app. The app compiles but then there is an error in cypress. It does not appear to be an error in a test case. It might be an issue with cypress on the windows instance.

I was not able to reproduce either case on my own windows workspace.

To recap, we have 3 follow ups to complete later:

Windows E2E tests on other regions
Enable amplify pull on ios project test
Enable Cypress run after generating graphql statements test.

Issue #, if available

N/A

Description of how you validated changes

Run the E2E tests.

Checklist

PR description included
yarn test passes
Tests are changed or added

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

dpilch · 2023-10-26T21:00:57Z

.codebuild/scripts/e2e_test_windows.sh

+set -e
+
+source ./shared-scripts.sh && _setupE2ETestsWindows
+codebuild-breakpoint


This is keeping the same behavior as E2E tests on linux. I'm not sure codebuild-breakpoint is strictly needed on either.

This is used when you want to run the tests in debug mode. I think the script is called yarn-e2e-debug.

dpilch · 2023-10-26T21:07:29Z

packages/amplify-codegen-e2e-core/src/categories/api.ts

-      .wait(
-        '"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud',
-      )
+      .wait('"amplify publish" will build all your local backend and frontend resources')


Strange error.

● codegen remove tests - iOS › Does not delete files during codegen remove -- 351 | 352 | Expected value "\"amplify publish\" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud" 353 | Received: 354 | "ision it in the cloud" 355 | 356 | Message: 357 | expected "ision it in the cloud" to contain "\"amplify publish\" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud" 358 | 359 | Difference: 360 | 361 | - Expected 362 | + Received 363 | 364 | - "amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud 365 | + ision it in the cloud

The message is correct when running manually. Maybe there is a maximum length? The "v" in "provision" is the 120th character.

This change was also made on the CLI E2E when enabling windows.
aws-amplify/amplify-cli@97c6de4

Alternatively, you can wrap the message with */ regex to ignore subsequent parts. Just FYI.

dpilch · 2023-10-26T21:13:42Z

shared-scripts.sh

 function _scanArtifacts {
-    if ! yarn ts-node .codebuild/scripts/scan_artifacts.ts; then


Get an error when using yarn over npx/npm. It looks like there are some workarounds, but it is easier to just use npx/npm here because there will be no functional difference to the scripts.

error Couldn't find the binary C:\codebuild\tmp\output\src2004\src\github.com\aws-amplify\amplify-codegen\node_modules\.bin\ts-node

dpilch · 2023-10-26T21:14:44Z

.codebuild/scripts/local_publish_helpers.sh

@@ -7,7 +7,7 @@ function startLocalRegistry {
    # Start local registry
    tmp_registry_log="$(mktemp)"
    echo "Registry output file: $tmp_registry_log"
-    (cd && nohup npx ${VERDACCIO_PACKAGE:-$default_verdaccio_package} -c $1 &>$tmp_registry_log &)
+    nohup npx ${VERDACCIO_PACKAGE:-$default_verdaccio_package} -c $1 &>$tmp_registry_log &


The cd is unnecessary here and caused some issues on Windows.

dpilch · 2023-10-26T21:21:35Z

packages/amplify-codegen-e2e-core/package.json

@@ -32,7 +32,7 @@
    "jest-circus": "^27.5.1",
    "jest-environment-node": "^26.6.2",
    "lodash": "^4.17.19",
-    "node-pty": "beta",
+    "node-pty": "1.0.0",


This might not be necessary for this change. I tried this when troubleshooting various issues. However, this is probably something we should do regardless so that our version is stable. I can try removing this and adding it to a separate change if it is desired.

This reverts commit c425469.

dpilch · 2023-10-30T19:38:32Z

scripts/split-e2e-tests.ts

+  // amplify-configure will always choose us-east-1 for the region.
+  // This will set all Windows jobs to use us-east-1 to avoid region mismatch.
+  // We will come back to this later to properly fix the testing issue.
+  const region = os === 'w' ? 'us-east-1' : AWS_REGIONS_TO_RUN_TESTS[jobIdx % AWS_REGIONS_TO_RUN_TESTS.length];


Another weird thing, this only seems to affect the amplify configure step. On commands like codegen add the selection is set correctly.

We might need to look into how we're sending the region selection command interactively and make it for both platforms. Running in us-east-1 is fine for now.

It is using j/k to navigate the list. I tried with up/down arrow as well and ran into the same issue. I've created a task to investigate further.

phani-srikar · 2023-10-30T21:50:07Z

.codebuild/scripts/e2e_test_windows.sh

+set -e
+
+source ./shared-scripts.sh && _setupE2ETestsWindows
+codebuild-breakpoint


This is used when you want to run the tests in debug mode. I think the script is called yarn-e2e-debug.

phani-srikar · 2023-10-30T21:51:31Z

packages/amplify-codegen-e2e-core/src/categories/api.ts

-      .wait(
-        '"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud',
-      )
+      .wait('"amplify publish" will build all your local backend and frontend resources')


Alternatively, you can wrap the message with */ regex to ignore subsequent parts. Just FYI.

phani-srikar · 2023-10-30T21:54:03Z

packages/amplify-codegen-e2e-tests/src/__tests__/graphql-documents-generator.test.ts

@@ -31,9 +32,10 @@ describe('GraphQL documents generator e2e tests', () => {
  });

  const schemaFileName = 'schema.graphql';
-  it('generates valid GraphQL documents for given schema', async () => {
+  // skip cypress test on windows
+  (isWindows() ? it.skip : it)('generates valid GraphQL documents for given schema', async () => {


Do we want to run these on windows as well eventually? If yes, we should have a task for it.

I've created tasks for the 2 disabled tests and the region issue.

phani-srikar · 2023-10-30T21:55:49Z

scripts/split-e2e-tests.ts

+  // amplify-configure will always choose us-east-1 for the region.
+  // This will set all Windows jobs to use us-east-1 to avoid region mismatch.
+  // We will come back to this later to properly fix the testing issue.
+  const region = os === 'w' ? 'us-east-1' : AWS_REGIONS_TO_RUN_TESTS[jobIdx % AWS_REGIONS_TO_RUN_TESTS.length];


We might need to look into how we're sending the region selection command interactively and make it for both platforms. Running in us-east-1 is fine for now.

dpilch force-pushed the windows-e2e branch 2 times, most recently from b152483 to ffcedcb Compare October 25, 2023 17:55

ci: e2e tests on windows

ad96f00

dpilch force-pushed the windows-e2e branch from 3a758a6 to ad96f00 Compare October 26, 2023 20:58

dpilch commented Oct 26, 2023

View reviewed changes

dpilch marked this pull request as ready for review October 26, 2023 21:21

dpilch requested review from a team as code owners October 26, 2023 21:21

dpilch marked this pull request as draft October 26, 2023 21:22

dpilch added 13 commits October 27, 2023 09:34

ci: enable verbose logging

5df6f87

ci: enable verbose logging

5953dab

ci: fix amplify pull windows

c81e415

ci: only run failing test

328a6ef

ci: fix windows test

c4d3216

ci: test logs

7cc49c8

test: skip ios pull test on windows

dd946a9

test: skip cypress test on windows

faba922

ci: cleanup

effafa2

ci: replace j/k with arrows

c425469

ci: fix isWindows import

b3bf606

Revert "ci: replace j/k with arrows"

48977ee

This reverts commit c425469.

ci: fix conditonal skip

066a5d9

dpilch marked this pull request as ready for review October 30, 2023 18:45

dpilch commented Oct 30, 2023

View reviewed changes

alharris-at approved these changes Oct 30, 2023

View reviewed changes

phani-srikar approved these changes Oct 30, 2023

View reviewed changes

dpilch merged commit 8ec490d into main Oct 31, 2023
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: e2e tests on windows #750

ci: e2e tests on windows #750

dpilch commented Oct 23, 2023 •

edited

Loading

dpilch Oct 26, 2023

phani-srikar Oct 30, 2023

dpilch Oct 26, 2023

phani-srikar Oct 30, 2023

dpilch Oct 26, 2023

dpilch Oct 26, 2023

dpilch Oct 26, 2023

dpilch Oct 30, 2023 •

edited

Loading

phani-srikar Oct 30, 2023

dpilch Oct 31, 2023

phani-srikar Oct 30, 2023

phani-srikar Oct 30, 2023

phani-srikar Oct 30, 2023

dpilch Oct 31, 2023

phani-srikar Oct 30, 2023

		function _scanArtifacts {
		if ! yarn ts-node .codebuild/scripts/scan_artifacts.ts; then

ci: e2e tests on windows #750

ci: e2e tests on windows #750

Conversation

dpilch commented Oct 23, 2023 • edited Loading

Description of changes

Issue #, if available

Description of how you validated changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpilch Oct 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpilch commented Oct 23, 2023 •

edited

Loading

dpilch Oct 30, 2023 •

edited

Loading