ci: smoke test automation #13162
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| name: Install Amplify CLI | ||
| description: Installs Amplify CLI | ||
| inputs: | ||
| cli-version: | ||
| description: a version of Amplify CLI | ||
| required: true | ||
| runs: | ||
| using: 'composite' | ||
| steps: | ||
| - name: Install CLI | ||
| shell: bash | ||
| if: runner.os != 'Windows' | ||
| run: | | ||
| npm install -g @aws-amplify/cli@${{ inputs.cli-version }} | ||
| AMPLIFY_PATH=$(which amplify) | ||
| echo "AMPLIFY_PATH=$AMPLIFY_PATH" | ||
| echo "AMPLIFY_PATH=$AMPLIFY_PATH" >> $GITHUB_ENV | ||
|
|
||
| - name: Install CLI Windows | ||
| shell: powershell | ||
| if: runner.os == 'Windows' | ||
| run: | | ||
| npm install -g @aws-amplify/cli@${{ inputs.cli-version }} | ||
| $AMPLIFY_PATH=(Get-Command amplify).Path | ||
| echo "AMPLIFY_PATH=$AMPLIFY_PATH" | ||
| "AMPLIFY_PATH=$AMPLIFY_PATH" >> $env:GITHUB_ENV |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| The Run Smoke Tests contains lightweight version of smoke tests for Windows. | ||
| GitHub Action's Windows workers are small and using our E2E testing framework for smoke testing overwhelms them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| name: Run Smoke Tests | ||
| description: Executes Smoke Tests | ||
| inputs: | ||
| role-to-assume: | ||
| description: an IAM role to use in tests | ||
| required: true | ||
| region: | ||
| description: an AWS region to run in | ||
| required: false | ||
| default: us-west-2 | ||
| runs: | ||
| using: 'composite' | ||
| steps: | ||
| - name: Verify Amplify Path | ||
| shell: bash | ||
| run: | | ||
| if [[ -z "$AMPLIFY_PATH" ]]; then | ||
| echo "AMPLIFY_PATH must be defined" | ||
| exit 1 | ||
| fi | ||
| echo "AMPLIFY_PATH=$AMPLIFY_PATH" | ||
| $AMPLIFY_PATH version | ||
|
|
||
| - name: Configure AWS Credentials | ||
| uses: aws-actions/configure-aws-credentials@04b98b3f9e85f563fb061be8751a0352327246b0 # v3.0.1 | ||
| with: | ||
| role-to-assume: ${{ inputs.role-to-assume }} | ||
| aws-region: ${{ inputs.region }} | ||
|
|
||
| - name: Install AWS CLI | ||
| shell: powershell | ||
| run: | | ||
| msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi /qn | ||
| aws --version | ||
|
|
||
| - name: Configure Profile | ||
| shell: bash | ||
| run: | | ||
| aws configure set region ${{ inputs.region }} --profile default | ||
| aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID --profile default | ||
| aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY --profile default | ||
| aws configure set aws_session_token $AWS_SESSION_TOKEN --profile default | ||
|
|
||
| - name: Run Smoke Tests | ||
| shell: bash | ||
| run: .github/actions/run-smoke-tests-windows/windows-smoke-tests.sh | ||
|
|
||
| - name: Cleanup | ||
| shell: bash | ||
| if: always() | ||
| run: | | ||
| cd $AMPLIFY_PROJ_DIR | ||
| $AMPLIFY_PATH delete --force |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "version": 1, | ||
| "serviceConfiguration": { | ||
| "serviceName": "AppSync", | ||
| "apiName": "myApiName", | ||
| "transformSchema": "type Todo @model { id: ID! content: String}", | ||
| "defaultAuthType": { "mode": "API_KEY" } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "version": 2, | ||
| "resourceName": "myAuthResource", | ||
| "serviceConfiguration": { | ||
| "serviceName": "Cognito", | ||
| "includeIdentityPool": false, | ||
| "userPoolConfiguration": { "requiredSignupAttributes": ["EMAIL", "PHONE_NUMBER"], "signinMethod": "USERNAME" } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| #!/bin/bash -e | ||
|
|
||
| $AMPLIFY_PATH version | ||
|
|
||
| random_guid=$(powershell -Command "[guid]::NewGuid().ToString()") | ||
| random_guid=${random_guid//-/} | ||
| random_guid=${random_guid:0:6} | ||
| projDir="smoke-test-${random_guid}" | ||
| echo "AMPLIFY_PROJ_DIR=$projDir" >> $GITHUB_ENV | ||
|
|
||
| # Read headless requests content. This must be done before we change directory to $projDir. | ||
| THIS_SCRIPT_PATH=$(dirname "${BASH_SOURCE[0]}") | ||
| ADD_AUTH_REQUEST=$(cat $THIS_SCRIPT_PATH/add_auth_request.json) | ||
| ADD_AUTH_REQUEST=${ADD_AUTH_REQUEST//[$'\r\n']} | ||
| ADD_API_REQUEST=$(cat $THIS_SCRIPT_PATH/add_api_request.json) | ||
| ADD_API_REQUEST=${ADD_API_REQUEST//[$'\r\n']} | ||
|
|
||
| mkdir $projDir | ||
| cd $projDir | ||
|
|
||
| $AMPLIFY_PATH init --yes | ||
|
|
||
| $AMPLIFY_PATH status | ||
|
|
||
| echo $ADD_AUTH_REQUEST | ||
| $AMPLIFY_PATH add auth --headless <<< $ADD_AUTH_REQUEST | ||
|
|
||
| $AMPLIFY_PATH status | ||
|
|
||
| $AMPLIFY_PATH push --yes | ||
|
|
||
| $AMPLIFY_PATH status | ||
|
|
||
| echo $ADD_API_REQUEST | ||
| $AMPLIFY_PATH add api --headless <<< $ADD_API_REQUEST | ||
|
|
||
| $AMPLIFY_PATH status | ||
|
|
||
| $AMPLIFY_PATH push --yes | ||
|
|
||
| $AMPLIFY_PATH status |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| name: Run Smoke Tests | ||
| description: Executes Smoke Tests | ||
| inputs: | ||
| role-to-assume: | ||
| description: an IAM role to use in tests | ||
| required: true | ||
| region: | ||
| description: an AWS region to run in | ||
| required: false | ||
| default: us-west-2 | ||
| runs: | ||
| using: 'composite' | ||
| steps: | ||
| - name: Verify Amplify Path | ||
| shell: bash | ||
| run: | | ||
| if [[ -z "$AMPLIFY_PATH" ]]; then | ||
| echo "AMPLIFY_PATH must be defined" | ||
| exit 1 | ||
| fi | ||
| echo "AMPLIFY_PATH=$AMPLIFY_PATH" | ||
| $AMPLIFY_PATH version | ||
|
|
||
| - name: Install Dependencies | ||
| shell: bash | ||
| run: yarn install --immutable | ||
|
|
||
| - name: Build | ||
| shell: bash | ||
| run: yarn build-tests | ||
|
|
||
| - name: Configure AWS Credentials | ||
| uses: aws-actions/configure-aws-credentials@04b98b3f9e85f563fb061be8751a0352327246b0 # v3.0.1 | ||
| with: | ||
| role-to-assume: ${{ inputs.role-to-assume }} | ||
| aws-region: ${{ inputs.region }} | ||
|
|
||
| # For iOS tests | ||
| - name: Setup Ruby | ||
| uses: ruby/setup-ruby@250fcd6a742febb1123a77a841497ccaa8b9e939 # v1.152.0 | ||
| if: runner.os == 'macOS' | ||
| with: | ||
| ruby-version: '3.2.1' | ||
|
|
||
| # For iOS tests | ||
| - name: Set Default Xcode Version | ||
| if: runner.os == 'macOS' | ||
| shell: bash | ||
| run: | | ||
| sudo xcode-select -s "/Applications/Xcode_14.1.app" | ||
| xcodebuild -version | ||
|
|
||
| - name: Run Smoke Tests | ||
| shell: bash | ||
| run: yarn smoketest -- --forceExit --no-cache --maxWorkers=2 | ||
| env: | ||
| CLI_REGION: ${{ inputs.region }} | ||
| CI: true | ||
| CIRCLECI: true | ||
|
|
||
| - name: Upload Report | ||
| uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce #v3.1.2 | ||
| if: always() | ||
|
There was a problem hiding this comment. is this not the default? There was a problem hiding this comment. No. By default step is skipped if previous step fails. This makes sure that report is uploaded regardless of test success. There was a problem hiding this comment. |
||
| with: | ||
| name: test report | ||
| path: packages/amplify-e2e-tests/amplify-e2e-reports | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| name: Smoke Tests - Canaries | ||
|
There was a problem hiding this comment. What's the notification mechanism for this canary? I.e. how would the oncall find out if a run has failed? There was a problem hiding this comment. This is a good question. Options to pursue:
I'd try that Slack integration first after merging this and explore cloudwatch if this is not working. |
||
|
|
||
| # This is required by aws-actions/configure-aws-credentials | ||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '0 16 * * *' # Everyday 16:00 UTC | ||
|
|
||
| jobs: | ||
| call-smoke-tests: | ||
| uses: ./.github/workflows/smoke-tests.yml | ||
| secrets: inherit | ||
| with: | ||
| versions: '["rc", "latest"]' | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| name: Smoke Tests - Manual Run | ||
|
|
||
| # This is required by aws-actions/configure-aws-credentials | ||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| os: | ||
| required: false | ||
| type: string | ||
| default: '["macos-latest-xl", "ubuntu-latest", "windows-latest"]' | ||
| versions: | ||
| required: false | ||
| type: string | ||
| default: '["rc", "latest"]' | ||
|
|
||
| jobs: | ||
| call-smoke-tests: | ||
| if: github.event_name == 'workflow_dispatch' | ||
| uses: ./.github/workflows/smoke-tests.yml | ||
| secrets: inherit | ||
| with: | ||
| os: ${{ github.event.inputs.os }} | ||
| versions: ${{ github.event.inputs.versions }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| name: Smoke Tests | ||
|
There was a problem hiding this comment. should this be an action rather than a workflow? Seems like it is being dispatched from the canary and manual workflow There was a problem hiding this comment. The reason this is "reusable workflow" (see https://docs.github.com/en/actions/using-workflows/reusing-workflows ) I created "canary" and "manual" as "lightweight entry points" Another reason to have And yet other reason is that creating single workflow definition with multiple "triggers" would require workarounds to have matrix populated correctly in every case. (Depending on trigger we'd need different logic to populate matrix which is hard problem in declarative yaml). |
||
|
|
||
| # This is required by aws-actions/configure-aws-credentials | ||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| os: | ||
| required: false | ||
| type: string | ||
| default: '["macos-latest-xl", "ubuntu-latest", "windows-latest"]' | ||
| versions: | ||
| required: false | ||
| type: string | ||
| default: '["rc", "latest"]' | ||
|
|
||
| env: | ||
| NODE_OPTIONS: --max-old-space-size=8096 | ||
|
|
||
| jobs: | ||
| run-smoke-tests: | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: ${{ fromJson(inputs.os) }} | ||
| cli-version: ${{ fromJson(inputs.versions) }} | ||
| name: Smoke Tests ${{ matrix.os }} ${{ matrix.cli-version }} | ||
| runs-on: ${{ matrix.os }} | ||
| steps: | ||
| - name: Checkout Code | ||
| uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Install Amplify CLI | ||
| uses: ./.github/actions/install-cli | ||
| with: | ||
| cli-version: ${{ matrix.cli-version }} | ||
|
|
||
| - name: Run Smoke Tests Unix | ||
| if: runner.os != 'Windows' | ||
| uses: ./.github/actions/run-smoke-tests | ||
| with: | ||
| role-to-assume: ${{ secrets.SMOKE_TESTS_ROLE_ARN }} | ||
|
|
||
| - name: Run Smoke Tests Windows | ||
| if: runner.os == 'Windows' | ||
| uses: ./.github/actions/run-smoke-tests-windows | ||
| with: | ||
| role-to-assume: ${{ secrets.SMOKE_TESTS_ROLE_ARN }} | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is this necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed to trigger test profile configuration
amplify-cli/packages/amplify-e2e-tests/src/configure_tests.ts
Line 4 in ce0a0ff