Skip to content

Commit

Permalink
Wrap maestro UI tests inside a retry command (duckduckgo#5405)
Browse files Browse the repository at this point in the history
Task/Issue URL:
https://app.asana.com/0/72649045549333/1209000807402387/f

### Description
Wraps Maestro UI tests in a `retry` wrapper to minimize any noise from
occasionally flakey tests (either from the test themselves or from
flakiness in the testing infrastructure). Where we do spot flakiness in
our tests, we should fix them and having the `retry` isn’t a substitute
for aiming for properly deterministic tests.

By default, all tests are being set with `maxRetries = 3` by default.

### 💡  Tip for reviewing the PR:
- each test suite has a `retry` block added, and then the rest of the
test steps have been indented. there are, therefore, mostly
whitespace-only changes
- if you want an easier time in the diff, **Hide whitespace:**
<img width="253" alt="Screenshot 2024-12-18 at 12 26 33"
src="https://github.com/user-attachments/assets/3dfe4eff-4583-48a8-96ea-12b4c2953737"
/>


### Steps to test this PR



- QA optional
- [x] Ensure end-to-end CI job passed:
https://github.com/duckduckgo/Android/actions/runs/12392897145

Co-authored-by: Craig Russell <1336281+CDRussell@users.noreply.github.com>
  • Loading branch information
CDRussell and CDRussell authored Dec 18, 2024
1 parent 92f9069 commit ac5f7fc
Show file tree
Hide file tree
Showing 54 changed files with 3,078 additions and 2,915 deletions.
13 changes: 12 additions & 1 deletion .github/workflows/end-to-end-robintest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,4 +115,15 @@ jobs:
timeout: 120
app-file: apk/release.apk
android-api-level: 33
workspace: .maestro/notifications_permissions_android13_plus
workspace: .maestro/notifications_permissions_android13_plus

- name: Create Asana task when workflow failed
if: ${{ failure() }}
uses: honeycombio/gha-create-asana-task@main
with:
asana-secret: ${{ secrets.GH_ASANA_SECRET }}
asana-workspace-id: ${{ secrets.GH_ASANA_WORKSPACE_ID }}
asana-project-id: ${{ secrets.GH_ASANA_AOR_PROJECT_ID }}
asana-section-id: ${{ secrets.GH_ASANA_INCOMING_ID }}
asana-task-name: GH Workflow Failure - End to end tests (Robin)
asana-task-description: The end to end workflow has failed. See https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,52 @@ appId: com.duckduckgo.mobile.android
tags:
- adClickTest
---
- launchApp:
clearState: true
- retry:
maxRetries: 3
commands:
- launchApp:
clearState: true

- runFlow: ../shared/onboarding.yaml
- runFlow: ../shared/onboarding.yaml

- inputText: "https://www.search-company.site/#ad-id-10"
- pressKey: Enter
- assertVisible:
text: ".*Got It.*"
- tapOn:
text: "Got It"
- tapOn:
id: "com.duckduckgo.mobile.android:id/fireIconMenu"
- tapOn:
text: "Cancel"
- assertVisible:
id: "ad-id-10"
- tapOn:
id: "ad-id-10"
- assertVisible:
text: "Publisher site"
- tapOn:
id: "com.duckduckgo.mobile.android:id/omnibarIconContainer"
- assertVisible:
text: "View Tracker Companies"
- tapOn:
text: "View Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were blocked from loading because they were identified as tracking requests. If a company's requests are loaded, it can allow them to profile you."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "convert.ad-company.site"
- assertVisible:
text: "ad-company.site"
- action: back
- assertVisible:
text: "View Non-Tracker Companies"
- tapOn:
text: "View Non-Tracker Companies"
- assertVisible:
text: "We did not identify any requests from third-party domains."
- assertVisible:
text: "About our Web Tracking Protections"
- inputText: "https://www.search-company.site/#ad-id-10"
- pressKey: Enter
- assertVisible:
text: ".*Got It.*"
- tapOn:
text: "Got It"
- tapOn:
id: "com.duckduckgo.mobile.android:id/fireIconMenu"
- tapOn:
text: "Cancel"
- assertVisible:
id: "ad-id-10"
- tapOn:
id: "ad-id-10"
- assertVisible:
text: "Publisher site"
- tapOn:
id: "com.duckduckgo.mobile.android:id/omnibarIconContainer"
- assertVisible:
text: "View Tracker Companies"
- tapOn:
text: "View Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were blocked from loading because they were identified as tracking requests. If a company's requests are loaded, it can allow them to profile you."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "convert.ad-company.site"
- assertVisible:
text: "ad-company.site"
- action: back
- assertVisible:
text: "View Non-Tracker Companies"
- tapOn:
text: "View Non-Tracker Companies"
- assertVisible:
text: "We did not identify any requests from third-party domains."
- assertVisible:
text: "About our Web Tracking Protections"
Original file line number Diff line number Diff line change
Expand Up @@ -2,55 +2,58 @@ appId: com.duckduckgo.mobile.android
tags:
- adClickTest
---
- launchApp:
clearState: true
- retry:
maxRetries: 3
commands:
- launchApp:
clearState: true

- runFlow: ../shared/onboarding.yaml
- runFlow: ../shared/onboarding.yaml

- inputText: "https://www.search-company.site/#ad-id-11"
- pressKey: Enter
- assertVisible:
text: ".*Got It.*"
- tapOn:
text: "Got It"
- tapOn:
id: "com.duckduckgo.mobile.android:id/fireIconMenu"
- tapOn:
text: "Cancel"
- assertVisible:
id: "ad-id-11"
- tapOn:
id: "ad-id-11"
- assertVisible:
text: "Publisher site"
- tapOn:
id: "com.duckduckgo.mobile.android:id/omnibarIconContainer"
- assertVisible:
text: "View Tracker Companies"
- tapOn:
text: "View Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were blocked from loading because they were identified as tracking requests. If a company's requests are loaded, it can allow them to profile you."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "ad-company.site"
- action: back
- assertVisible:
text: "View Non-Tracker Companies"
- tapOn:
text: "View Non-Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were loaded. If a company's requests are loaded, it can allow them to profile you, though our other web tracking protections still apply."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: "The following domain’s requests were loaded because a publisher-company.site ad on DuckDuckGo was recently clicked. These requests help evaluate ad effectiveness. All ads on DuckDuckGo are non-profiling."
- assertVisible:
text: "How our search ads impact our protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "convert.ad-company.site"
- inputText: "https://www.search-company.site/#ad-id-11"
- pressKey: Enter
- assertVisible:
text: ".*Got It.*"
- tapOn:
text: "Got It"
- tapOn:
id: "com.duckduckgo.mobile.android:id/fireIconMenu"
- tapOn:
text: "Cancel"
- assertVisible:
id: "ad-id-11"
- tapOn:
id: "ad-id-11"
- assertVisible:
text: "Publisher site"
- tapOn:
id: "com.duckduckgo.mobile.android:id/omnibarIconContainer"
- assertVisible:
text: "View Tracker Companies"
- tapOn:
text: "View Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were blocked from loading because they were identified as tracking requests. If a company's requests are loaded, it can allow them to profile you."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "ad-company.site"
- action: back
- assertVisible:
text: "View Non-Tracker Companies"
- tapOn:
text: "View Non-Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were loaded. If a company's requests are loaded, it can allow them to profile you, though our other web tracking protections still apply."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: "The following domain’s requests were loaded because a publisher-company.site ad on DuckDuckGo was recently clicked. These requests help evaluate ad effectiveness. All ads on DuckDuckGo are non-profiling."
- assertVisible:
text: "How our search ads impact our protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "convert.ad-company.site"
Original file line number Diff line number Diff line change
Expand Up @@ -2,55 +2,58 @@ appId: com.duckduckgo.mobile.android
tags:
- adClickTest
---
- launchApp:
clearState: true
- retry:
maxRetries: 3
commands:
- launchApp:
clearState: true

- runFlow: ../shared/onboarding.yaml
- runFlow: ../shared/onboarding.yaml

- inputText: "https://www.search-company.site/#ad-id-12"
- pressKey: Enter
- assertVisible:
text: ".*Got It.*"
- tapOn:
text: "Got It"
- tapOn:
id: "com.duckduckgo.mobile.android:id/fireIconMenu"
- tapOn:
text: "Cancel"
- assertVisible:
id: "ad-id-12"
- tapOn:
id: "ad-id-12"
- assertVisible:
text: "Publisher site"
- tapOn:
id: "com.duckduckgo.mobile.android:id/omnibarIconContainer"
- assertVisible:
text: "View Tracker Companies"
- tapOn:
text: "View Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were blocked from loading because they were identified as tracking requests. If a company's requests are loaded, it can allow them to profile you."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "ad-company.site"
- action: back
- assertVisible:
text: "View Non-Tracker Companies"
- tapOn:
text: "View Non-Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were loaded. If a company's requests are loaded, it can allow them to profile you, though our other web tracking protections still apply."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: "The following domain’s requests were loaded because a publisher-company.site ad on DuckDuckGo was recently clicked. These requests help evaluate ad effectiveness. All ads on DuckDuckGo are non-profiling."
- assertVisible:
text: "How our search ads impact our protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "convert.ad-company.site"
- inputText: "https://www.search-company.site/#ad-id-12"
- pressKey: Enter
- assertVisible:
text: ".*Got It.*"
- tapOn:
text: "Got It"
- tapOn:
id: "com.duckduckgo.mobile.android:id/fireIconMenu"
- tapOn:
text: "Cancel"
- assertVisible:
id: "ad-id-12"
- tapOn:
id: "ad-id-12"
- assertVisible:
text: "Publisher site"
- tapOn:
id: "com.duckduckgo.mobile.android:id/omnibarIconContainer"
- assertVisible:
text: "View Tracker Companies"
- tapOn:
text: "View Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were blocked from loading because they were identified as tracking requests. If a company's requests are loaded, it can allow them to profile you."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "ad-company.site"
- action: back
- assertVisible:
text: "View Non-Tracker Companies"
- tapOn:
text: "View Non-Tracker Companies"
- assertVisible:
text: "The following third-party domains’ requests were loaded. If a company's requests are loaded, it can allow them to profile you, though our other web tracking protections still apply."
- assertVisible:
text: "About our Web Tracking Protections"
- assertVisible:
text: "The following domain’s requests were loaded because a publisher-company.site ad on DuckDuckGo was recently clicked. These requests help evaluate ad effectiveness. All ads on DuckDuckGo are non-profiling."
- assertVisible:
text: "How our search ads impact our protections"
- assertVisible:
text: ".*Ad Company"
- assertVisible:
text: "convert.ad-company.site"
Loading

0 comments on commit ac5f7fc

Please sign in to comment.