forked from dotnet/android
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[tests] Add .apk test job to azure-pipelines.yaml (dotnet#3042)
Adds a `Test` stage to `build-tools/automation/azure-pipelines.yaml`.
At present, this stage only contains one test job, which builds and
runs the following emulator tests:
* Mono.Android_Tests
* Mono.Android-Tests (AOT + LLVM)
* Mono.Android-Tests (Bundle)
* Mono.Android-Tests (Profiled AOT)
* Mono.Android_TestsAppBundle
* Mono.Android_TestsMultiDex
* Xamarin.Android.JcwGen_Tests
* Xamarin.Android.Locale_Tests
* Xamarin.Android.Locale-Tests (AOT)
* Xamarin.Android.Locale-Tests (Profiled AOT)
* Xamarin.Android.EmbeddedDSO_Test
* Xamarin.Forms.Performance.Integration.Droid
* Xamarin.Forms.Performance.Integration.Droid (AOT)
* Xamarin.Forms.Performance.Integration.Droid (Profiled AOT)
* Xamarin.Forms.Performance.Integration.Droid (Bundle)
Note in particular that the BCL tests are currently absent.
Execution of the BCL Test suite will also need to be added to the
`Test` stage in the future.
Other Fixes:
* Introductory support has been added for running `.apk` tests
against a *system* Xamarin.Android installation.
We can now build and run `Xamarin.Android.JcwGen_Tests`,
`Mono.Android_Tests`, and
`Xamarin.Forms.Performance.Integration.Droid` tests against a
system installation of Xamarin.Android.
* Produce an error when any test failures occur during
instrumentation.
The instrumentation exit code value of `-1` indicates success,
and any other values returned here should be reported as errors.
A non `-1` value indicates that crash or test failure occurred.
* Produce an error if `adb logcat` timing data can't be parsed.
We'll now error out when parsing `adb logcat` for timing data if
the running application died, or if we fail to detect application
startup.
* Don't always report `adb logcat` stderr as a warning
`adb` invocations are prone to returning non zero exit codes even
when they are successful. This can make our MSBuild output a bit
noisy [^0] when they are invoked using the `<Adb/>` task.
The new optional `Adb.WriteOutputAsMessage` property can now be
provided to the adb task, so that we don't always write messages
coming from stderr as MSBuild warnings. This is used in our
targets which wait for emulator startup to make it easier to
detect real failures in test instrumentation and emulator startup.
[^0]: Example warnings:
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Android Debug Bridge version 1.0.40
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Version 28.0.2-5303910
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Installed as /Users/peter/android-toolchain/sdk/platform-tools/adb
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192]
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:287] adb_query: host-serial:emulator-5570:features
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:137] _adb_connect: host:version
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:107] writex: fd=3 len=16 30303063686f73743a76657273696f6e 000chost:version
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:97] readx: fd=3 wanted=4 got=4 4f4b4159 OKAY
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:165] _adb_connect: return fd 3
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:197] adb_connect: service host-serial:emulator-5570:features
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4- Loading branch information
Showing
13 changed files
with
306 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
28 changes: 28 additions & 0 deletions
28
build-tools/automation/yaml-templates/apk-instrumentation.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| parameters: | ||
| configuration: [] | ||
| testName: [] | ||
| project: [] | ||
| testResultsFiles: [] | ||
| extraBuildArgs: "" | ||
| testResultsFormat: NUnit | ||
| packageType: Apk | ||
|
|
||
| steps: | ||
| - task: MSBuild@1 | ||
| displayName: run ${{ parameters.testName }} | ||
| inputs: | ||
| solution: ${{ parameters.project }} | ||
| configuration: ${{ parameters.configuration }} | ||
| msbuildArguments: > | ||
| /t:AcquireAndroidTarget,SignAndroidPackage,UndeployTestApks,DeployTest${{ parameters.packageType }}s,RunTestApks,RenameApkTestCases,ReportComponentFailures | ||
| /bl:$(System.DefaultWorkingDirectory)/bin/Test${{ parameters.configuration }}/run${{ parameters.testName }}.binlog | ||
| ${{ parameters.extraBuildArgs }} | ||
| condition: succeededOrFailed() | ||
|
|
||
| - task: PublishTestResults@2 | ||
| displayName: publish ${{ parameters.testName }} results | ||
| inputs: | ||
| testResultsFormat: ${{ parameters.testResultsFormat }} | ||
| testResultsFiles: ${{ parameters.testResultsFiles }} | ||
| testRunTitle: ${{ parameters.testName }} | ||
| condition: succeededOrFailed() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| parameters: | ||
| artifactDirectory: $(System.DefaultWorkingDirectory) | ||
|
|
||
| steps: | ||
| - powershell: | | ||
| if ([Environment]::OSVersion.Platform -eq "Unix") { | ||
| $installer = Get-ChildItem -Path "${{ parameters.artifactDirectory }}/*" -Include *.pkg -File | ||
| } else { | ||
| $installer = Get-ChildItem -Path "${{ parameters.artifactDirectory }}\*" -Include *.vsix -File | ||
| } | ||
| if (![System.IO.File]::Exists($installer)) { | ||
| throw [System.IO.FileNotFoundException] "Installer not found in $artifactDirectory." | ||
| } | ||
| Write-Host "##vso[task.setvariable variable=XA.Provisionator.Args]$installer" | ||
| displayName: find installer and set provisionator variable | ||
|
|
||
| - task: provisionator@2 | ||
| inputs: | ||
| provisionator_uri: $(provisionator-uri) | ||
| github_token: $(GitHub.Token) | ||
| provisioning_script: $(XA.Provisionator.Args) | ||
| provisioning_extra_args: -vv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.