Add APK test job to azure-pipelines.yaml #3042
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pjcollins
commented
Apr 29, 2019
pjcollins
commented
Apr 29, 2019
|
jonpryor
reviewed
May 14, 2019
pjcollins
commented
May 31, 2019
pjcollins
changed the title
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
This presumably should be rebased atop commit 7768eec or later. |
pjcollins
force-pushed
the
system-test
branch
2 times, most recently
from
5a200e2
to
c6460f6
Compare
|
We've now got green AZP results against the .pkg installer produced by the
I'll take this out of it's draft stage once grendello#1 and #3051 land, and this is rebased on those changes. Support for the following test runs is still required:
The "Performance APK" tests should also be considered here, though perhaps that should land in a separate PR which also introduces a new way of reporting/visualizing that data. I believe these tests are used primarily for reporting data to the Jenkins performance plots. |
|
@jonpryor this is pretty close to being merge-able when you have a chance to look, just need to add BCL tests and swap out SDK provisioning with the new bootstrap tool when it lands. We may want to reconsider the yaml template usage and/or folder structure. |
build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/ProcessLogcatTiming.cs
Outdated
Show resolved
Hide resolved
...in.Android.LibraryProjectZip-LibBinding/Xamarin.Android.LibraryProjectZip-LibBinding.targets
Outdated
Show resolved
Hide resolved
...arin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.targets
Outdated
Show resolved
Hide resolved
pjcollins
force-pushed
the
system-test
branch
2 times, most recently
from
88bbd48
to
7655f90
Compare
jonpryor
reviewed
Jun 11, 2019
jonpryor
reviewed
Jun 11, 2019
jonpryor
reviewed
Jun 11, 2019
jonpryor
reviewed
Jun 11, 2019
pjcollins
force-pushed
the
system-test
branch
3 times, most recently
from
55b48fe
to
79b48e9
Compare
jonpryor
reviewed
Jun 12, 2019
...n.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/CreateAndroidEmulator.cs
Outdated
Show resolved
Hide resolved
pjcollins
force-pushed
the
system-test
branch
5 times, most recently
from
9f66efe
to
9c9d76a
Compare
|
The most recent commit which disables the |
|
The |
The instrumentatione exit code value of `-1` indicates success, and any other values returned here should be reported as errors. A non `-1` value indicates that crash or test failure occurred.
Adb invocations are prone to returning non zero exit codes even when they are successful. This can make our msbuild output a bit noisy[0] when they are invoked using the Adb.cs task. The new optional parameter `WriteOutputAsMessage` can now be provided to the adb task, so that we don't always write messages coming from stderr as msbuild warnings. This is used in our targets which wait for emulator startup to make it easier to detect real failures in test instrumentation. [0] ``` /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Android Debug Bridge version 1.0.40 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Version 28.0.2-5303910 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Installed as /Users/peter/android-toolchain/sdk/platform-tools/adb [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:287] adb_query: host-serial:emulator-5570:features [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:137] _adb_connect: host:version [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:107] writex: fd=3 len=16 30303063686f73743a76657273696f6e 000chost:version [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:97] readx: fd=3 wanted=4 got=4 4f4b4159 OKAY [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:165] _adb_connect: return fd 3 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:197] adb_connect: service host-serial:emulator-5570:features [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj] ```
We'll now error out when parsing logcat for timing data if the running application died, or if we fail to detect application startup.
[build] Test failures should not produce a publish error MSBuild will error if any test failures occur during our APK tests, and producing an extra error during the publish step only adds "noise" to the overall build health.
pjcollins
added a commit
to pjcollins/android
that referenced
this pull request
Jun 25, 2019
Adds a `Test` stage to `build-tools/automation/azure-pipelines.yaml`.
At present, this stage only contains one test job, which builds and
runs the following emulator tests:
* Mono.Android_Tests
* Mono.Android-Tests (AOT + LLVM)
* Mono.Android-Tests (Bundle)
* Mono.Android-Tests (Profiled AOT)
* Mono.Android_TestsAppBundle
* Mono.Android_TestsMultiDex
* Xamarin.Android.JcwGen_Tests
* Xamarin.Android.Locale_Tests
* Xamarin.Android.Locale-Tests (AOT)
* Xamarin.Android.Locale-Tests (Profiled AOT)
* Xamarin.Android.EmbeddedDSO_Test
* Xamarin.Forms.Performance.Integration.Droid
* Xamarin.Forms.Performance.Integration.Droid (AOT)
* Xamarin.Forms.Performance.Integration.Droid (Profiled AOT)
* Xamarin.Forms.Performance.Integration.Droid (Bundle)
Note in particular that the BCL tests are currently absent.
Execution of the BCL Test suite will also need to be added to the
`Test` stage in the future.
Other Fixes:
* Introductory support has been added for running `.apk` tests
against a *system* Xamarin.Android installation.
We can now build and run `Xamarin.Android.JcwGen_Tests`,
`Mono.Android_Tests`, and
`Xamarin.Forms.Performance.Integration.Droid` tests against a
system installation of Xamarin.Android.
* Produce an error when any test failures occur during
instrumentation.
The instrumentation exit code value of `-1` indicates success,
and any other values returned here should be reported as errors.
A non `-1` value indicates that crash or test failure occurred.
* Produce an error if `adb logcat` timing data can't be parsed.
We'll now error out when parsing `adb logcat` for timing data if
the running application died, or if we fail to detect application
startup.
* Don't always report `adb logcat` stderr as a warning
`adb` invocations are prone to returning non zero exit codes even
when they are successful. This can make our MSBuild output a bit
noisy [^0] when they are invoked using the `<Adb/>` task.
The new optional `Adb.WriteOutputAsMessage` property can now be
provided to the adb task, so that we don't always write messages
coming from stderr as MSBuild warnings. This is used in our
targets which wait for emulator startup to make it easier to
detect real failures in test instrumentation and emulator startup.
[^0]: Example warnings:
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Android Debug Bridge version 1.0.40
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Version 28.0.2-5303910
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192] Installed as /Users/peter/android-toolchain/sdk/platform-tools/adb
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_trace.cpp:192]
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:287] adb_query: host-serial:emulator-5570:features
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:137] _adb_connect: host:version
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:107] writex: fd=3 len=16 30303063686f73743a76657273696f6e 000chost:version
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:97] readx: fd=3 wanted=4 got=4 4f4b4159 OKAY
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:165] _adb_connect: return fd 3
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_client.cpp:197] adb_connect: service host-serial:emulator-5570:features
…/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48 3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a
Teststage to azure-pipelines.yaml. At present, this stage onlycontains one test job, which builds and runs the following emulator tests:
Performance reporting has not been considered as part of this PR. Execution of the
BCL Test suite will also need to be added to the
Teststage in the future.Other Fixes: