Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add APK test job to azure-pipelines.yaml #3042

Merged
merged 7 commits into from
Jun 18, 2019
Merged

Conversation

pjcollins
Copy link
Member

@pjcollins pjcollins commented Apr 29, 2019

Adds a Test stage to azure-pipelines.yaml. At present, this stage only
contains one test job, which builds and runs the following emulator tests:

Mono.Android_Tests
Mono.Android-Tests (AOT + LLVM)
Mono.Android-Tests (Profiled AOT)
Mono.Android-Tests (Bundle)
Mono.Android_TestsAppBundle
Mono.Android_TestsMultiDex
Xamarin.Android.JcwGen_Tests
Xamarin.Android.Locale_Tests
Xamarin.Android.Locale-Tests (AOT)
Xamarin.Android.Locale-Tests (Profiled AOT)
Xamarin.Android.EmbeddedDSO_Test
Xamarin.Forms.Performance.Integration.Droid 
Xamarin.Forms.Performance.Integration.Droid (AOT)
Xamarin.Forms.Performance.Integration.Droid (Profiled AOT)
Xamarin.Forms.Performance.Integration.Droid (Bundle)

Performance reporting has not been considered as part of this PR. Execution of the
BCL Test suite will also need to be added to the Test stage in the future.

Other Fixes:

* Support has been added for running APK tests against an XA installation

    We can now build and run Xamarin.Android.JcwGen_Tests, Mono.Android_Tests,
    and Xamarin.Forms.Performance.Integration.Droid tests against a system
    installation of Xamarin.Android.

* Produce an error when any test failures occur during instrumentation

    The instrumentation exit code value of `-1` indicates success, and any
    other values returned here should be reported as errors. A non `-1` value
    indicates that crash or test failure occurred.

* Produce an error if logcat timing data can't be parsed

    We'll now error out when parsing logcat for timing data if the running
    application died, or if we fail to detect application startup.

* Don't always report logcat stderr as a warning

    Adb invocations are prone to returning non zero exit codes even when
    they are successful. This can make our msbuild output a bit noisy[0] when
    they are invoked using the Adb.cs task. The new optional parameter
    `WriteOutputAsMessage` can now be provided to the adb task, so that we
    don't always write messages coming from stderr as msbuild warnings. This
    is used in our targets which wait for emulator startup to make it easier
    to detect real failures in test instrumentation and emulator startup.

    [0]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Android Debug Bridge version 1.0.40 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Version 28.0.2-5303910 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Installed as /Users/peter/android-toolchain/sdk/platform-tools/adb [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192]  [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:287] adb_query: host-serial:emulator-5570:features [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:137] _adb_connect: host:version [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:107] writex: fd=3 len=16 30303063686f73743a76657273696f6e 000chost:version [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:97] readx: fd=3 wanted=4 got=4 4f4b4159 OKAY [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:165] _adb_connect: return fd 3 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:197] adb_connect: service host-serial:emulator-5570:features [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
        /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]

@jonpryor
Copy link
Member

Build failure:

[2019-04-29T20:10:34.848Z]   /Users/builder/jenkins/workspace/xamarin-android-pr-pipeline-release/xamarin-android/build-tools/scripts/TestApks.targets(181,5): error MSB4044: The "Xamarin.Android.Tools.BootstrapTasks.BundleTool" task was not given a value for the required parameter "JarPath". [/Users/builder/jenkins/workspace/xamarin-android-pr-pipeline-release/xamarin-android/tests/RunApkTests.targets]

@pjcollins pjcollins changed the title Support running unit tests against a system installation Add APK test job to azure-pipelines.yaml May 31, 2019
@pjcollins
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jonpryor
Copy link
Member

jonpryor commented Jun 1, 2019

This presumably should be rebased atop commit 7768eec or later.

@pjcollins pjcollins force-pushed the system-test branch 2 times, most recently from 5a200e2 to c6460f6 Compare June 4, 2019 18:42
@pjcollins
Copy link
Member Author

pjcollins commented Jun 4, 2019

We've now got green AZP results against the .pkg installer produced by the Mac Build job for the following suites running in Xamarin.Android (Test APK Instrumentation):

  • Mono.Android_Tests
  • Mono.Android_TestsAppBundle
  • Mono.Android_TestsMultiDex
  • Xamarin.Android.JcwGen_Tests
  • Xamarin.Android.Locale_Tests
  • Xamain.Android.EmbeddedDSO_Test

I'll take this out of it's draft stage once grendello#1 and #3051 land, and this is rebased on those changes.

Support for the following test runs is still required:

  • Xamarin.Android.MakeBundle-Tests.csproj edit: this is not ran on device.
  • Xamarin.Forms.Performance.Integration.Droid.csproj
  • Mono.Android-Tests.csproj (AOT)
  • Mono.Android-Tests.csproj (Profiled AOT)
  • Mono.Android-Tests.csproj (Bundle)
  • Xamarin.Forms.Performance.Integration.Droid.csproj (AOT)
  • Xamarin.Forms.Performance.Integration.Droid.csproj (Profiled AOT)
  • Xamarin.Forms.Performance.Integration.Droid.csproj (Bundle)
  • Xamarin.Android.Locale-Tests.csproj (AOT)
  • Xamarin.Android.Locale-Tests.csproj (Profiled AOT)
  • Xamarin.Android.Bcl-Tests.csproj

The "Performance APK" tests should also be considered here, though perhaps that should land in a separate PR which also introduces a new way of reporting/visualizing that data. I believe these tests are used primarily for reporting data to the Jenkins performance plots.

@pjcollins
Copy link
Member Author

@jonpryor this is pretty close to being merge-able when you have a chance to look, just need to add BCL tests and swap out SDK provisioning with the new bootstrap tool when it lands. We may want to reconsider the yaml template usage and/or folder structure.

@pjcollins pjcollins force-pushed the system-test branch 3 times, most recently from 55b48fe to 79b48e9 Compare June 11, 2019 21:21
@pjcollins pjcollins force-pushed the system-test branch 5 times, most recently from 9f66efe to 9c9d76a Compare June 12, 2019 03:22
@jonpryor
Copy link
Member

The most recent commit which disables the jnimarshalmethod-gen.exe tests is odd; does /Users/peter/android-toolchain/jdk/jre/lib/jli/libjli.dylib exist?

@radekdoulik
Copy link
Member

The jnimarshalmethod-gen.exe issue should be fixed by #3235

The instrumentatione exit code value of `-1` indicates success, and any
other values returned here should be reported as errors. A non `-1` value
indicates that crash or test failure occurred.
Adb invocations are prone to returning non zero exit codes even when
they are successful. This can make our msbuild output a bit noisy[0] when
they are invoked using the Adb.cs task. The new optional parameter
`WriteOutputAsMessage` can now be provided to the adb task, so that we
don't always write messages coming from stderr as msbuild warnings. This
is used in our targets which wait for emulator startup to make it easier
to detect real failures in test instrumentation.

[0]
```
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Android Debug Bridge version 1.0.40 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Version 28.0.2-5303910 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Installed as /Users/peter/android-toolchain/sdk/platform-tools/adb [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192]  [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:287] adb_query: host-serial:emulator-5570:features [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:137] _adb_connect: host:version [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:107] writex: fd=3 len=16 30303063686f73743a76657273696f6e 000chost:version [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:97] readx: fd=3 wanted=4 got=4 4f4b4159 OKAY [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:165] _adb_connect: return fd 3 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:197] adb_connect: service host-serial:emulator-5570:features [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
  /Users/peter/source/pj/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4 [/Users/peter/source/pj/xamarin-android/tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj]
```
We'll now error out when parsing logcat for timing data if the running
application died, or if we fail to detect application startup.
[build] Test failures should not produce a publish error

MSBuild will error if any test failures occur during our APK tests, and
producing an extra error during the publish step only adds "noise" to
the overall build health.
@jonpryor jonpryor merged commit fd8336c into dotnet:master Jun 18, 2019
@pjcollins pjcollins deleted the system-test branch June 18, 2019 22:01
pjcollins added a commit to pjcollins/android that referenced this pull request Jun 25, 2019
Adds a `Test` stage to `build-tools/automation/azure-pipelines.yaml`.
At present, this stage only contains one test job, which builds and
runs the following emulator tests:

  * Mono.Android_Tests
  * Mono.Android-Tests (AOT + LLVM)
  * Mono.Android-Tests (Bundle)
  * Mono.Android-Tests (Profiled AOT)
  * Mono.Android_TestsAppBundle
  * Mono.Android_TestsMultiDex
  * Xamarin.Android.JcwGen_Tests
  * Xamarin.Android.Locale_Tests
  * Xamarin.Android.Locale-Tests (AOT)
  * Xamarin.Android.Locale-Tests (Profiled AOT)
  * Xamarin.Android.EmbeddedDSO_Test
  * Xamarin.Forms.Performance.Integration.Droid 
  * Xamarin.Forms.Performance.Integration.Droid (AOT)
  * Xamarin.Forms.Performance.Integration.Droid (Profiled AOT)
  * Xamarin.Forms.Performance.Integration.Droid (Bundle)

Note in particular that the BCL tests are currently absent.
Execution of the BCL Test suite will also need to be added to the
`Test` stage in the future.

Other Fixes:

  * Introductory support has been added for running `.apk` tests
    against a *system* Xamarin.Android installation.

    We can now build and run `Xamarin.Android.JcwGen_Tests`,
    `Mono.Android_Tests`, and
    `Xamarin.Forms.Performance.Integration.Droid` tests against a
    system installation of Xamarin.Android.

  * Produce an error when any test failures occur during
    instrumentation.

    The instrumentation exit code value of `-1` indicates success,
    and any other values returned here should be reported as errors.
    A non `-1` value indicates that crash or test failure occurred.

  * Produce an error if `adb logcat` timing data can't be parsed.

    We'll now error out when parsing `adb logcat` for timing data if
    the running application died, or if we fail to detect application
    startup.

  * Don't always report `adb logcat` stderr as a warning

    `adb` invocations are prone to returning non zero exit codes even
    when they are successful.  This can make our MSBuild output a bit
    noisy [^0] when they are invoked using the `<Adb/>` task.
    The new optional `Adb.WriteOutputAsMessage` property can now be
    provided to the adb task, so that we don't always write messages
    coming from stderr as MSBuild warnings.  This is used in our
    targets which wait for emulator startup to make it easier to
    detect real failures in test instrumentation and emulator startup.

[^0]: Example warnings:

      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Android Debug Bridge version 1.0.40
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Version 28.0.2-5303910
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] Installed as /Users/peter/android-toolchain/sdk/platform-tools/adb
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_trace.cpp:192] 
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:287] adb_query: host-serial:emulator-5570:features
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:137] _adb_connect: host:version
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:107] writex: fd=3 len=16 30303063686f73743a76657273696f6e 000chost:version
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:97] readx: fd=3 wanted=4 got=4 4f4b4159 OKAY
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:165] _adb_connect: return fd 3
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_client.cpp:197] adb_connect: service host-serial:emulator-5570:features
      …/xamarin-android/build-tools/scripts/TestApks.targets(68,5): warning : adb D 04-29 15:10:48  3527 42649 adb_io.cpp:81] readx: fd=3 wanted=4
@github-actions github-actions bot locked and limited conversation to collaborators Jan 30, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants