[mono] iOS Test runner #34976

EgorBo · 2020-04-14T22:31:28Z

It's currently WIP, it doesn't use runtime packs yet but it's already possible to run any test suite (well, any without remote-executor based tests inside, see dotnet/arcade#5129 such tests usually have <IncludeRemoteExecutor>true</IncludeRemoteExecutor> inside csprojs). It uses dotnet/xharness test runner, copies everything including the whole BCL into a test folder and runs the AOT/Bundle generation.

Steps to run a test suite:

# make sure everything is built for iOS (can be arm64 instead of x64 for devices and fullAOT)
./build.sh -c Release -os iOS -arch x64 -subset Mono+Libs+Libs.Tests /p:DisableCrossgen=true

# TestName can be any test suite, e.g. System.Collections.Tests
./dotnet.sh msbuild src/mono/mono.proj /t:BuildAppleTestBundle \
/p:Configuration=Release /p:TestName=System.Buffers.Tests \
/p:TargetOS=iOS /p:TargetArchitecture=x64

for x64 it should build everything and launch an iphone simulator with tests. For arm64 devices it needs xharness tool (with mlaunch inside - it's not yet there) and a provisioning profile.
If you want to try arm64 now you can run the script above ^ (replace x64 with arm64) and open the xcode project (generated) and deploy to a personal device with auto-generated provisioning. Some tests might crash on arm64, e.g.:

Assertion at /Users/egorbo/prj/runtime-3/src/mono/mono/metadata/class.c:1845, 
condition `method->is_inflated' not met

or some missing *aotdata stuff. Arm64 needs some work.

I guess the expected workflow is the following:

cd src/libraries/System.Buffers/tests
dotnet publish -c Release -r ios-x64
dotnet tool xharness test ios --app path/to/app/bunndle.app

EgorBo · 2020-04-14T22:32:48Z

/cc @steveisok @akoeplinger @marek-safar @mandel-macaque

safern · 2020-04-14T23:31:48Z

Adding my 2 cents here. I think we should unify this with our current testing infrastructure/targets to run libraries tests instead of diverging paths by having some stuff under mono and some other under libraries.

We could use a lot of what we have by adding some targets/commands to execute the app bundler and invoking the tests and what not.

steveisok · 2020-04-15T00:20:33Z

@safern At the very least, I would expect this PR to drive towards that.

safern · 2020-04-15T00:28:11Z

@safern At the very least, I would expect this PR to drive towards that.

Yes, I think it should as a first step.

marek-safar · 2020-04-15T08:39:45Z

I think we should unify this with our current testing infrastructure/targets to run libraries tests instead of diverging paths by having some stuff under mono and some other under libraries.

@safern what would you like to be moved and to where? For now, only very specific libraries configurations are supported but the same infrastructure will be used for all other tests in dotnet/runtime so not sure if it makes sense to move it libraries as it's not libraries specific.

src/mono/msbuild/AppleAppBuilder/Xcode.cs

marek-safar · 2020-04-15T08:42:54Z

src/mono/mono.proj

+    <Error Condition="!Exists('$(TestDir)')" Text="TestDir=$(TestDir) doesn't exist" />
+    <Error Condition="!Exists('$(BclDir)')" Text="BclDir=$(BclDir) doesn't exist" />
+
+    <Copy SourceFiles="@(AppleTestRunnerBinaries)" DestinationFolder="$(TestDir)\%(RecursiveDir)" />


I don't think this should be done here

src/mono/msbuild/AppleTestRunner/AppleTestRunner.csproj

src/mono/msbuild/AppleTestRunner/AppleTestRunner.cs

src/mono/msbuild/AppleTestRunner/AppleTestRunner.csproj

safern · 2020-04-15T15:56:19Z

@safern what would you like to be moved and to where? For now, only very specific libraries configurations are supported but the same infrastructure will be used for all other tests in dotnet/runtime so not sure if it makes sense to move it libraries as it's not libraries specific

I’d like to have this hooked up under /eng/testing and have the targets and all underlying logic there. Even though it will not be used by all configurations what we’re running are libraries tests. I don’t like all this manual copying of files and duplicating path calculation and all things involved here. We could hook it up in the infra we already have to run tests and everything would be much simpler. Instead of having to run specific mono targets I envision this as dotnet build /t:test /p:TargetOS=iOS /p:TargetArchitecture=x64 on a test project and the test project will be executed on the simulator.

Or build libs.tests -test -os iOS -arch x64 and all the tests will be executed on the simulator. I want to be consistent on how we run tests for all configurations this makes dev experience easier rather than having to remember different commands and what project to call etc.

I think this is fine as is for the purpose of this PR, but we should eventually hook it up as o suggest above. I can definitely help with that, I started taking a look at how all this works yesterday and I think it is possible.

marek-safar · 2020-04-15T16:31:32Z

@steveisok could you ensure @safern concerns are addressed

safern · 2020-04-15T16:32:52Z

cc: @ViktorHofer @danmosemsft

steveisok · 2020-04-15T16:46:29Z

@safern After talking w/ Egor & Alex, I'd like to land this PR so that we have a runner to use. Then in a separate PR, we can hook into what you outlined. We have additional ongoing work that I'd like to come together as well (in-tree runtime pack, AOT targets, etc).

safern · 2020-04-15T16:49:06Z

@safern After talking w/ Egor & Alex, I'd like to land this PR so that we have a runner to use. Then in a separate PR, we can hook into what you outlined.

Yeah I don't want to block this PR. I just wanted to raise my concerns as a heads up that we should envision hooking it up the same way we already run libraries tests and of course happy to help with doing that.

src/mono/mono.proj

src/mono/msbuild/AppleTestRunner/AppleTestRunner.cs

danmoseley · 2020-04-16T01:22:08Z

src/mono/msbuild/AppleTestRunner/AppleTestRunner.cs

+    }
+}
+
+public class SimpleTestRunner : iOSApplicatonEntryPoint, IDevice


iOSApplicatonEntryPoint -- is this a typo? iOSApplicationEntryPoint

ah, it's in dotnet/xharness. I didn't know about this repo 😸

cc @mandel-macaque ^

also, while you are there could you please add events for "test is started", "test is finished (with result)" ?

@danmosemsft I'm known for my typos, I'll get that fixed! @EgorBo dotnet/xharness#36

danmoseley · 2020-04-16T01:38:35Z

src/mono/msbuild/AppleAppBuilder/AppleAppBuilder.cs

@@ -34,7 +34,7 @@ public class AppleAppBuilderTask : Task
    /// Path to Mono public headers (*.h)
    /// </summary>
    [Required]
-    public string MonoInclude { get; set; } = ""!;
+    public string MonoRuntimeHeaders { get; set; } = ""!;


What does ""! achieve here? Is this the "null-forgiving" operator?

These properties are guaranteed to be non-null always (due to [Required] attribute) the compiler forces me to do inline init to leave them non-nullable. I wonder how @stephentoub solves such cases.

Co-Authored-By: Dan Moseley <danmose@microsoft.com>

EgorBo · 2020-04-16T09:48:37Z

src/mono/msbuild/AppleAppBuilder/Templates/main-console.m

+        CGRect caretRect = [logLabel caretRectForPosition:logLabel.endOfDocument];
+        [logLabel scrollRectToVisible:caretRect animated:NO];
+        [logLabel setScrollEnabled:NO];
+        [logLabel setScrollEnabled:YES];
    });


Some hack to automatically scroll to the bottom in UITextView

why do you need to toggle setScrollEnabled? is the scrollRectToVisible not enough?

@akoeplinger I don't know but without it doesn't work properly, I tried different hacks from SO this one works as expected (https://stackoverflow.com/questions/19124037/scroll-to-bottom-of-uitextview-erratic-in-ios-7/20989956#20989956)

safern · 2020-04-16T18:23:09Z

src/mono/msbuild/AppleTestRunner/AppleTestRunner.cs

@@ -62,7 +62,7 @@ public static async Task<int> Main(string[] args)

    protected override IEnumerable<TestAssemblyInfo> GetTestAssemblies()
    {
-        foreach (string file in testLibs)
+        foreach (string file in testLibs!)


Why is the null forgiving operator needed here?

@safern I forgot to remove it

Will fix it in the next PR

akoeplinger · 2020-04-17T12:53:31Z

runtime (Installer Build and Test Linux_x64 Debug) passed on AzDO, the status didn't make it to GitHub: https://dev.azure.com/dnceng/public/_build/results?buildId=603747&view=logs&j=0d59b9e3-002e-58b9-0bdb-d3cd1e08882b&t=90f7f7ef-163c-557b-3338-db1b69e0bef4&l=21

Prototype

f9a926e

EgorBo requested a review from marek-safar as a code owner April 14, 2020 22:31

Dotnet-GitSync-Bot added the area-Infrastructure label Apr 14, 2020

EgorBo added area-Infrastructure-mono os-ios Apple iOS and removed area-Infrastructure labels Apr 14, 2020

dotnet deleted a comment Apr 14, 2020

marek-safar reviewed Apr 15, 2020

View reviewed changes

akoeplinger reviewed Apr 15, 2020

View reviewed changes

src/mono/msbuild/AppleTestRunner/AppleTestRunner.csproj Outdated Show resolved Hide resolved

Move Microsoft.DotNet.XHarness.Tests.Runners to versions.prop

4f0100e

EgorBo added 2 commits April 15, 2020 20:53

Update dotnet-tools.json

07bc7cd

Address feedback

7148281

akoeplinger approved these changes Apr 15, 2020

View reviewed changes

EgorBo added 3 commits April 16, 2020 01:26

use dylibs for simulator

625b9cc

add aotdata (for aot)

fadf3e5

Fix scrolling

7b30a9c

danmoseley reviewed Apr 16, 2020

View reviewed changes

src/mono/mono.proj Outdated Show resolved Hide resolved

danmoseley reviewed Apr 16, 2020

View reviewed changes

src/mono/msbuild/AppleTestRunner/AppleTestRunner.cs Outdated Show resolved Hide resolved

danmoseley reviewed Apr 16, 2020

View reviewed changes

EgorBo and others added 3 commits April 16, 2020 12:32

Update src/mono/msbuild/AppleTestRunner/AppleTestRunner.cs

15a4e65

Co-Authored-By: Dan Moseley <danmose@microsoft.com>

Address feedback

b04617b

Fix nullability in SimpleTestRunner

7af4801

EgorBo commented Apr 16, 2020

View reviewed changes

EgorBo added 3 commits April 16, 2020 15:39

Update Microsoft.XmlSerializer.Generator.Tests.csproj

780b1fb

Add SystemNative_InitializeTerminalAndSignalHandling to pal_log

2a5f6bf

Undo Console.ios changes

2a61ec9

mandel-macaque mentioned this pull request Apr 16, 2020

Create events to notify that the test execution has been completed dotnet/xharness#36

Closed

EgorBo added 2 commits April 16, 2020 17:38

fix typo

487d9d0

Disable data-outfile again

c26cf8b

jaredpar mentioned this pull request Apr 16, 2020

Restore failing on our projects #35057

Closed

safern reviewed Apr 16, 2020

View reviewed changes

safern mentioned this pull request Apr 17, 2020

Several tests fail on mono with mono_threads_pthread_kill: pthread_kill failed with error 11 #32377

Closed

EgorBo changed the title ~~[WIP] iOS Test runner~~ [mono] iOS Test runner Apr 17, 2020

EgorBo merged commit 3468524 into dotnet:master Apr 17, 2020

ghost locked as resolved and limited conversation to collaborators Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mono] iOS Test runner #34976

[mono] iOS Test runner #34976

EgorBo commented Apr 14, 2020 •

edited

Loading

EgorBo commented Apr 14, 2020

safern commented Apr 14, 2020

steveisok commented Apr 15, 2020

safern commented Apr 15, 2020

marek-safar commented Apr 15, 2020

marek-safar Apr 15, 2020

safern commented Apr 15, 2020

marek-safar commented Apr 15, 2020

safern commented Apr 15, 2020

steveisok commented Apr 15, 2020 •

edited

Loading

safern commented Apr 15, 2020

danmoseley Apr 16, 2020

danmoseley Apr 16, 2020

EgorBo Apr 16, 2020

EgorBo Apr 16, 2020

mandel-macaque Apr 16, 2020

danmoseley Apr 16, 2020

EgorBo Apr 16, 2020

EgorBo Apr 16, 2020

akoeplinger Apr 16, 2020

EgorBo Apr 16, 2020

safern Apr 16, 2020

EgorBo Apr 16, 2020

EgorBo Apr 17, 2020

akoeplinger commented Apr 17, 2020

[mono] iOS Test runner #34976

[mono] iOS Test runner #34976

Conversation

EgorBo commented Apr 14, 2020 • edited Loading

EgorBo commented Apr 14, 2020

safern commented Apr 14, 2020

steveisok commented Apr 15, 2020

safern commented Apr 15, 2020

marek-safar commented Apr 15, 2020

Choose a reason for hiding this comment

safern commented Apr 15, 2020

marek-safar commented Apr 15, 2020

safern commented Apr 15, 2020

steveisok commented Apr 15, 2020 • edited Loading

safern commented Apr 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akoeplinger commented Apr 17, 2020

EgorBo commented Apr 14, 2020 •

edited

Loading

steveisok commented Apr 15, 2020 •

edited

Loading