Move MacCatalyst to runtime-extra-platforms pipeline #64452
Conversation
|
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries Issue DetailsSince #63572 no longer happens in CI, we can promote MacCatalyst templates from the
|
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
The plist timeout looks like it has been fixed. Not sure if the MacCatalyst failures are totally solved. One of the functional tests failed under mysterious circumstances. |
… smoke tests to runtime.yml. Do the same to iOS_arm64 and tvOS_arm64 as they were missing previously.
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| buildConfig: Release | ||
| runtimeFlavor: mono | ||
| platforms: | ||
| - MacCatalyst_x64 |
There was a problem hiding this comment.
Since the runtime-extra platforms leg doesn't run on every PR, should we add a mac catalyst leg to the main runtime pipeline or the dev-innerloop pipeline that DOES NOT RUN TESTS, just to protect the vertical build and to make sure we don't break developer inerloop scenarios?
|
@premun Could you please look at MacCatalyst arm64 failure? It's the functional tests but they seem to fail randomly. |
|
So the issue is that on all Apple platforms (iOS, tvOS, MacCatalyst), we have problems detecting app's exit code. The original (Xamarin) way of testing these platforms is always via the XHarness TestRunner that is included in the apps. For all scenarios where we don't have the test runner (HelloiOS, functional tests), where we only fire up the app and wait for it to quit, XHarness is scanning MacOS syslogs for the event where the app terminated with non-zero code and parse the exit code from there. Unfortunately, this is not 100% reliable and sometimes we just don't see this line in the logs. We have no power over flushing of this log and there might be other reasons like MacOS cycling the log away when it reaches some size and starting a new one. We could implement some artifical waiting, though I it would be quite complicated. What I think would be better, since these tests only take few seconds, is to employ the DevWF retries and configure it to retry the run (can be on the same machine). I think this might help? |
That makes sense, thanks. What I need to do is just to update https://github.com/dotnet/runtime/blob/main/eng/test-configuration.json with an appropriate rule for the functional tests, right? |
|
@MaximLipnin something like that, yeah. But to be honest, I am not an expert in this area so not sure if this file goes where it needs and also if we can scope it to some platforms only for example. I might delegate you to the First Responder channel where people will be able to help you. |
|
@premun Thanks for helping! BTW, there would also be an option like adding |
|
Well, the whole Apple platform is one giant hack to be honest 😅 We have already a switch for a similar flow called |
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Doesn't look like retries happened on the MacCatalyst failure. @premun will we still see Attempt 1 and subsequent retries in the artifacts tab? |
|
Currently the test retry logic doesn't take into account our particular case 😢 Apart from it, another Alexander's suggestion: since xharness reads the whole /var/log/system.log, would it make sense to switch to calling the |
|
Looking at the docs (Test Retry Documentation.md), it seems that the problem is that our work item doesn't return exit code 0 (point 4 in How to get on board). It seems that DevWF is looking for failed tests, not failed work items. I tried the filtering before but that won't solve the problem as it only leaves out lines of log which is already missing what we need. The non-filtered log we are scanning is actually very short (~10 lines) as there's not much of system logging during the few seconds the app runs. For example for the last build from this PR it's here:
FWIW, the line we're looking for is this:
|
…n OSX 11.00 queue
Updated to keep arm64 in runtime-staging for now |
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Do we need to wait for #65068 which is going to update the arcade dependency to the latest version? |
|
@MaximLipnin there is a blocker and the dependency PR will take one more day to complete. However, we cannot merge this PR as it would break runtime due to some EOL errors. So we need to wait. |
# Conflicts: # eng/Version.Details.xml # eng/Versions.props # global.json
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…mitCalliNonBlittable test failing on MacCatalyst x64
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…amicMethodEmitCalliNonBlittable test on MacCatalyst x64 as the respective issue is addressed
|
/azp run runtime-extra-platforms |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Failures are unrelated, merging |
ghost
locked as resolved and limited conversation to collaborators
Since #63572 no longer happens in CI, we can promote MacCatalyst templates from the
runtime-stagingto theruntime-extra-platformspipeline.For now we will keep MacCatalyst arm64 legs in runtime-staging as they are not stable on OSX 11.00 queues (see #64452 (comment))