New MacOS13 image is slower than previous one.

[christiangal-indi]

Description

Using the same Xcode 14.3.1 and the same code on the 2 different images of MacOs13 results in very different compile and test run times.

Image Version: 20230417.1
Complete pipeline 25-33 min

Image Version: 20230924.1
Complete pipeline 45-60(timeout) min

Here you can see the difference from "Wednesday"

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Agent name: 'Azure Pipelines 10'
Agent machine name: 'Mac-1696247891010'
Current agent version: '3.225.0'
Operating System
macOS
13.6
22G120
Runner Image
Image: macos-13
Version: 20230924.1
Included Software: https://github.com/actions/runner-images/blob/macos-13/20230924.1/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20230924.1

Is it regression?

Yes, Image Version: 20230417.1

Expected behavior

The run should take around 30 min.

Actual behavior

The run takes more than 40 min, and sometimes, fail with a 60min timeout.

Repro steps

You could run a simple task to compile a project in IOS and you would notice the difference.

Using this example as a guide:
https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/xcode?view=azure-devops

basically this yml here:
pool:
vmImage: 'macos-13'

steps:

• task: Xcode@5
  inputs:
  actions: 'build'
  scheme: ''
  sdk: 'iphoneos'
  configuration: 'Release'
  xcWorkspacePath: '**/*.xcodeproj/project.xcworkspace'
  xcodeVersion: 14.3.1

[amseddi]

Looks similar to #8023.
Did you try specifying the run target to iOS 16 simulator?

[Alexey-Ayupov]

Hello, @christiangal-indi. Could you please provide minimal repro steps?

[christiangal-indi]

Looks similar to #8023. Did you try specifying the run target to iOS 16 simulator?

Yes, those times are using the same simulator with version (16.4). That's why I have created another issue as it is not the same case. This is about the images itself and not the simulator that was selected.

[christiangal-indi]

Hello, @christiangal-indi. Could you please provide minimal repro steps?
I have updated the description.

[andriyslyusar]

We can see very similar result in increasing execution time in our project. Average time to builds and run unit test increase from 25 to 60 minutes.

[luis-ifolor]

We're also seeing a huge increase of execution time in our project just by switching to this image and trying to run the tests against iOS 17. Tests timeouts are happening where they previously didn't and the simulator process is also sometimes timing out.

[Vyazovoy]

I'm seeing similar issues: either much slower or timed out

[honkmaster]

We've noted the same: tests often fail to start, potentially linked to issue #8472. When they run, they're at least 25% slower. But if we're being honest: Isn't this more of a problem with Xcode than with the runner images?

[christiangal-indi]

We've noted the same: tests often fail to start, potentially linked to issue #8472. When they run, they're at least 25% slower. But if we're being honest: Isn't this more of a problem with Xcode than with the runner images?

Not sure, it is the same xcode running in 2 different runner images. Something has changed between the 2 images.

[GioPalusa]

I am too seeing this issue where our pipelines, where they have doubled in time and are now timing out on most runs.

[mluisbrown]

We are also seeing a 2-3x increase in CI time with macos-13 and Xcode 15 running tests with iOS 17 simulators. I've also tried the macos-13-xl runner and it is equally slow.

Using the same macos-13 runner with Xcode 14.3.1 and iOS 16.4 simulators was fine. Running the CI test suite locally )on an M1 Pro MacBook Pro) with Xcode 15 and iOS 17 is not noticeably slower than with Xcode 14.3.1 and iOS 16.4.

This is blocking our ability to move to using Xcode 15.

[gredman]

Same, but I am hoping it’s caused by this issue, fixed in 15.0.1 (and 15.1):

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)

[mluisbrown]

Running the CI test suite locally )on an M1 Pro MacBook Pro) with Xcode 15 and iOS 17 is not noticeably slower than with Xcode 14.3.1 and iOS 16.4.

I was wrong about that, just assuming that 6-7 minutes to run all tests was "normal" on my machine. Turns out that with Xcode 15.0.1 and iOS 17.0.1 simulators tests run in 2-3 minutes, so I was seeing the issue locally too, it just wasn't as obvious.

[TomaszLizer]

We are also seeing a 2-3x increase in CI time with macos-13 and Xcode 15 running tests with iOS 17 simulators. I've also tried the macos-13-xl runner and it is equally slow.

Using the same macos-13 runner with Xcode 14.3.1 and iOS 16.4 simulators was fine. Running the CI test suite locally )on an M1 Pro MacBook Pro) with Xcode 15 and iOS 17 is not noticeably slower than with Xcode 14.3.1 and iOS 16.4.

This is blocking our ability to move to using Xcode 15.

@mluisbrown Quite similar on my side: macos-13 runner with Xcode 14.3 - build and tests takes ~35-40 minutes.
Switching to Xcode 15 ends in timeout after 90 minutes and based on logs Fastlane has barely started building app (less than half of the way)

Same, but I am hoping it’s caused by this issue, fixed in 15.0.1 (and 15.1):

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)

@gredman I fear not based on the above 😢

[mluisbrown]

@TomaszLizer our app is building fine, and in reasonable time, with Xcode 15. It's the testing that is killing it. There are times when there's a 15 minute wait whilst it appears to be doing nothing at all!

[TomaszLizer]

Oh, interesting 🤔
Fingers crossed then 😃

[honkmaster]

Xcode 15.0.1 RC is published.

https://developer.apple.com/documentation/xcode-release-notes/xcode-15_0_1-release-notes

[christiangal-indi]

We are also seeing a 2-3x increase in CI time with macos-13 and Xcode 15 running tests with iOS 17 simulators. I've also tried the macos-13-xl runner and it is equally slow.
Using the same macos-13 runner with Xcode 14.3.1 and iOS 16.4 simulators was fine. Running the CI test suite locally )on an M1 Pro MacBook Pro) with Xcode 15 and iOS 17 is not noticeably slower than with Xcode 14.3.1 and iOS 16.4.
This is blocking our ability to move to using Xcode 15.

@mluisbrown Quite similar on my side: macos-13 runner with Xcode 14.3 - build and tests takes ~35-40 minutes. Switching to Xcode 15 ends in timeout after 90 minutes and based on logs Fastlane has barely started building app (less than half of the way)

Same, but I am hoping it’s caused by this issue, fixed in 15.0.1 (and 15.1):

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)

@gredman I fear not based on the above 😢

We have seen problems in testing with the combination of Xcode15 and Fastlane, also on local. We were getting error 65 on tests execution and taking fastlane out solved the problem. So fastlane is the one that need to take action here. Test that on your side.

[mluisbrown]

@christiangal-indi I seriously doubt that Fastlane is the problem here, and have seen no evidence to support that. Xcode 15 has a known bug which causes this, which is fixed in Xcode 15.0.1 (and 15.1). Once 15.0.1 is released and made available on a GH runner that will be the first thing to test to see if/how things improve. I would only consider looking at other causes if that doesn't help.

[svenmuennich]

Using the same macos-13 runner with Xcode 14.3.1 and iOS 16.4 simulators was fine. Running the CI test suite locally )on an M1 Pro MacBook Pro) with Xcode 15 and iOS 17 is not noticeably slower than with Xcode 14.3.1 and iOS 16.4.

We successfully use Xcode 15 on macOS 13 (M1) runners by pairing them with iOS 16.4 simulators. They are not available by default, but creating them is easy, e.g.:

# Make iOS 16.4 simulator runtime available to Xcode 15
sudo mkdir -p /Library/Developer/CoreSimulator/Profiles/Runtimes
sudo ln -s /Applications/Xcode_14.3.1.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS\ 16.4.simruntime
# Create an iOS 16.4 simulator
xcrun simctl create "iPhone 14 Pro (16.4)" "com.apple.CoreSimulator.SimDeviceType.iPhone-14-Pro" "com.apple.CoreSimulator.SimRuntime.iOS-16-4"

With this combination (Xcode 15, iOS 16.4 simulator) we see similar test durations as with Xcode 14.

[alexrozanski]

@svenmuennich this was really useful, thank you! interestingly enough, it looks like simply linking the iOS 16.4 simulator to /Library/Developer/CoreSimulator/Profiles/Runtimes/ makes the 16.4 simulators available to Xcode — when I forgot to pass iPhone 14 Pro (16.4) as a device argument to fastlane it picked up the iOS 16.4 iPhone SE simulator which it hadn't previously.

[mluisbrown]

The macos-13-xl runner now includes Xcode 15.0.1 and unfortunately tests are still taking more than 2-3x as long as with Xcode 14.3.1 and iOS 16.x simulators.

[ilia-shipitsin]

@mluisbrown , do tests run fast on your local mac ? if so, we would like to do the same.

[mluisbrown]

@ilia-shipitsin yes, tests run fast on my local Mac, which is an M1 Pro MacBook Pro with 32GB of RAM. We run tests using Fastlane with the command bundle execute fastlane tests

Test startup and completion on my Mac:

[19:09:12]: Running Tests: ▸ Touching <redacted>.xctest (in target '<redacted>' from project '<redacted>')
[19:10:33]: ▸ All tests
[19:10:33]: ▸ Test Suite <redacted>.xctest started
...
[19:11:40]: ▸ All tests
[19:11:41]: ▸ All tests
[19:11:41]: ▸ 2023-10-23 19:11:41.838 xcodebuild[42965:20947029] [MT] IDETestOperationsObserverDebug: 149.749 elapsed -- Testing started completed.

Test startup and completion on GitHub using macos-13-xl runner with Xcode 15.0.1:

[09:01:41]: Running Tests: ▸ Touching <redacted>.xctest (in target '<redacted>' from project '<redacted>')
[09:09:19]: ▸ All tests
[09:09:20]: ▸ Test Suite <redacted>.xctest started
...
[09:50:17]: ▸ All tests
[09:50:51]: ▸ All tests
[09:50:52]: ▸ 2023-10-23 09:50:52.410 xcodebuild[13716:52437] [MT] IDETestOperationsObserverDebug: 2950.128 elapsed -- Testing started completed.

There's an inexplicable almost 8 minutes delay between Running Tests and ▸ All tests which in my Mac only takes about 80s.

I didn't analyse the whole log but this was another thing that stood out:

On my Mac:

[19:10:55]: ▸ SettingsRowsTests
[19:10:55]: ▸     ✓ aligned (0.006 seconds)
[19:10:55]: ▸     ✓ heading (0.008 seconds)
[19:10:55]: ▸     ✓ correctly (0.009 seconds)
[19:10:55]: ▸     ✓ correctly (0.009 seconds)
[19:10:55]: ▸     ✓ correctly (0.010 seconds)
[19:10:55]: ▸     ✓ correctly (0.011 seconds)

On GitHub:

[09:19:28]: ▸ SettingsRowsTests
[09:19:28]: ▸     ✓ aligned (0.020 seconds)
[09:19:28]: ▸     ✓ heading (0.076 seconds)
[09:19:28]: ▸     ✓ correctly (0.049 seconds)
[09:19:28]: ▸     ✓ correctly (0.024 seconds)
[09:19:28]: ▸     ✓ correctly (0.028 seconds)
[09:21:26]: ▸     ✓ correctly (117.639 seconds)

That 6th test took 117.6s on GH and only 0.011s on my Mac.

Obviously the test suite always ran slower on GH than on my Mac. On my Mac the full fastlane tests command (pull dependencies, build app from scratch, test) took about 7 minutes. On GH with Xcode 14.3.1 and iOS 16.4 it usually takes around 30 minutes, whereas with Xcode 15.0.1 and iOS 17.0.1 it took 75m.

Our Xcode 14.3.1 CI runs on the macos-13 runner whereas I ran the Xcode 15.0.1 tests on the macos-13-xl runner (which is the only one that has Xcode 15.0.1 at the moment).

[TomaszLizer]

Tried running fast lane scan with latest runner image (macOS-13 20231018.14) and performance drop is brutal with Xcode 15(.0.1) (previously it was taking 30-40 minutes), with Xcode 15 it end with timeout after 90 minutes.
Workaround that @svenmuennich shared seems to be working pretty well :)
First try and success after ~40 minutes (not sure why those builds vary that much).

[gabe4coding]

On my side, on our workflow i'm adding this pkg installation that seems to work: yeetd
Now the tests are running with success and with the same speed (or better), prior Xcode 15. This doesn't need to switch to iOS 16 simulator.

I found out about it on this article by Bitrise.

I'm running various tests to make sure that the workaround is stable, but i'm starting to see the light!

Here how to use it:

 - name: Install yeetd
   run: |
       wget https://github.com/biscuitehh/yeetd/releases/download/1.0/yeetd-normal.pkg
       sudo installer -pkg yeetd-normal.pkg -target /
       yeetd &

[mluisbrown]

That's concerning, since Apple specifically made the 15.0.1 patch release to fix this issue 😞 AFAIK it was the only change from 15.0!

[gabe4coding]

There is another mention on Xcode 15.1 Release Notes

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)

[mluisbrown]

It mentions the same Radar numbers as in the 15.0.1 Release notes. But I hope that having it again means "No, for real this time" 😄

[honkmaster]

But if we look closely: It only mentions the extended time to launch, not the extended run time.

[gabe4coding]

Right, and that is actually fixed, i don't see it anymore.
I sent a Feedback to Apple, mentioning this issue and this resolution, that could give them an hint, if they don't already know.
In the meanwhile yeetd works perfectly.

[larry-cariq]

On my side, on our workflow i'm adding this pkg installation that seems to work: yeetd Now the tests are running with success and with the same speed (or better), prior Xcode 15. This doesn't need to switch to iOS 16 simulator.

@gabe4coding Wow, this worked for me on macos-13 with Xcode 15.0.1. Thanks for the tip!

[TomaszLizer]

I stumbled on this issue once again.
Initially I have used iOS 16.4 simulators install workaround. That seemed fine, but now one of repos in my organisation shows same issue - stuck on Loading after running xcodebuild command through Fastlane.
I then tried using yeetd workaround but with mixed success (mostly timeouts: Xcode 14.3.1 build took ~11 mins, timeout set to 15 mins, with Xcode 15 it is either never ending loading, or timeout during build).

I am not sure whether that is actually Xcode15 issue since we are running iOS 16.4 simulators. I can also see quite big variability of build times in CI (eg ~7-13 minutes for particular repo).

[gabe4coding]

It looks like that Xcode 15.1 Beta 2 brings improvements on this. Also confirmed on Circle CI thread.
But on GHA when we will have it? 😢

[TomaszLizer]

Looks like I have some sort of solution.
What helped was migrating to macOS-13-xlarge GitHub runner (arm64 M1 based).
Not only stability improved, but it seems to be more cost (eg builds are 2-3 times faster).
So right now on my side I have:

• fastlane
• yeetd
• Xcode 15.0.1
• macos-13-xlarge runner

Will go with final update after we go through all the repos, but that is promising :)

[gabe4coding]

That is exactly my config, but with xcodebuild ;)

[TomaszLizer]

That was the missing piece :D
I suppose there is something broken in the virtualisation process when runner is on x86 machine (I remember someone mentioning that and even creating bug report to apple, but can't remember when/where).

[icecoffin]

For us, yeetd + Xcode 15.0.1 / iPhone 15 Pro + macos-13-xl is ~30% slower than Xcode 14.2 / iPhone 14 Pro + macos-12-xl. Both setups are using fastlane. It's not getting stuck or timing out, it's just that compile and test times are slower.

[eirikvaa]

Running with the following on GitHub Actions:

• Xcode 15.1
• iOS 17.0.1 simulator
• Using yeetd

The first run (without caching) and the second (with caching) align with jobs running on Xcode 14.3.1. It looks promising, but needs more testing!

[mluisbrown]

I've run a few jobs using Xcode 15.1 with the iOS 17.2 simulator, and without using yeetd:

With the macos-13-xl (Extra Large) runner I got times comparable with those using Xcode 14.3.1 and iOS 16.4.
With the regular `macos-13' runner times are on average a little slower than with Xcode 14.3.1, but only in the order of 10-20% slower.

Given that the XL runners cost twice as much as the regular ones, it's not worth the cost to use them.

[TomaszLizer]

@mluisbrown Large runners are twice as expensive but in all my usage those are at least twice as fast - that is pure saving in time and money :)

[mluisbrown]

@TomaszLizer as I mentioned above, the XL runner was only, at best, about 20% faster running CI for the project I'm working on, so the extra cost is not worth it. If they were consistently at least twice as fast then obviously it would make sense to use them.

[derbaxter-suncor]

Was there ever a solution to this?

[derbaxter-suncor]

And not just a workaround

[Vyazovoy]

From my experience the only "solution" is to use macOS 14 😁

[Cheesebaron]

Not available yet in AzDo :'(

[lm2s]

The only "solution" that I know of, is to use M1 agents because they're faster. But those are only available in Github and not available for Azure DevOps. Also there's no information nor timeline or even if they'll be available in Azure DevOps, it's a joke at this point

[mluisbrown]

The original problem was that the image was just slower, even using the same version of Xcode. My problem started with Xcode 15, which did indeed have bugs that caused CI slowness, to the point that it was unfeasible for us to use Xcode 15.

This was "solved" once Xcode 15.1 was available on the GH runners, although CI runs still took considerably longer than before. Around 50 minutes instead of 30 minutes, but it was usable. It's unclear to me how much of the remaining problem is the macOS runners or Xcode 15. Using the xl runners (which use M1 Macs) the performance does improve quite a bit, but the runners also cost 2x as much. If they cost 2x as much but ran in half the time, that would be a a no brainer, but IME the time saving is not 2x, so you get a bit faster, but pay more.

We have since migrated to using Xcode Cloud for our CI as GH Actions for iOS CI became untenable for several reasons:

• much more expensive
• too slow
• runners far too slow to adopt new versions of Xcode

[jbiel]

We believe this issue is affecting our Azure DevOps pipelines using MS hosted agents. Some Fastlane builds that used to take ~9 minutes now time out after 2 hours. We need to move to Xcode 15/iOS SDK 17; macOS 13 is required for that. We're using the default version of Xcode (15.0.1) and are going to try the latest available on AzDO (15.2) to see if there's improvement. Not having ARM64 agents and/or macOS 14 available on AzDO seems to be putting us in a tight spot.

[mluisbrown]

Do your Fastlane builds also include testing? The slowness bug in Xcode 15 affected mainly running tests, not building.

Xcode 15.0.1 did not fix this slowness bug, but 15.1 did.