New macOS 13 runners default to iOS 17 beta simulators for Xcode 14.3.1 CI jobs.

[mluisbrown]

Description

The latest macOS 13 runners enable the iOS 17 beta simulators that come with Xcode 15.

This means that any Xcode test action which specifies the device name only (without an OS specifier) will run using the iOS 17 simulator by default instead of the latest iOS 16 simulator. This will cause many CI actions to fail as code has not yet been adapted for iOS 17.

Moreover, it is a failure that is not obvious and very hard to debug, as it's completely unexpected that an Xcode 14.3.1 CI job would use beta iOS 17 simulators from Xcode 15.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Current runner version: '2.307.1'
Operating System
macOS
13.5
22G74
Runner Image
Image: macos-13
Version: 20230801.2
Included Software: https://github.com/actions/runner-images/blob/macos-13/20230801.2/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20230801.2
Runner Image Provisioner
2.0.264.1

Unfortunately I cannot include a link to a build as it's a private repo.

Is it regression?

20230611.2

Expected behavior

iOS 17 beta simulators should only be selected if explicitly specified, instead of by default.

Actual behavior

iOS 17 beta simulators are selected by default, if no other OS specifier is used.

Repro steps

Run any Xcode CI job which selects a test device (eg using fastlane run_tests and selecting iPhone SE (3rd generation) as the test device.

For example, in our project we have the following Fastlane lane which we use for CI:

lane :tests do
    run_tests(project: "<redacted>.xcodeproj",
        scheme: "<redacted>",
        devices: ["iPhone SE (3rd generation)"],
        result_bundle: true
    )
end

This is now selecting an iOS 17 version of the simulator when previously it was selecting an iOS 16.4 simulator. The potential workaround is to specify the device as "iPhone SE (3rd generation) (16.4)" to fix the iOS version at 16.4, but that should not be necessary.

[joshuapoq]

This probably explains why our builds are now reaching an hour and timing out. It's taking from 2 to 11 minutes of time between each test bundle for our frameworks (24 test bundles). It was previously taking between 3 and 15 seconds.

[mluisbrown]

I have been able to confirm that adding the OS version in the device string, eg: "iPhone SE (3rd generation) (16.4)" is a workaround for this problem.

[ilia-shipitsin]

@mluisbrown , we added simulators for XCode-15 yesterday. MacOs-13 currently is shipped with simulators for XCode-14 and XCode-15 as well. Whichever is "default", I'm afraid someone will complain.

I'm not sure what would be an approach to make everyone happy, some people may want to be "iOS 17" as default, other "iOS 16.4".

Maybe, the best approach would be to specify explicitly, what do you think ?

[Vyazovoy]

I think this is the problem:

iOS 17 simulators hava to be installed only for Xcode 15. So if it is true that the default selected Xcode version is 14.2 then it should not see any iOS 17 simulators. The runtime should be marked as "unavailable". You can see that if you open Xcode 14.2 settings. So it is definitely a regression.

[mluisbrown]

@ilia-shipitsin as @Vyazovoy points out, the defaults for Xcode 14.3.1 should be the iOS 16 runtimes and the defaults for Xcode 15 should be the iOS 17 runtimes.
It makes no sense at all to use the iOS 17 runtimes with Xcode 14.3.1.

It's a regression as absolutely no one expects that Xcode 14.x would have iOS 17 runtimes installed.

[Baza207]

This does seem to be a regression, but it feels like it's maybe an Apple/Xcode issue. You can actually use the iOS 17 sim from Xcode 14.3.1, I tried it locally and it works fine. However because iOS 17 is still beta, it's buggy. 🙃 It's a bad default and should be fixed, but we should probably submit Apple feedbacks for it. Also, I could be wrong but I don't think it's possible to install sims for just one version of Xcode anymore. Locally for me at least they seem to sync between 14 and 15 of Xcode (add/delete in one place, it's available/gone from the other) and I'm guessing having a macOS-13 agent per Xcode major version would also not be on anyones list of requests.

The above mentioned work-around of adding (16.4) to the end of the sim name didn't work, but checking out the docs you can do something like iPhone 14,OS=16.4 and it seems to be working for us now. (Converts to -destination platform=iOS Simulator,name=iPhone 14,OS=16.4 for xcodebuild.)
Look for destinationSimulators here.

Really not the greatest, but I guess that's what happens when we work with a beta. Better case would be macOS wasn't still in beta, and this could be tested in a separate beta. But I'd much prefer having Xcode 15 beta available now, rather than after it's released. 🙈

Let's hope the next Xcode beta sorts things out. 🤞😅

[ilia-shipitsin]

@Vyazovoy , @mluisbrown I need your help.

do you have (locally) installed both XCode-14.X and XCode-15.X and all working in the expected way ?
if so, please share details how did you get it working.

for image generating we use https://github.com/actions/runner-images/blob/main/images/macos/provision/core/xcode.ps1#L30-L39

which invokes xcodebuild -downloadAllPlatforms in loop.
if you have other steps, we'll adjust our steps

[mluisbrown]

@ilia-shipitsin I just installed Xcode 15 beta to check this. Now all my iOS 16.4 simulators have (16.4) appended to the end of their description. So apparently with both versions installed you must specify the OS version. It's not clear what determines the default if no OS version is specified.

Nevertheless, I imagine this will be an issue for a lot of projects. At this moment I can't think of a good solution 😞