spi.yml information to skip certain configurations

[mattmassicotte]

hello!

I've recently been using SPI a lot more to make sure my packages build and work for all platforms. I noticed that one of my packages is always failing for older Swift versions on macOS, and this surprised me.

https://swiftpackageindex.com/ChimeHQ/Rearrange/builds

It turns out the reason why is Rearrange depends on a minimum macOS SDK of 12.0. However, Swift has no (non-hacky) way to handle this source-level dependency. So, building with older Xcodes will always fail.

I got to thinking, it might be nice to explain to your system that certain configurations can be skipped, since they aren't supported. Is this something I can communicate to you via a .spi.yml?

[daveverwer]

Ah, interesting. So you're saying that you could technically compile it with Xcode 13 but using Swift 5.5 instead of Swift 5.6. I actually just went to test this in Xcode and couldn't see the build setting to switch Swift versions. There's this one:

But that's not what we're looking for.

There's always going to be compromises displaying language compatibility as apps on Apple platforms are so closely tied to versions of Xcode. That's one reason we decided to compile using xcodebuild with specific versions of Xcode installed. What we're really measuring is "Compatibility with the version of Xcode that Swift X.X shipped with", but that's a little long-winded.

I got to thinking, it might be nice to explain to your system that certain configurations can be skipped since they aren't supported. Is this something I can communicate to you via a .spi.yml?

Is this just to save us time on builds that we know will fail? We've had suggestions along these lines before (for example, #727) and the complexity of adding that, while not huge, is more than just letting the builder machines fail. It may come to the point where that flips, but I don't think we're there yet.

[mattmassicotte]

Yeah, it really was just about saving wasted work. But, I totally see the complexity not being worthwhile.

Because you cannot describe SDK dependencies in Package.swift and also cannot do conditional compilation based on available SDK. The only way around this, I can think of is to use canImport or the Swift compiler version itself has a proxy for available SDK. I don't like either of these options, but have used them in the past.

Anyways, all this to say that I don't think this is an SPI issue. But! If you were to offer a minimum version/SDK option in .spi.yml, I'd use it :)

[daveverwer]

That's great feedback, thanks!

Rather than close this issue, I'll transfer it to discussions. That way it can remain open for others to contribute, but I don't think we'll commit to any changes just yet.

Thanks again and congratulations on the full set of green lights for Swift 5.5 🚀

[mattmassicotte]

Ok, this makes sense. And thank you :)

[mattmassicotte]

I actually had another thought on this, and I now have a potentially contentious question. What is the purpose of Swift minor revision compatibility? We know that Xcode does not allow you to select minor Swift versions. Is for people that are manually installing Swift toolchain outside of Xcode?

Conversely, I do see some benefit in an Xcode compatibility matrix. This is particularly helpful when new major releases of Xcode comes out, precisely because of the language and SDK changes it contains.

[finestructure]

There have been some syntax additions over the years that aren't backwards compatible. For instance one patch release, I believe it was 5.3, added someArray.map(\.someKeypath). If you use that, your package isn't 5.2 compatible.

And of course, some packages simply ship with tools-version: 5.3 in Package.swift! That will make it immediately unusable from Swift < 5.3 :)

But on the whole it is true that the patch version differences have become much smaller and thus many packages simply bump the tools-version to reduce test effort. We've had cases where a package claims to be 5.4+ but actually turned out to compile just fine on older Swift versions.

[mattmassicotte]

Thanks for this context! Of course, it does make sense that there will be significant language changes.

I guess I'm mostly surprised there's such an emphasis on minor 5 versions when they are pretty much entirely out of developers control. And, my expectation is that will not change with Swift 6. And, why not Swift 4, since Xcode lets you select it today?

I'm not advocating for change, just trying to understand!

[daveverwer]

I'm mostly surprised there's such an emphasis on minor 5 versions

We see developers submitting packages all the time that rely on the new features introduced in the latest Swift minor version release. Having the compatibility matrix show that it's only compatible with the very latest Swift is useful.

Why not Swift 4?

Mainly practicality and lack of real-world use. The last version of Xcode that shipped with 4.2 as default won't run on Big Sur or Monterey and you certainly can't submit apps to the store with it anymore. Now, that's not to say there's no one using it, but it's certainly not widely used these days.

[mattmassicotte]

Thank you, both, very much for explaining all this!

[orchetect]

Compatibility with the version of Xcode that Swift X.X shipped with

I wanted to tag onto this as I'm running into a similar issue now where this methodology is falling into a bit of an uncanny valley.

I have a package where it will build and test successfully for Swift 5.9 and later -- but requires Xcode 16 for it to build. Naturally the build result on SwiftPackageIndex then shows that it is only compatible with Swift 6.0, as Xcode 16 is "the version of Xcode that [it] shipped with." But the system will only use Xcode 15.4 for Swift 5.9 & 5.10. So this result is misleading, of course.

The SPI manifest has no way of specifying a specific Xcode version, but if it did this reporting inconsistency could be solved by requesting that all build tests be done with Xcode 16.0 (or at the very least, an Xcode version "latestRelease" specifier.) It would be great to have this increased flexibility.

I understand that there may be an argument against this, as having build results advertise compatibility with a Swift version should probably imply that all tooling capable of building that version should succeed.

But perhaps that points to a broader issue: should build reporting also matrix its Xcode versions for each Swift version, and display the Xcode version used to produce the build result? Not to say that we should overburden the automated system by testing every Xcode version. But if a build fails on Xcode 15.4, what if the build system subsequently tried Xcode 16.0? And whichever (if any) succeeds, that Xcode version is reported along with the Swift version.

My point about being able to specify an Xcode version in the SPI manifest would simply reduce this churn and allow a developer to specify the minimum Xcode version required to get a positive build result.

In that sense, it could be best of all worlds.

[finestructure]

Could you please reference the package to better understand what the build matrix currently looks like?

[orchetect]

The repo in question: https://swiftpackageindex.com/orchetect/OSCKit

In summary:

• Package supports Swift 5.9+, but requires Xcode 16 to compile
• Swift 6.0 passes for all platforms, naturally, because SPI is building with Xcode 16
• If Swift 5.9 and 5.10 were also built with Xcode 16, they would likewise show as passing. But they don't show as passing of course, because SPI uses Xcode 15.4 for those Swift versions.

[finestructure]

The reason I was asking is that we also run plain swift build builds which should be independent of Xcode versions. Looking at the 5.10 build of your package on the build details page there does seem to be a genuine build error with 5.10:

/Users/admin/builder/spi-builder-workspace/Sources/OSCKit/OSCServerUDPDelegate.swift:33:28: error: expected ',' separator
        oscServer: sending _OSCServerProtocol,
                           ^

If this build was successful we would show 5.10 compatibility on the package page.

[orchetect]

Ok that's good to know, thank you for checking. That must have been a result of the recent package refactor, so no worries then.