[service][rfc] Configurations built by the service

[danimtb]

In the latest release of changes to the pipeline of the service, we added support for new compilers, gcc 10 and clang 10 and 11 in Linux and apple-clang 12.0 in Mac OS. As stated in the supported platforms and configurations, due to the frequent updates of the apple-clang compiler, we decided (right from the beginning of the conan-center-index) to rotate the latest 3 versions of it to keep the pace of new versions and avoiding scalability issues spending too much time generating many binaries that will not be used. Therefore, we will stop generating packages for the apple-clang 9.1 compiler soon.

To better serve the users and to have a fast CI for contributors, it is necessary to be efficient and select the configurations built by the center carefully. Otherwise, generating more and more configurations will get very challenging over time. We are working on getting some stats about the usage of the packages in ConanCenter so we can have a better perspective around this.

However, we wanted to ask you about it to know what do you think might be the considerations to stop supporting old compiler versions and, for example, move all the compilers to that same rolling version policy, meaning that we could only generate the latest 3 versions of each compiler or maybe keep generating binaries for the oldest compiler version and the two latest (again, these are only examples).

With that, the time to build packages in PRs could be reduced significantly and would open the possibility to support other configurations that might be of interest to more ConanCenter users without facing enormous scalability challenges in the future due to the number of packages to be generated.

• Do you think that we are generating configurations that are not useful?
• Would you mind if we stop generating packages for old compiler versions?
• Would you like to have packages generated for other compilers or architectures?

Thanks

[Croydon]

Here is my personal take on it.

macOS Apple Clang

This is probably the easiest one. macOS developers seem to be used to targeting newer compiler versions. We also cut old Apple Clang versions rather regularly in Bincrafters and I never heard anyone complain. I think the current policy of only generating the latest three compiler versions is fine.

Windows MSVC

CCI supports 2015, 2017 and 2019 currently. I wouldn't be entirely surprised if VS 2019 becomes an evergreen IDE similar like Windows 10 has become evergreen. VS is now much more modular and ships regularly new features.

Thus said, the discussion is more about the compiler versions and not the IDE version (which is also better reflected in the new MSVC compiler setting in Conan).

I would say that supporting the three latest MSVC versions and rotate out the oldest one if a new one gets released, should be fine. Identical to the macOS situation in its handling, but with the huge difference that there are not that often new MSVC versions (if you don't count minor releases). So, if you compare the years of support for an individual MSVC version, it would be much longer supported than the support for an individual Apple Clang version, which is probably fitting, given that Windows development is more conservative.

Linux Clang

As far as I know, Clang is not the default compiler on any major Linux distribution right now. Hence, I would not mind a schema with a fixed number of supported versions and rotating out the oldest ones.

Currently, Clang 3.9+ is supported. With 9 compiler versions in total.

Clang 3.9 got released in Sep 2016. Clang 4 in March 2017 and Clang 5 in September 2017 - just to name the oldest (Source).

They are releasing every 6 months a new version which does strengthen the argument to only support a fixed number of compiler versions. Basically, we can think in years of support which makes it less abstract.

Example: We want to support all Clang versions of the last 4 years, hence we support the latest 8 Clang versions.

This is somewhat arbitrary and more a feeling, but 4 years of support would sound good to me.

Linux GCC

Given that GCC is the default compiler in all major Linux distributions (of which I am currently aware), this might be the most complicated case regarding which versions are meaningful to support.

Currently, GCC 4.9+ is supported. With 7 compiler versions in total.

GCC 4.9 got releases in April 2014 and since then one release per year, which is their release schedule (Source).

Meaning, if we support the same amount of GCC compiler versions as we do with Clang versions, we cover twice as many years of support for an individual compiler release. Cutting off the old GCC versions is probably the most painful, hence supporting GCC versions longer than Clang versions might make sense.

Again somewhat arbitrary and more a feeling, but 8 years of support a.k.a the latest 8 GCC versions might sound good - for instance, RHEL versions are also supported for 8 years, if you don't count the extended support - and it would be the same amount of supported compiler versions than for Clang.

---

The issue description goes more in the direction of reducing what we already have, while my suggestion goes more in the direction of not letting it grow further.

Another thing which might be considered in this regard: Maybe it does not have to be an all-or-nothing thing. Example: We want to support the newest 8 GCC versions, but only for the newest 4 versions we build all configurations and for the others only a subset of those. This would reduce the number of total builds further without cycling out older compilers (too) quickly.

---

Reducing the number of builds also ties in with #2813. With the differences, that this issue is about the general configuration support and #2813 would allow recipe specific limitations in cases it makes sense, i.e. something like #2813 would also help to bring CI time down for the heaviest recipes we have.

---

When it comes to new potential configurations, I'm most interested in Clang for Windows, but at this point, I'm not even sure if having binary packages for this makes much sense as it aims at being ABI compatible with MSVC. Having test coverage via CI might be great though.

[a4z]

Since packages can be rebuild and cached and archived, removing exotic compiler is a good idea.

I think everyone in professional environment ends up to rebuild everything them self anyway. (and if they don't they should)

So the binary packages from the cci are maybe only a time to hello world reducer for new conan user and hobbyist.
This is important, but I think even for new users it is understandable that software needs to be build.
(possible adopting the docs to use packages that build fast can do a fast time to hello world for others)

Therefore, imho, everything beside most recent/ common compiler can be removed, since I think most new users will either have current compiler, or understand that software needs to be compiled 1 time.

So maybe keep the last 2 of every kind (MSVC, XCode/Clang, latests clang on Linux) and on Linux the 2 most popular, might be current ubuntu and latest ubuntu LTS

[jgsogo]

I'll try to highlight one of the main issues here (from my perspective): this is not a matter of storage, if that was the problem we could remove packages from older revisions and we will save many gigabytes.

The problem is that we can only be sure that the package builds for a given compiler if we actually build it (once we build, we will store). I totally agree that building from sources is not a big handicap... if the recipe works.

[dmn-star]

Would you like to have packages generated for other compilers or architectures?

Yes Android & iOS

[a4z]

Yes Android & iOS

for this we might first have this conan-io/conan#7876 resolved

and then, there are so may settings, like os.api_level/ Android os.version/ iOS , where it might be good to agree, so that looks a bit more complicated

[prince-chrismc]

Would you like to have packages generated for other compilers or architectures?

• The new MSVC compiler! https://github.com/conan-io/conan/pull/8201/files
• FreeBSD would be lovely too

I would also really appreciate Alpine binaries but conan-io/conan#3972 🤷 Is it doable for only CCI? prince-chrismc/user-management@881b28d#diff-981783d07c49301cd9b135809fe286aaed8d8c2e2dcfe391663bb4f1c37f0721R53-R54

[n-bes]

I think that main goal of cci is contain verified recipes. Binaries are optional. So toolchains (environments) count should be maximal. Recipes can have different support level:

package \ toolchain	gcc 4	gcc 10
boost	recipe only	recipe and binaries

[chgans]

What are the plans regarding android? Most (all?) the libs i tried to install had to be built from source

[jgsogo]

Hi, @chgans . We are not generating binaries for Android (yet), so it is expected that you need to build from sources all the libraries. We probably will, but there are some steps to take before, this is the plan:

• We need to create new docker images, using the same base image for all the compilers, that way we are sure that all the applications (build-requires) can run in any of them. @uilianries is doing a huge work here, hopefully soon they will be ready.
• We need to start using the two-profiles approach in this repo. We have started to use two profiles to generate MacOS M1 binaries (yes, we have started to build them and it is our first cross-building configuration in this repo), it will let us gain confidence with this approach. @SSE4 implemented it and most of the libraries (up to now I've tested ~200) are building out of the box.
• Then we need to regenerate all the binaries using the new docker images. It is very likely that we will need to drop some clang versions (older ones).
• We will activate two-profiles for all the configurations, and we will start adding other configurations, Android will be some of the first ones to consider.

It will take time, but we have already started 😊

---

BTW, can I ask you about your experience building the libraries for Android? Do they work (most of them)? Are you using our android_ndk package as a build-requires? Thanks!

[chgans]

Hi,

We would like to switch our deps management from a completely inhouse adhoc system to conman w/artifactory.
So i'm only investigating for now.

I'm using the official Conan docker container for Android. Took me some time to get my setup right (build/host profiles), but i'm finally able to get this working on armv8:

[requires]
boost/1.76.0
benchmark/1.5.3
mosquitto/1.6.12
protobuf/3.15.5
rapidjson/1.1.0
sqlite3/3.35.5
# libtorrent/1.2.3
openssl/1.1.1k
gtest/1.10.0
# libusb/1.0.23
# qt/5.15.2
# To be created:
# libs3
# libmarnav/0.10.0

[options]
qt:opengl=es2
libusb:enable_udev=False
boost:i18n_backend_iconv=libiconv

Not bad! libtorrent problem is #5666, libusb problem is #5675, trying to get Qt now!

If we do the switch to conan, then we'll have to maintain our own repo as we target custom embedded Linux, Android, Windows and Linux 64 bits. all of these in 32 and 64 bits! (:cry:)

I haven't attempted to do actual builds for Android, that will be next step in my investigation.

We'll use our own container, eg. Full Android SDK/NDK, conan, jfrog, .. with prebuilt Qt (and other monsters).

Update: Qt fails, as it needs the SDK, I wonder if Conan's Qt for Android supports multi-arch (Qt does)

[dmn-star]

Android builds on CCI are long overdue, almost every recipe update ends in tedious troubleshooting. For example now again boost #5747

[jgsogo]

FYI: conan-io/conan-docker-tools#267, it's also relevant for this issue.

[jgsogo]

Hi! After a long journey, we have prepared some new docker images that we are planning to use to generate the packages in ConanCenter (it won't happen soon, we need to define how to transition to those images). You can read more about it here:

• blogpost: https://blog.conan.io/2021/08/09/Modern-docker-images.html
• readme: https://github.com/conan-io/conan-docker-tools/tree/master/modern

Main implications:

• if using those images, we won't generate GCC 4.9 and Linux/Clang versions older than 10.

---

I wanted to share this with you here first, because we are considering removing NOW those compilers and configurations that won't be generated in the future. Also, it is more or less aligned with previous comments. Does anyone sees a big problem if we remove those old compiler versions?

Thanks for your feedback.... and please, spread the word about this topic.

[ericLemanissier]

• we won't generate GCC 4.9 and Clang versions older than 10.0.0.

Do you mean apple-clang older than 10.0.0 ? Or will there really be only 2 versions of clang (10 and 11) ?

[SSE4]

Do you mean apple-clang older than 10.0.0 ? Or will there really be only 2 versions of clang (10 and 11) ?

this is about regular clang. that's all about Linux docker images right now.

[jgsogo]

• we won't generate GCC 4.9 and Clang versions older than 10.0.0.

Do you mean apple-clang older than 10.0.0 ? Or will there really be only 2 versions of clang (10 and 11) ?

For macOS we will keep generating the latest three major versions (10, 11 and 12). We have started to collect metrics and we will have information about the configurations being requested, that will help us to decide if we should stop generating some of them... or if we should add others. For gcc-4.9 and linux-clang it is more a technical reason given the docker images we will be using.

[jgsogo]

We already have some data from the previous 7 days in ConanCenter. These are install requests for the compilers we are building right now:

Windows:

compiler
Visual Studio 14	0,81%
Visual Studio 15	11,06%
Visual Studio 16	88,13%

Linux

compiler
GCC 4	0,81%
GCC 5	4,59%
GCC 6	0,57%
GCC 7	26,39%
GCC 8	15,30%
GCC 9	35,81%
GCC 10	16,53%

compiler
clang 3	0,19%
clang 4	0,05%
clang 5	0,39%
clang 6	4,70%
clang 7	9,96%
clang 8	3,65%
clang 9	20,60%
clang 10	32,24%
clang 11	28,22%

Macos

compiler
apple-clang 10	1,11%
apple-clang 11	21,95%
apple-clang 12	76,94%

---

Given these numbers IMHO we can start thinking about deprecating some old compilers and be a bit more aggressive with it. Wdyt?

[jgsogo]

(Linux context) My biggest concern (given the technical reasons for docker containers) would be clang 9 (20,60%), why is it so common? Is it the default for Linux distributions?

[a4z]

the Linux GCC Number is slightly surprising, but on the other hand, gcc 7.5 is the default compiler on ubuntu 18.04 LTS , and gcc 9 on 20.04 , so no big surprise either ....

I do not think there is a Linux dist with default clang, so that is surprising

[ericLemanissier]

clang9 is the default clang on openSUSE Leap 15.2

[dmn-star]

I do not think there is a Linux dist with default clang, so that is surprising

OpenMandriva :-)

---

Llvm Download for Linux ( clang version <-> distro )
https://pkgs.org/download/llvm

[jgsogo]

Checked clang versions (by default I meant the one that is installed with apt-get install clang):

• Ubuntu 18.04: clang 6
• Ubuntu 20.04: clang 10

[dmn-star]

@jgsogo see my comment

[chgans]

We have a massive code base stuck on clang-9. Linux people interested in building their own stuff with conan are IMHO unlikely to use default distro compiler. If you like default stuff, then you would use apt instead of conan to get your lib and dev packages installed.
As well, the compiler survey doesn't include Android. Do you guys have any figure regarding Android?

[jgsogo]

Hi @chgans, we are trying to get information (numbers) and reaching out so we know better the impact that any decisions could have.

We will try to keep building configurations as long as they are used, but on the other hand, we should consider resources and the time it takes to build each pull-request and provide feedback to contributors. We also need to take into account human resources and our own technical limitations (we have found problems to build the new docker images for clang 9).

As well, the compiler survey doesn't include Android. Do you guys have any figure regarding Android?

(This is not from a survey, these are actual package installations - Sorry if I misunderstood and it was a false friend)

About new configurations (Android, iOS,...), it is something we are considering and it is very related to that resource-configurations balance. Some people think that it is more valuable to have configurations far away from each other than to have every single GCC version, and probably recipes themselves are more valuable if they manage properly different cross-building scenarios,... although they are not tested for some compiler versions or flags. Of course, here there are different opinions and necessities, and we cannot satisfy them all.

---

I don't think there is such a strong correlation between using the default compiler in the distro and Conan usage. Conan lets you manage the graph of dependencies, change requirements, versions, options,... from one project to another. It decouples the build from the underlying system libraries. ConanCenter probably provides libraries that are not available via the system package manager (if there is a system package manager available)...

About your clang-9, @chgans , are you (your company) using the packages directly from ConanCenter? We tend to think that companies build from sources their own packages and upload them to an internal Conan server, because they usually need to apply some additional patch. I'm glad to hear it is not your case and you can use packages directly (probably you are also contributing whenever you need a new version o some build to fix).

[chgans]

Our big project stuck on clang9 is not using conan, but we would like to move to conan. And yes we will have to use our own conan repo since we do cross-builds for a in-house embedded Linux OS. And we have started a new line of Android based products. We actually have to support a big range of products, some stuck with (very) old gcc.
Nonetheless we do Linux, MacOSX and Windows build of this software, mainly for convenience of developers.

[eigenwhat]

Regarding additional platforms, I'm biased since I don't need to support Android in my work projects, but compiling to iOS is not much different than compiling for Mac OS, especially when it comes to CMake projects. If I were to pick one or the other, I'd look at iOS first since it's already better supported in other parts of the ecosystem. Mobile development is huge and there is plenty of demand for cross-platform C++ development, so I think it's a strong candidate. It also covers some important cross-compilation corner cases, like a library that has both build-time and run-time dependencies on the same package but with different settings and options. (e.g. proj requires the binary from sqlite in the build context and the library in the host context.)

I don't know how C3I currently handles building for MacOS, but if there are Mac agents it shouldn't be much trouble to include iOS as well. Since these are libraries and not apps you shouldn't need to deal with all the code signing hell. That's all one step further down the line. For the most part we're accustomed to building dependencies from source anyhow (e.g. CocoaPods) so for me at least my principle concern is that the recipes actually work moreso than having binaries available on ConanCenter. It may even be best not to publish iOS binaries on ConanCenter because there are some unique ABI concerns with bitcode on ARM that Conan currently doesn't handle all too well but which can be mitigated in an in-house CI environment.

[PolyPik]

Hi I would like to ask why are Windows MinGW GCC build configurations not supported by the Conan Center build service?

I know that Visual Studio may be an industry standard for Windows C and C++, but MinGW is also significant for providing the open source GCC toolchains to Windows developers.

Also I know that one can provide the --build option to conan install to get one's needed packages built using MinGW, but certain packages can have a lengthy build process and it will be the downstream project developer who will be bearing the burden of that wait time. So for the benefit of the developers who use the MinGW toolchains, I would like to see Windows MinGW GCC added as a configuration supported by the build service.

[fuchse-ego]

Hi I would like to ask why are Windows MinGW GCC build configurations not supported by the Conan Center build service?

I know that Visual Studio may be an industry standard for Windows C and C++, but MinGW is also significant for providing the open source GCC toolchains to Windows developers.

Also I know that one can provide the --build option to conan install to get one's needed packages built using MinGW, but certain packages can have a lengthy build process and it will be the downstream project developer who will be bearing the burden of that wait time. So for the benefit of the developers who use the MinGW toolchains, I would like to see Windows MinGW GCC added as a configuration supported by the build service.

Especially since it is not free anymore for companies (which want to distribute internal or closed source software using conan). Are there technical advantages using only msvc under windows?

[bldrvnlw]

@Croydon mentioned that the policy on macos os is the latest threes versions.

As of May 2024 this would imply that there should be apple-clang 13, 14 and 15 for an artifact such as zlib. In fact there is on apple-clang 13. Is there a reason for this?

[Croydon]

The policy changed.

Docs are currently not really up to date, but for macOS it already shows that only apple-clang 13 is supported https://github.com/conan-io/conan-center-index/blob/master/docs/supported_platforms_and_configurations.md

[bldrvnlw]

Was this change made because apple-clang 13 is ABI compatible with apple-clang 14 and 15? If so is it possible to alter the compiler version for the dependencies? (That's probably a question for the conan forum I realize)

[bldrvnlw]

To answer my own question James indicates a method to solve this