[package] xorg/system: Missing xvmc dependency when using prebuilt binaries

[Ahajha]

Description

Since #22081 (which removed xvmc) was merged, there have been periodic build failures in a few PRs, for example #21387. Even building a direct dependency of xorg can fail, for example here: #19897

I believe the issue that the package pulls the latest xorg package revision, but because the package ID for xorg is cleared, the latest package (which does contain the xvmc requirement) is pulled. When the main package is built, the dependency thinks it needs xvmc, but the main package pulls the latest recipe, which does not pull xvmc. So I think one or both of the following need to happen:

• Encode which packages are part of xorg as part of the package ID.
• Adjust the embed mode of xorg to always rebuild if the package ID changes.

A fix for this potentially be applied to other system packages.

Package and Environment Details

• Package Name/Version: xorg/system
• Operating System+version: Linux Ubuntu 22.04
• Compiler+version: GCC 11
• Conan version: conan 2.0.17 (haven't reproduced with Conan 1.x yet, unsure if it's a problem)
• Python version: Python 3.10.12

Conan profile

[settings]
os=Linux
arch=x86_64
compiler=gcc
compiler.version=11
compiler.libcxx=libstdc++11
build_type=Release

[conf]
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True
tools.build:compiler_executables={"c": "gcc-11", "cpp": "g++-11"}

Steps to reproduce

Important: Make sure your system does not contain xvmc from the system package manager. It should be safe to remove (see discussion on the PR that removed it from xorg/system, but I don't take responsibility if it breaks things.

I've removed all xorg and tk packages from my system:

conan remove tk -c
conan remove xorg -c

I can then reproduce the failure by navigating to recipes/tk/all and running

conan test test_package tk/8.6.10 -o */*:shared=True

I was not able to reproduce the problem with different approaches

• A conan create instead of a conan test (yet, it might be possible)
• Non-shared packages (this I suspect to be part of the problem, as shared libraries are not 'embedded' in Conan 2.0)
• Conan 1.x (yet, it might be possible)

(This is not a tk specific problem, just one that has been affected by this)

Logs

Click to expand log

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
*/*:shared=True
[conf]
tools.build:compiler_executables={'c': 'gcc-11', 'cpp': 'g++-11'}
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux
[conf]
tools.build:compiler_executables={'c': 'gcc-12', 'cpp': 'g++-12'}


======== Launching test_package ========

======== Computing dependency graph ========
Graph root
    tk/8.6.10 (test package): /home/alex/Documents/conan-center-index/recipes/tk/all/test_package/conanfile.py
Requirements
    brotli/1.1.0#d56d7bb9ca722942aba17369cb5c0519 - Cache
    bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4 - Cache
    expat/2.5.0#91e43e4544923e4c934bfad1fa4306f9 - Cache
    fontconfig/2.13.93#cd0da06660d36b0b4aed4944ba3528a4 - Cache
    freetype/2.13.2#c9ee90dc6f88356febac6fcdba0b2249 - Cache
    libpng/1.6.40#0337b7b87ac125307ed70fd064c31eea - Cache
    tcl/8.6.10#e21f620a22b2c2cdedb608c094e5ea15 - Cache
    tk/8.6.10#6a920c37309003cc5b210dc93ca418ad - Cache
    util-linux-libuuid/2.39.2#16074ebe9338e959492de3de88fe17ab - Cache
    xorg/system#c7039fe88e71c34398de15694304f0f0 - Cache
    zlib/1.3#06023034579559bb64357db3a53f88a4 - Cache
Build requirements
    gperf/3.1#a9335f8833b550f10f655fd243c72978 - Cache
    meson/1.2.2#aace9dcc1db58fa42ecb5292f724092d - Cache
    meson/1.2.3#7d2bfe1bfc0c4edd9579e812344d5490 - Cache
    ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21 - Cache
    pkgconf/2.0.3#f996677e96e61e6552d85e83756c328b - Cache
Resolved version ranges
    zlib/[>=1.2.11 <2]: zlib/1.3

======== Computing necessary packages ========
Requirements
    brotli/1.1.0#d56d7bb9ca722942aba17369cb5c0519:31b216f5bb11630e2c0601548f2a3d71f2558be9#43c8a9b3a8dc427d2dbcb60497cddf19 - Cache
    bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:582795513620c02434315fe471662206b3a90c44#fede95964d03837da15b665b9fc8badf - Cache
    expat/2.5.0#91e43e4544923e4c934bfad1fa4306f9:cb30f6fe72b581624d514ba67bfcf3b85327844a#473ccdfe4e45077157ca7e50c97d5837 - Cache
    fontconfig/2.13.93#cd0da06660d36b0b4aed4944ba3528a4:b871b671d2106209680b25f34fa7249904c34bfd#1eb576409924b5e60f20406f170dd796 - Cache
    freetype/2.13.2#c9ee90dc6f88356febac6fcdba0b2249:9278f7ec43a6ecbe20123191128f92bc7a2d1cc8#5b12a71a267600c4c83c71886c2a26a5 - Cache
    libpng/1.6.40#0337b7b87ac125307ed70fd064c31eea:84ae3df57199b67cf54d99de695e2f68bbb208a8#f3a11d6d6ef6f6876db5b26b98a8f786 - Cache
    tcl/8.6.10#e21f620a22b2c2cdedb608c094e5ea15:2348813ec5b5185df22084ef0b842611c301d967#5f61f66f6431c0b10ba0933a5ac2d33b - Cache
    tk/8.6.10#6a920c37309003cc5b210dc93ca418ad:c6334a53aa7ecde593ce05b2cbebe42f49cc8edd#e9d49a71a550d620cb37392b2f05e5f2 - Cache
    util-linux-libuuid/2.39.2#16074ebe9338e959492de3de88fe17ab:abe5e2b04ea92ce2ee91bc9834317dbe66628206#43b664355fe21d758d598d7a22dcf441 - Cache
    xorg/system#c7039fe88e71c34398de15694304f0f0:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    zlib/1.3#06023034579559bb64357db3a53f88a4:abe5e2b04ea92ce2ee91bc9834317dbe66628206#be744f18843976f205e00744293521b8 - Cache
Build requirements
Skipped binaries
    gperf/3.1, meson/1.2.2, meson/1.2.3, ninja/1.11.1, pkgconf/2.0.3
xorg/system: System requirements:  already installed
xorg/system: System requirements:  already installed

======== Installing packages ========
brotli/1.1.0: Already installed! (1 of 11)
bzip2/1.0.8: Already installed! (2 of 11)
expat/2.5.0: Already installed! (3 of 11)
util-linux-libuuid/2.39.2: Already installed! (4 of 11)
xorg/system: Already installed! (5 of 11)
zlib/1.3: Already installed! (6 of 11)
libpng/1.6.40: Already installed! (7 of 11)
tcl/8.6.10: Already installed! (8 of 11)
freetype/2.13.2: Already installed! (9 of 11)
fontconfig/2.13.93: Already installed! (10 of 11)
tk/8.6.10: Already installed! (11 of 11)
tk/8.6.10: Setting TK_LIBRARY environment variable: /home/alex/.conan2/p/tk030ece97b68c5/p/lib/tk8.6
tk/8.6.10: Setting TK_ROOT environment variable: /home/alex/.conan2/p/tk030ece97b68c5/p
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'cpp_info.names' used in: freetype/2.13.2, zlib/1.3, expat/2.5.0, bzip2/1.0.8, libpng/1.6.40, tcl/8.6.10, brotli/1.1.0, fontconfig/2.13.93
WARN: deprecated:     'cpp_info.build_modules' used in: freetype/2.13.2, bzip2/1.0.8
WARN: deprecated:     'env_info' used in: freetype/2.13.2, bzip2/1.0.8, tk/8.6.10, tcl/8.6.10, fontconfig/2.13.93
WARN: deprecated:     'cpp_info.filenames' used in: freetype/2.13.2
WARN: deprecated:     'user_info' used in: freetype/2.13.2

======== Testing the package ========
Removing previously existing 'test_package' build folder: /home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release
tk/8.6.10 (test package): Test package build: build/gcc-11-x86_64-release
tk/8.6.10 (test package): Test package build folder: /home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release
tk/8.6.10 (test package): Writing generators to /home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release/generators
tk/8.6.10 (test package): Generator 'CMakeToolchain' calling 'generate()'
tk/8.6.10 (test package): CMakeToolchain generated: conan_toolchain.cmake
tk/8.6.10 (test package): CMakeToolchain generated: CMakePresets.json
tk/8.6.10 (test package): CMakeToolchain generated: ../../../CMakeUserPresets.json
tk/8.6.10 (test package): Generator 'CMakeDeps' calling 'generate()'
tk/8.6.10 (test package): CMakeDeps necessary find_package() and targets for your CMakeLists.txt
    find_package(tk)
    target_link_libraries(... tk::tk)
tk/8.6.10 (test package): Generating aggregated env files
tk/8.6.10 (test package): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']

======== Testing the package: Building ========
tk/8.6.10 (test package): Calling build()
tk/8.6.10 (test package): Running CMake.configure()
tk/8.6.10 (test package): RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/home/alex/Documents/conan-center-index/recipes/tk/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/home/alex/Documents/conan-center-index/recipes/tk/all/test_package"
-- Using Conan toolchain: /home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release/generators/conan_toolchain.cmake
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc-11 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Conan: Target declared 'tk::tk'
-- Conan: Target declared 'tcl::tcl'
-- Conan: Target declared 'ZLIB::ZLIB'
-- Conan: Target declared 'Fontconfig::Fontconfig'
-- Conan: Target declared 'Freetype::Freetype'
-- Conan: Target declared 'PNG::PNG'
-- Conan: Target declared 'BZip2::BZip2'
-- Conan: Including build module from '/home/alex/.conan2/p/bzip2d6a4ddc21ef98/p/lib/cmake/conan-official-bzip2-variables.cmake'
-- Conan: Component target declared 'brotli::brotlicommon'
-- Conan: Component target declared 'brotli::brotlidec'
-- Conan: Component target declared 'brotli::brotlienc'
-- Conan: Target declared 'brotli::brotli'
-- Conan: Including build module from '/home/alex/.conan2/p/freet49a535f860e05/p/lib/cmake/conan-official-freetype-variables.cmake'
-- Conan: Target declared 'expat::expat'
-- Conan: Target declared 'libuuid::libuuid'
-- Conan: Component target declared 'xorg::x11'
-- Conan: Component target declared 'xorg::x11-xcb'
-- Conan: Component target declared 'xorg::fontenc'
-- Conan: Component target declared 'xorg::ice'
-- Conan: Component target declared 'xorg::xau'
-- Conan: Component target declared 'xorg::xaw7'
-- Conan: Component target declared 'xorg::xcomposite'
-- Conan: Component target declared 'xorg::xcursor'
-- Conan: Component target declared 'xorg::xdamage'
-- Conan: Component target declared 'xorg::xdmcp'
-- Conan: Component target declared 'xorg::xext'
-- Conan: Component target declared 'xorg::xfixes'
-- Conan: Component target declared 'xorg::xi'
-- Conan: Component target declared 'xorg::xinerama'
-- Conan: Component target declared 'xorg::xkbfile'
-- Conan: Component target declared 'xorg::xmu'
-- Conan: Component target declared 'xorg::xmuu'
-- Conan: Component target declared 'xorg::xpm'
-- Conan: Component target declared 'xorg::xrandr'
-- Conan: Component target declared 'xorg::xrender'
-- Conan: Component target declared 'xorg::xres'
-- Conan: Component target declared 'xorg::xscrnsaver'
-- Conan: Component target declared 'xorg::xt'
-- Conan: Component target declared 'xorg::xtst'
-- Conan: Component target declared 'xorg::xv'
-- Conan: Component target declared 'xorg::xxf86vm'
-- Conan: Component target declared 'xorg::xcb-xkb'
-- Conan: Component target declared 'xorg::xcb-icccm'
-- Conan: Component target declared 'xorg::xcb-image'
-- Conan: Component target declared 'xorg::xcb-keysyms'
-- Conan: Component target declared 'xorg::xcb-randr'
-- Conan: Component target declared 'xorg::xcb-render'
-- Conan: Component target declared 'xorg::xcb-renderutil'
-- Conan: Component target declared 'xorg::xcb-shape'
-- Conan: Component target declared 'xorg::xcb-shm'
-- Conan: Component target declared 'xorg::xcb-sync'
-- Conan: Component target declared 'xorg::xcb-xfixes'
-- Conan: Component target declared 'xorg::xcb-xinerama'
-- Conan: Component target declared 'xorg::xcb'
-- Conan: Component target declared 'xorg::xcb-atom'
-- Conan: Component target declared 'xorg::xcb-aux'
-- Conan: Component target declared 'xorg::xcb-event'
-- Conan: Component target declared 'xorg::xcb-util'
-- Conan: Component target declared 'xorg::xcb-dri3'
-- Conan: Component target declared 'xorg::xcb-cursor'
-- Conan: Component target declared 'xorg::uuid'
-- Conan: Component target declared 'xorg::sm'
-- Conan: Target declared 'xorg::xorg'
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release

tk/8.6.10 (test package): Running CMake.build()
tk/8.6.10 (test package): RUN: cmake --build "/home/alex/Documents/conan-center-index/recipes/tk/all/test_package/build/gcc-11-x86_64-release" -- -j16
[ 50%] Building C object CMakeFiles/example.dir/src/test_package.c.o
[100%] Linking C executable example
/usr/bin/ld: warning: libXvMC.so.1, needed by /home/alex/.conan2/p/tk030ece97b68c5/p/lib/libtk8.6.so, not found (try using -rpath or -rpath-link)
[100%] Built target example


======== Testing the package: Executing test ========
tk/8.6.10 (test package): Running test()
tk/8.6.10 (test package): RUN: ./example
./example: error while loading shared libraries: libXvMC.so.1: cannot open shared object file: No such file or directory

ERROR: tk/8.6.10 (test package): Error in test() method, line 26
	self.run(cmd, env="conanrun")
	ConanException: Error 127 while executing

[PerikiyoXD]

Possible affected PR: #21681 (comment)

[Ahajha]

Update: I spoke with @jcar87 about the issue, and he says it's also related to the fact that C3I uses a GCC built from source (I think only GCC11 at time of writing, just a guess from me). Notably, in most distributions, -Wl,--as-needed is default on, but it is not the default for GCC as a whole.

So this means that whenever we tell something to link with xvmc, it will always have that as a link requirement (whether it actually needs it or not). So when it gets pulled by something else, even a test package, it will be missing that dependency and thus fail.

The temporary solution is to add -Wl,--as-needed to binaries that would have that xvmc dependency encoded into it. This has already been done for a few recipes.

Perhaps we could also add this flag to the link flags for xorg, but I don't know if the recipe revision would work properly, since that's half the reason this is an issue in the first place.

[PerikiyoXD]

My naive questions are:

1. Why would we need to strip xvmc in the first place?
2. If it was because of RHEL9 dropping support for it, why don't we just drop support for RHEL9 instead of dropping support for all distros/package managers?
3. I've read that xvmc is not really used so much and is a legacy feature, but, it's a feature that can and might be needed, right?

[PerikiyoXD]

After reading thoroughly, I've understood the issue with xvmc linking.

[Ahajha]

Just in case, I'll add some extra info: xvmc can still be used if it's needed through the system package manager feature, it's just not used in any of CCI's recipes. xorg really serves as a convenience bundle of packages, so it's odd to include it by default.

[Ahajha]

@jcar87 It seems there haven't been as many failures lately, and my sdl_image PR (#22575) passes CI now, was there some internal CI change that would have fixed it? Or is it just a case of binaries being regenerated so the issue is slowly "fixing itself"?

[jcar87]

I think you're absolutely right on all accounts (I need to a confirm a couple of things):

• xvmc was listed as a component in package_info() up until https://github.com/conan-io/conan-center-index/pull/22081/files (whther it was needed or not, is irrelevant - the removal itself shouldn't have caused this :D )
• some recipes may have been built if the "old" xorg/system was pulled, but then then the binaries are used in a new docker container where the newer revision is pulled. Here however I think something isn't fully clear - I would need to see some logs, but it's curious .. things shouldn't just link against all xorg libraries, surely - and should have to opt-in to the ones they need. This could be a "build script" issue, or a "recipe" issue
• if "xvmc" is linked at link time, but not actually needed, then this would explain why the --as-needed works as a workaround
• which brings me to.. if --as-needed is needed, in 2024 I would say that's a bug of our gcc11 compiler or when we build binutils (remember we build the compiler from source)

So we have two potential issues:

• why are all xorg libraries being linked on the consumer side? - is this something that happens even on a regular distro outside of Conan, or is this a result of how the xorg/system recipe is laid out
• is the toolchain in our docker images used in CI not passing --as-needed as would be the default on all modern distros? this needs to be corrected

[ericLemanissier]

why are all xorg libraries being linked on the consumer side? - is this something that happens even on a regular distro outside of Conan, or is this a result of how the xorg/system recipe is laid out

AFAIU, this is a conan specific situation, which happens in two different ways:

• When a recipe directly requires xorg, AutotoolsDeps injects all the components of all the requirements into the link, no matter what the library being built asked for. I remember fixing this issue at least once by simply removing this generator. PkgConfigDeps was sufficient for the build, and has the advantage of returning the linking information per-dependency. (striked out because I'm not sure it actually worked, and I cannot remember which recipe it was)
• when a recipe A requires B, and B requires xorg, and B is not explicit in package_info that only specific components of xorg are actual requirements Then all components of xorg are are "transmitted" to A, so A links to all of xorg.