[bug] requires of a dependency of a dependency in build context not visible

[maximilianmuehlbauer]

Describe the bug

conan 2.3.0

For building Matlab Code with s-functions, certain packages have to be propagated with build=True, visible=True traits. It seems however that the dependencies of such packages in build context disappear at some point.

As minimal example, we have the following dependency tree:
pkgb -> libpkgb -> libpkgb_dependency

where pkgb contains the sfunction which should only reside in build context (think about a cross build scenario, the sfunction has to execute in Matlab only at build time and generates code), this sfunction depends on libpkgb (some shared library) and the generated code also depends on libpgkb so that one has to be available in both contexts. libpkgb_dependency (another shared library) is just a dependency of libpkgb, so it also has to be available in both contexts.

This s-function is then consumed by pkga (e.g. a Simulink model) as follows:
pkga (host context) -> libpkgb
pkga (build context) -> pkgb with visible=True trait.

It now seems that the dependencies of pkgb in build context disappear when pkga is consumed by pkga_consumer in host context. This makes it impossible to e.g. aggregate Matlab packages with s-functions for opening Matlab inside a VirtualBuildEnv to e.g. generate code.

Please see below zip file for exemplary conanfiles and the requirement traits used (maybe those are also incomplete?).

How to reproduce it

Execute ./create_all.sh:
pkga has libpkgb in its build requirements as expected:

pkga/sth@sth/sth: Forced build from source
Requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkga/sth@sth/sth#5de6164f0525938b5251b51dce86bb93:a9fea83b8cb8528171eec79a036ec70971964fe6 - Build
Build requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkgb/sth@sth/sth#64a4a8d5db953083748e775d6a8e6ac2:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache

pkga_consumer has libpkgb not anymore in its build requirements:

pkga_consumer/sth@sth/sth: Forced build from source
Requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkga/sth@sth/sth#5de6164f0525938b5251b51dce86bb93:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
    pkga_consumer/sth@sth/sth#36e174720d405f5bf14b41af15132a1e:163876653e2bf77f8d6c9a484c16abf689d068ed - Build
Build requirements
    pkgb/sth@sth/sth#64a4a8d5db953083748e775d6a8e6ac2:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
Skipped binaries
    libpkgb/sth@sth/sth, libpkgb_dependency/sth@sth/sth

dependency_propagation_build.zip

[memsharded]

Hi @maximilianmuehlbauer

Thanks for your report.

To summarize the graph would be:

pkga_consumer -(requires)-> pkga(shared) --------(requires)--------->libb(shared) -(requires)-> libb_dep(shared)
                                    \--(build-require) ->pkgb(shared)-->/

I'd like to start with a recommendation: build=True, visible=True will be really discouraged in most cases. I have already had some experience trying it, and I would advice against this.

If the intention is that the pkga_consumer needs some tool-requires for itself, because it needs such tool requires to build itself, then that means that pkgb (shared-library) should be a direct dependency for pkga_consumer and it shouldn't be inherited from pkga

It would be also good to understand how shared libraries can be used as "tool-requires" (build=True), as this puts such libraries in the "build" context, so the binaries might actually be incompatible (cross-compilation) or inconsistent/incorrect (building for debug will still bring Release for the build context by default).

Thanks very much for your reproducible code, I will use it to make a test, and understand what could be happening.

[maximilianmuehlbauer]

Thanks for the quick reply!

I'd like to start with a recommendation: build=True, visible=True will be really discouraged in most cases. I have already had some experience trying it, and I would advice against this.

Yes, Matlab is a very special case... one might have a model that depends on a Simulink library (like a C++ header-only lib), that depends on a library, ... and at some point there's an sfunction. For building the final model, all of that is needed in the build environment which is where we run Matlab to generate the C++ code.

If the intention is that the pkga_consumer needs some tool-requires for itself, because it needs such tool requires to build itself, then that means that pkgb (shared-library) should be a direct dependency for pkga_consumer and it shouldn't be inherited from pkga

Yes, essentially that is what we want; however making an explicit tool requirement from pkga_consumer to the sfunction is hard as the name, version etc. of that sfunction would need to stay in sync with a possibly long dependency tree in between. We would like to give Matlab developers the possibility to depend on a library which might come with such sfunction in some place without explicitly having to specify that sfunction.

It would be also good to understand how shared libraries can be used as "tool-requires" (build=True), as this puts such libraries in the "build" context, so the binaries might actually be incompatible (cross-compilation) or inconsistent/incorrect (building for debug will still bring Release for the build context by default).

Yes, those libraries will be incompatible (and cross-compiling actually works with pkga as root). The sfunction in pkgb might need libpkgb to generate code (but it's a normal requires of pkgb -> to my understanding, it should land in build context as soon as pkg is put in build context in that case) while the generated code links against libpkgb for possibly a different architecture.

[memsharded]

I have been having a look and so far it seems the build=True trait has higher priority than the visible=True one, so build-requires are still not being fully propagated. I will keep investigating tomorrow if it is possible to do this propagation without risks of breaking.

[memsharded]

This is quite challenging, and needs to be considered (small) step by step, defining how build requirements need to propagate when made visible requires some deeper research and careful consider what could be breaking. I am assigning it for 2.5 (2.4 will be soon), to start working on it, but it might take longer time and require a few iterations and releases to move forward.

[maximilianmuehlbauer]

This is quite challenging, and needs to be considered (small) step by step, defining how build requirements need to propagate when made visible requires some deeper research and careful consider what could be breaking. I am assigning it for 2.5 (2.4 will be soon), to start working on it, but it might take longer time and require a few iterations and releases to move forward.

Thanks for the feedback! We're happy to test with our use case as soon as that's possible. From having a quick look, could it be that this issues comes from the following lines where visible=False is being set:

conan/conans/model/requires.py

Lines 280 to 281 in 36148df

	downstream_require = Requirement(require.ref, headers=False, libs=False, build=True,
	run=True, visible=False, direct=False)