[package] boost/1.74.0: build error for Android

[andioz]

With the newest recipe, Android build fails.

Configuration:
[settings]
arch=armv8
arch_build=x86_64
build_type=Release
compiler=clang
compiler.libcxx=libc++
compiler.version=8
os=Android
os.api_level=21
os_build=Linux
[options]
[build_requires]
*: android_ndk_installer/r20@bincrafters/stable
[env]

Sorry for the short information here, but I'm currently not able to test it in detail, here is the relevant output from our CI runner:

ERROR: boost/1.74.0: Error in package_info() method, line 1164
	assert len(non_used) == 0, "These libraries were not used in conan components: {}".format(non_used)
	AssertionError: These libraries were not used in conan components: {'boost_log_setup', 'boost_log'}

I guess it has todo with the latest changes, and with the Android specific log parts?

[andioz]

I'm currently trying to find a workaround in my project, want to pin the previous version in the requirements, but this is not working...

    requires = (#
        ...
        "boost/1.74.0@_/_#587f8b8e956a18e45486af06c1c973e3",
        ...
    )

Would need to clear the cache before running it, and need to set CONAN_REVISIONS_ENABLED=1 temporarily? Does somebody know how to enable this just for this one requirement in my project's conanfile.py?

[andioz]

OK, I'm trying this, looks good (why iconv is a problem too, I don't know):

        self.options["boost"].without_log = True  # TODO android: "These libraries were not used in conan components"
        self.options["boost"].without_locale = True  # TODO android: require 'libiconv' not used in components requires

[datalogics-kam]

We're having problems at DL as well, in a nightly job that worked until the last two nights. Here's what we're seeing.

conans.errors.ConanExceptionInUserConanfileMethod: 

	ConanException: boost/1.74.0 package_info(): Package require 'libiconv' declared in components requires but not defined as a recipe requirement

Looking at the PR, at surface level, this is interesting in dependencies/dependencies-1.74.0.yml.

  locale:
  - iconv
  - icu

Locale requires either iconv or icu (or perhaps neither) depending on the boost:i18n_backend setting, which in our case is:

        'boost:i18n_backend': 'icu'

Defining iconv as a requirement is not an option for us. We don't use it, and LGPL/GPL licenses are prohibited in our code.

[andioz]

Additional info: a teammate reports the same issue for iOS!

[prince-chrismc]

As a fellow CCI (enterprise) consumer I would strongly recommend working with recipe revisions along with lockfiles if you need reproducible builds. including the recipe revision like self.requires("boost/1.74.0#587f8b8e956a18e45486af06c1c973e3") along with enabling it in your config is a must for any CI system.

@madebr I know you are working on more fixes/changes in the 1.75.0 PR is this something related some of your changes?

[andioz]

Including the recipe revision like self.requires("boost/1.74.0#587f8b8e956a18e45486af06c1c973e3") along with enabling it in your config is a must for any CI system.

Yeah, I recently watched the videos in JFrog Academy, and I learned about that. Not sure yet how this will fit into our current project setup. If I remember right, there can only be one revision (of the same version) in Conan cache, right? What if 2 projects uses different revisions? How to handle this? And using revisions requires a client configuration change, otherwise the revision is ignored (I tried it and it looks like that to me).

Overall, this is one of the newer features I have to learn about.

[madebr]

@prince-chrismc

Probably, this is the next issue on the list to look at.
Looks like iOS is working so that's that.

[datalogics-kam]

including the recipe revision like self.requires("boost/1.74.0#587f8b8e956a18e45486af06c1c973e3") along with enabling it in your config is a must for any CI system.

This is a good idea, however, we've run into strange problems in requirements resolution, where the caching in the requirements resolver stumbles if you require a recipe revision and a transitive requirement doesn't.

Our solution was to create a conan-pinned-recipes repo on our Artifactory and put that earlier in the remote list than the Conan Center. We upload the actual recipe we want to pin there. Having it be a separate repo makes it easy to list the recipes we pinned.

❯ conan search -r conan-pinned-recipes
Existing package recipes:

boost/1.72.0
bzip2/1.0.8
libpng/1.6.37
zlib/1.2.11

[prince-chrismc]

If I remember right, there can only be one revision (of the same version) in Conan cache, right?

Sadly, yes.

The trick is when your requirements change you flush the cache since you know it's stale.

What if 2 projects uses different revisions? How to handle this?

Option One

always run conan install --update which forces the cache to be updated with the exact recipe revision.

UPDATE: you need list all of you transitive dependencies in the conanfile in order for this to work

I've deployed this at scale for ~6 months and I've avoid this exact headache which lead to me to find a few bugs 🐛 in components generator.

Option Two

Lockfiles: the workflow was explained to me here, conan-io/docs#1904

⚠️ I have not deployed this at scale but I have a working POC

I think this will the only reasonable way moving forward if you want to have a short minimal requires and still have reproducible builds. This is exactly how Yarn, Go Modules, and Pypi work so many engineers should grasp it quickly.

https://github.com/prince-chrismc/user-management/blob/d6fb746a2ab03e6a12610687bbf04cc9e14942ee/backend/conanfile.py#L23-L27

https://github.com/prince-chrismc/user-management/blob/d6fb746a2ab03e6a12610687bbf04cc9e14942ee/.github/workflows/upload.yml#L49-L50

[andioz]

Thanks @prince-chrismc for giving really useful suggestions to the right way. I think I have to postpone my rethinking about using lock files, revision, etc. But it is a must, definitely, to freeze not only the packages, but even the recipe versions. Was no issue until now. My projects are CMake based, and I run conan commands using execute_process() from inside. Maybe I have to do an extra step to create the lockfiles too.

One question: the lockfiles should be stored with the project in git, right? So we have generated files in git.

[prince-chrismc]

the lockfiles should be stored with the project in git, right?

Yes! My "base lockfile" can be found here https://github.com/prince-chrismc/user-management/blob/d6fb746a2ab03e6a12610687bbf04cc9e14942ee/backend/conan.lock

For working locally I let cmake create the lockfiles https://github.com/prince-chrismc/user-management/blob/d6fb746a2ab03e6a12610687bbf04cc9e14942ee/backend/cmake/conan-setup.cmake#L16 however in CI to create a conan package it needs to be done before the CMake configurration

https://github.com/conan-io/cmake-conan offered a few good helpers

[madebr]

@andioz
I think it works now with #3872

I see that boost is unable to detect libiconv for the locale module.
Instead of error'ing, b2 just doesn't build the locale module.
This causes the libiconv requirement not being used by a component and an error emitted.

So on android, the locale module should not be used.
I can build it using:

conan create . boost/1.74.0@ -pr android21 -o boost:without_locale=True -o boost:without_log=True

Can you please verify?

[andioz]

So on android, the locale module should not be used.

Great, I'm looking forward for the updated recipe! Regarding locale: maybe we should disable it internally for platforms where it's not available? I see such a behavior in OpenCV for example. I think it would be cool if a recipe could be used on any supported platform without "required options". Which sounds weird 😵

Edit: I will try tomorrow

[ohanar]

@madebr Seems like #3895 is related to the issues with the locale module.

[prince-chrismc]

@ohanar please test the PR this way we and merge quickly!

[andioz]

OK, here we go, looks good!

First I tried conan create from your branch, but I ommited the options (I guess this is what you meant, because this 2 without's are my current workaround).

conan create . boost/1.74.0@ -pr <path/to/my/android/profile>

Works, nice. Then I used my regular project on this recipe, even with disabling my workarounds, and it works too!

So, from my point of view, it looks good. Just for understanding: the log and locale parts are now build and would work? Here is the output from the boost build:

Performing configuration checks

    - default address-model    : 64-bit
    - default architecture     : arm
    - C++11 mutex              : yes
    - lockfree boost::atomic_flag : yes
    - has stat::st_mtim        : yes
    - has stat::st_mtimensec   : yes
    - has stat::st_mtimespec   : no
    - Boost.Config Feature Check: cxx11_auto_declarations : yes
    - Boost.Config Feature Check: cxx11_constexpr : yes
    - Boost.Config Feature Check: cxx11_defaulted_functions : yes
    - Boost.Config Feature Check: cxx11_final : yes
    - Boost.Config Feature Check: cxx11_hdr_mutex : yes
    - Boost.Config Feature Check: cxx11_hdr_tuple : yes
    - Boost.Config Feature Check: cxx11_lambdas : yes
    - Boost.Config Feature Check: cxx11_noexcept : yes
    - Boost.Config Feature Check: cxx11_nullptr : yes
    - Boost.Config Feature Check: cxx11_rvalue_references : yes
    - Boost.Config Feature Check: cxx11_template_aliases : yes
    - Boost.Config Feature Check: cxx11_thread_local : yes
    - Boost.Config Feature Check: cxx11_variadic_templates : yes
    - has_icu builds           : no
    - zlib                     : yes
    - bzip2                    : yes
    - iconv (libc)             : no
    - iconv (separate)         : yes
    - native-atomic-int32-supported : yes
    - native-syslog-supported  : yes
    - pthread-supports-robust-mutexes : no
    - gcc visibility           : yes
    - long double support      : yes
    - libbacktrace builds      : no
    - addr2line builds         : no
    - WinDbg builds            : no
    - WinDbgCached builds      : no
    - BOOST_COMP_GNUC >= 4.3.0 : no

Component configuration:

    - atomic                   : building
    - chrono                   : building
    - container                : building
    - context                  : building
    - contract                 : building
    - coroutine                : building
    - date_time                : building
    - exception                : building
    - fiber                    : building
    - filesystem               : building
    - graph                    : building
    - graph_parallel           : not building
    - headers                  : not building
    - iostreams                : building
    - locale                   : building
    - log                      : building
    - math                     : building
    - mpi                      : not building
    - nowide                   : not building
    - program_options          : building
    - python                   : not building
    - random                   : building
    - regex                    : building
    - serialization            : building
    - stacktrace               : building
    - system                   : building
    - test                     : building
    - thread                   : building
    - timer                    : building
    - type_erasure             : building
    - wave                     : building

Great work! ⭐⭐⭐⭐⭐

[prince-chrismc]

Great work! ⭐⭐⭐⭐⭐

I'd like to nominate madebr as community member of the month!

[madebr]

Thanks for the nomination,
but please also do a final test when the recipe is finalized.

I just fixed (:crossed_fingers: ) fibers and nowide, which might cause problems for your use cases.

[andioz]

Today's check still working 👍