Skip to content

Latest commit

 

History

History
402 lines (336 loc) · 15.6 KB

README.adoc

File metadata and controls

402 lines (336 loc) · 15.6 KB
Raw

Modular Boost w/B2

This is a work-in-progress version of the Boost C++ Libraries that build as modular components. What does it mean to be modular? It means that..

  • Each library is separable from the rest.

  • Each library does not assume a particular directory arrangement in a parent Boost super-project.

  • The Boost super-project doesn’t need to exist at all, only requirement is that library dependencies be pre-declared.

  • No need for creating the headers tree.

  • Makes it possible for package managers, and hence users, to use the individual libraries they need.

Goals

Generally the main goal is to move the main Boost git repos to a modular structure over time without breaking any functionality. That is not an easy goal to achieve. As there are 153 library repos, 12 tool repos, and the super project repo to adjust along the way.

Happy to say though, that the bulk of the technical work is done. What’s left is to start merging pull requests for the transitional period. Then working on followup changes to fully become the modular structure.

Structure

To manage this effort safely there are a couple resources that I’ve created:

  • A GitHub Project to keep track of the work.

  • The "modular" super-project git repo.

  • The "transitional" super-project forked git repo.

  • Dozens of library and tool forked git repos with the modular PR changes.

Project

There is a project to track the past, present, and future state of this effort at https://github.com/users/grafikrobot/projects/1 with the following views:

  • Overall Tech Tasks: This is mainly infrastructure type tasks that need to happen.

  • Pull Requests: The PR’s and their status against the mainline Boost repos.

  • Merge Process: How to process the PRs for each Boost git repo. It’s the result of asking all Boost library maintainers if they would like to approve and merge PRs, or have the Boost project owners do whatever is needed, or something else.

Modular Super-Project

The modular super-project at https://github.com/grafikrobot/boost-b2-modular follows a similar structure as the main Boost super-project repo. It’s composed of many git submodules for all the libraries, tools, and support code with some differences:

  • All submodules track the "modular" branch on each which holds the changes.

  • There is only one tool submodule repo. The tools/boost_build project that holds the build support installing and staging things like the monolithic headers tree. Which is a dependency for the libs/headers submodule.

  • There is no B2 project root. The only B2 file is a convenience (for testing) project-config.jam file. It declares the B2 /boost project search locations.

Hence this git repo models the fully modular structure for libraries. As might be seen from package manager, or from selective use of libraries in some other project.

This git repo tests the ability to build each of the 153 libraries on Ubuntu and Windows as a GitHub action (https://github.com/grafikrobot/boost-b2-modular/actions/workflows/check-libs.yml)

Transitional Super-Project

The transitional super-project at https://github.com/grafikrobot/boostorg.boost is a fork of the Boost super-project develop branch (branched into the modular branch). It contains the changes needed to create a transition from the monolithic to the modular structure. And is the source of the super-project PRs (indirectly and selectively). It has all the same Boost super-project repo plus a few more utility tools to aid in the modular development.

It also has a series of CI checks as GitHub Actions:

  • Check Libraries: Identical to the modular checks except in this repo structure. Except it only checks that B2 parses and declares the targets without errors. It doesn’t try to compile or run tests.

  • CI: This is the same as the Boost super-project runs. Done here to verify it’s as green as the Boost super-project while making pre-PR changes.

  • Modular Release: Runs an equivalent process as the Boost super-project to create a full release snapshot. I.e. the same as what runs on Circle CI to create the release archives the users download. Except it does it in the transitional structure.

Technicalities

Globally Unique Project Name/Identifier

In order to avoid confusion, and to make portable cross-lib references possible, we define consistent top-level project names for libraries. For all libraries that name is the GitHub repo name for the library. Everywhere else we will refer to that as the lib-name.

For B2 projects we accomplish using the unique

Library Root Build

The biggest change for modular build support is that all libraries now have a lib-root/build.jam file. The file defines a project, some specific targets, and Boost specific install targets for all libraries. Consider this the public face of the library.

project /boost/[lib-name]

Defines the canonical externally usable project (and target).

alias boost_[lib-name]

Defines an explicit target that others can refer to to use the library in its default configuration. Some libraries will have additional, similar, targets for the cases where they have alternate, or additional, use cases. This target exists even for header-only libraries. As it’s a way to get usage requirements applied. Specifically for getting include paths added.

alias all

Defines a target that will build everything that makes sense to build. This minimally includes the library to build, i.e. the boost_[lib-name] target. But preferably also include examples and tests. Having this target makes it easier to check as much of the project as possible in an automated way.

boost-library [lib-name]

Conditionally calls Boost specific declaration functions. The conditional aspect is that the boost-library rule only exists in the context of the super-project. Other times it will be ignored. Which allows to make use of Boost specific definitions while still allowing sans-super-project use cases.

boost-library [lib-name] : install boost_[lib-name] ..

As part of the boost-library declaration, the install defines the targets that will get built and installed as part of the super-project.

This is an example of what a complete build.jam looks like. In this case for the Boost.Atomic library:

require-b2 5.1 ; (1)

project /boost/atomic (2)
    : common-requirements (3)
        <include>include (4)
        <library>/boost/align//boost_align (5)
        <library>/boost/assert//boost_assert
        <library>/boost/config//boost_config
        <library>/boost/predef//boost_predef
        <library>/boost/preprocessor//boost_preprocessor
        <library>/boost/type_traits//boost_type_traits
        <library>/boost/winapi//boost_winapi
    ;

explicit
    [ alias boost_atomic : build//boost_atomic ] (6)
    [ alias all : boost_atomic test ] (7)
    ;

call-if : boost-library atomic (8)
    : install boost_atomic (9)
    ;

  1. Specifies the minimum B2 version needed for this project.

  2. The canonical project alias. Scoped to the /boost project.

  3. Requires that are also usage-requirements.

  4. All Boost libraries need to specify their include directory.

  5. Any intra-boost library dependencies need to be listed. These are only the "public" dependencies. These are what users will see as dependencies.

  6. The library, or libraries, to build. This would be an empty alias for header only libraries.

  7. The all target. In this example builds the library and tests.

  8. Defines the Boost library, and calls the boost-* specific rules given.

  9. Declares the targets to install.

Important
 The boost-library declaration should be the only Boost specific part of the build file. Everything else needs to be generic B2 declarations. Following this is what makes it possible to build the libraries without the super-project. And hence be able to be consumed in a modular way.

Rules & Changes

In order to make modular building work, that is building without the super-project, we need to follow some basic rules. Which also means making changes in existing libraries to follow those rules.

Important
 No super-project relative references anywhere.
Important
 No Boost specific build declarations except for the boost-library.
Important
 No inter-library dependencies to internal targets.

Following those rules here are some examples of the modular changes needed for current libraries..

References of any kind to the headers in boost-root/boost need to be changed to references to the library-root/include/boost equivalent.

Before
doxygen tagfile
    :
        ../../../boost/accumulators/framework/depends_on.hpp # (1)
        ../../../boost/accumulators/framework/extractor.hpp
After
doxygen tagfile
    :
        ../include/boost/accumulators/framework/depends_on.hpp # (1)
        ../include/boost/accumulators/framework/extractor.hpp

Addition of the boost-root as an include location needs to be removed. As it may not exist. And includes for library dependencies will refer to the include directory per library.

Before
exe fibonacci
    : fibonacci.cpp
    : <include>$(BOOST_ROOT) # (1)
After
exe fibonacci
    : fibonacci.cpp
    # (1)
    ;
Before
exe interval_container
    :
        interval_container_/interval_container.cpp
    :
        <include>../../.. # (1)
        <include>$(BOOST_ROOT)
    ;
After
exe interval_container
    :
        interval_container_/interval_container.cpp
    :
    # (1)
    ;

All library dependency references need to be of the /boost/[lib-name]//[target] canonical form. As those are the references that will work in both the super-project layout and portable modular layout.

Before
<library>/boost//serialization/<warnings>off ; # (1)
After
<library>/boost/serialization//boost_serialization/<warnings>off ; # (1)
Before
project random_multi_points
    : requirements
        <include>.
        <library>../../../../program_options/build//boost_program_options # (1)
        <link>static
    ;
After
project random_multi_points
    : requirements
        <include>.
        <library>/boost/program_options//boost_program_options # (1)
        <link>static
    ;

Dependencies of targets (for build, examples, tests, docs, etc) need to be included even if they are to header-only libraries. As the monolithic boost-root headers directory may not exist.

Before
[ run thread_safety_checking.cpp
    : : : <debug-symbols>on <library>.//test_impl_lib_backtrace $(LINKSHARED_BT)
        # (1)
    : backtrace_lib_threaded ]
After
[ run thread_safety_checking.cpp
    : : : <debug-symbols>on <library>.//test_impl_lib_backtrace $(LINKSHARED_BT)
        <library>/boost/optional//boost_optional # (1)
    : backtrace_lib_threaded ]

Sometimes it’s more convenient to add the inter-library dependencies on the particular project instead of main targets. For example, if your tests all depend on additional libraries than the user public ones.

Before
project boost-geometry-test
    :
    requirements
        <include>.
        <toolset>msvc:<asynch-exceptions>on
        <toolset>msvc:<cxxflags>/bigobj
        <toolset>clang:<cxxflags>-Wno-unneeded-internal-declaration
        <toolset>intel:<define>BOOST_GEOMETRY_TEST_ONLY_ONE_TYPE
        <host-os>windows,<toolset>intel:<cxxflags>/bigobj
        # (1)
    ;
After
project boost-geometry-test
    :
    requirements
        <include>.
        <toolset>msvc:<asynch-exceptions>on
        <toolset>msvc:<cxxflags>/bigobj
        <toolset>clang:<cxxflags>-Wno-unneeded-internal-declaration
        <toolset>intel:<define>BOOST_GEOMETRY_TEST_ONLY_ONE_TYPE
        <host-os>windows,<toolset>intel:<cxxflags>/bigobj
        <source>/boost/test//boost_test # (1)
        <source>/boost/foreach//boost_foreach
        <source>/boost/assign//boost_assign
    ;

Boost also makes use of library provided B2 extensions (modules in B2 parlance). For example Boost.Config provides a config.jam module. And Boost.Predef provides a predef.jam module. Previously those would be imported through a relative path. But that will no longer work int he modular layout. As there’s no guarantee those relative paths will exist. For this B2 has a new feature to provide search paths for those modules: rule import-search ( reference ). The key aspect in the example below is that the /boost/config/checks reference is expanded out to find where the /boost/config project is located and the project subdirectory checks in that is added to the paths B2 will search for modules with the import. [1]

Before
import ../../config/checks/config : requires ; # (1)
After
import-search /boost/config/checks ; # (1)
import config : requires ;

Build files are not the only place where references to the Boost root happen. Because the Boost root is in the include search path there are C++ source files that #include other sources relative to the root. In such cases it may also be needed to pair the C++ change with a B2 change to add new #include directories.

Before
#include "libs/math/test/log1p_expm1_test.hpp" // (1)
After
#include "log1p_expm1_test.hpp" // (1)
Before
#define BOOST_USER_CONFIG <libs/stacktrace/example/user_config.hpp> // (1)
After
#define BOOST_USER_CONFIG <example/user_config.hpp> // (1)
1. https://www.bfgroup.xyz/b2/manual/release/index.html#bbv2.reference.rules.import-search