Parallel docbuild, cythonization: Use GNU make's POSIX jobserver protocol

[mkoeppe]

The correct way out of the mess with recommending MAKE='make -j8' instead of make -j8 (which we do because of the outdated GNU make 3.81 on macOS - see #21610, #30345) is to use GNU make's POSIX job server protocol - https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html

• Create a wrapper script in Python that tries to acquire >= A, <= B jobs from the jobserver within <= T wall clock seconds, sets the environment variable SAGE_NUM_THREADS to min(C, NUM_JOBS_RECEIVED), and invokes an arbitrary script.

• Use this script as a wrapper for the sagelib build and docbuild.

• Replace the recommendation MAKE='make -j8' make by the familiar make -j8.

• In the top-level Makefile, filter out -j so that also users who continue to follow the old recommendation MAKE='make -j8' make benefit from the improvements, and to get rid of the "disabling jobserver" messages.

CC: @dimpase @orlitzky @jhpalmieri @Etn40ff

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/30369

[mkoeppe]

comment:1

For example https://github.com/catkin/catkin_tools/blob/master/catkin_tools/execution/job_server.py

[orlitzky]

comment:2

Copying my comment from #30345:

We're having trouble here because we're conflating two different things. The sage distribution has a top-level makefile that's used to initiate high-level tasks -- in particular installing SPKGs -- convenient to execute. Many of those high-level tasks themselves have build systems, separate from sage, that involve Make. We're passing the distribution's make flags down to the individual build systems and trying to coordinate the two as if it was one large build system, but it really isn't.

When in doubt, copy Gentoo: our package manager has --jobs and --load-average arguments that control how many package installation jobs get executed at once. We then also have a MAKEFLAGS variable that is passed to the individual build systems. (A better name would be something like BUILDJOBS, but the name is historical.) Sage's distribution Makefile can likely always be invoked with -j1 with no performance penalty; it's the individual build systems that need something like -j8.

So, my suggestion: we copy Gentoo, and create something like a SAGE_BUILD_JOBS variable that defaults to 4 (everyone has at least dual-core by now?). We can then document that the top-level make need not be invoked with -j<anything>, and within that top-level Makefile, we would pass -j${SAGE_BUILD_JOBS} to any Make-based SPKG build systems. If we encounter a non-Make build system, we can then still do what the user asked, and pass whatever flags are necessary to run the build with ${SAGE_BUILD_JOBS} threads.

I think that's less complicated that trying to coordinate the two levels, and doesn't wed us further to GNU's implementation of Make.

[dimpase]

comment:3

IIRC, Sage cannot be built with a nonGnu make, anyway.

IMHO Gentoo's way looks a bit like a workaround for old buggy Gnu Make.

[mkoeppe]

comment:4

Replying to @orlitzky:

Copying my comment from #30345:

We're having trouble here because we're conflating two different things. The sage distribution has a top-level makefile that's used to initiate high-level tasks -- in particular installing SPKGs -- convenient to execute. Many of those high-level tasks themselves have build systems, separate from sage, that involve Make. We're passing the distribution's make flags down to the individual build systems and trying to coordinate the two as if it was one large build system, but it really isn't.

No, the point of the jobserver is that one does NOT have to distinguish these "two different things". The jobserver manages the load.

We do not have a problem with the jobserver. We are careful to pass down the make options to sub-makes. This works.

We have a problem with the python-based build systems that do not use the jobserver, and this ticket proposes a specific way to solve it.

[orlitzky]

comment:5

Replying to @dimpase:

IIRC, Sage cannot be built with a nonGnu make, anyway.

The Makefiles generated by autotools are POSIX compatible, it's only the hand-written Makefiles that are a problem. We generate those with a bootstrap script anyway, so at least in theory it would be possible to coerce them into being POSIX compatible.

IMHO Gentoo's way looks a bit like a workaround for old buggy Gnu Make.

It's just a way to let users specify how many jobs should be used to build each package in a source-based distribution. The number of packages that will be installed simultaneously is something else. Our situation isn't much different, except that the "package manager" is also Make.

[mkoeppe]

comment:6

Replying to @orlitzky:

It's just a way to let users specify how many jobs should be used to build each package in a source-based distribution. The number of packages that will be installed simultaneously is something else. Our situation isn't much different, except that the "package manager" is also Make.

Users should have no reason to be interested in this distinction. All that matters is throughput.

[orlitzky]

comment:7

Replying to @mkoeppe:

No, the point of the jobserver is that one does NOT have to distinguish these "two different things". The jobserver manages the load.

You can certainly integrate the two cleanly with the jobserver approach, but that doesn't change my opinion that they're conceptually two different quantities. The first is how many high-level tasks I'd like to run at the same time (doc building, package installation), and the second is how many threads I'd like to use to perform each of those tasks.

Users should have no reason to be interested in this distinction. All that matters is throughput.

Maybe not, but we could default both to the same number to accomplish that out-of-the-box. Experience has shown that most people are happy with one package installation at a time, and then a configurable number of jobs. High-level tasks are generally free, while low level ones tend to chew up CPU/RAM. What I personally care about is the number of low-level build threads, which should be the number of package-installation threads times the number of per-package-build-threads. Again using Gentoo as an example, I usually leave this at one package at a time, and then ~four build threads per package.

[mkoeppe]

comment:8

Replying to @orlitzky:

we could default both to the same number to accomplish that out-of-the-box. Experience has shown that most people are happy with one package installation at a time, and then a configurable number of jobs.

Sorry, this does not match the typical experience in Sage at all. Throughput comes from building many packages in parallel.

[orlitzky]

comment:9

One other thing that distinguishes Gentoo and sage-the-distribution is that Sage is fortunate enough to only have C, C++, Fortran, and python packages for now. Some day that will probably change, and then it's a lot simpler to pass a flag to whatever newfangled build system is involved than it would be to integrate that build system with the GNU Make job server.

[mkoeppe]

comment:10

Replying to @orlitzky:

.... whatever newfangled build system ...

such as ninja? Note https://github.com/Kitware/ninja/releases, https://forums.gentoo.org/viewtopic-t-1110292-view-previous.html

[orlitzky]

comment:11

Replying to @mkoeppe:

Replying to @orlitzky:

.... whatever newfangled build system ...

such as ninja? Note https://github.com/Kitware/ninja/releases, https://forums.gentoo.org/viewtopic-t-1110292-view-previous.html

I actually had Meson in mind, but this is also a good example. Utilizing the job server has the potential to be more efficient, if e.g. you're only able to build one package at a time due to a bunch of linear dependencies (in that case, you're getting 1*N instead of M*N jobs where M is the number of jobs you want your package manager to execute in parallel). But as with ninja, the trade-off is that someone would have to write and maintain a huge patch to the build system to support the jobserver protocol, and then it only works with GNU make. Without the patch, we pass e.g. -j8 to ninja (just like with Make) and it works fine. Maybe slightly less efficient, but a lot less complicated.

[orlitzky]

comment:12

And actually, I knew that Meson didn't use Make as its backend, but... it uses Ninja! So they're the same example.

[mkoeppe]

comment:13

Anyway, thanks for the discussion. It is good that we are aware of the longer term context. We'll see in which direction things move (I predict that more build systems embrace the job server protocol, or improvements of it).

To return to the technical point of this ticket: I propose to create a wrapper script in Python that tries to acquire >= A, <= B jobs from the jobserver within <= T wall clock seconds, sets the environment variable SAGE_NUM_THREADS to min(C, NUM_JOBS_RECEIVED), and invokes an arbitrary script. Using this for the sagelib build and docbuild will allow us to replace the recommendation MAKE='make -j8' make by the familiar make -j8, and will likely be an improvement over the status quo in terms of load management.

[mkoeppe]

comment:17

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.