[FEATURE REQUEST] Build libtorrent using Bazel

[qzmfranklin]

Please provide the following information

libtorrent version (or branch):
master, or pretty much any release

platform/architecture:
Ubuntu 16.04, x86_64

compiler and compiler version:
GCC 5.4.0
clang 5.0.0

Feature request:

As mentioned in the title of this ticket, this is a feature request. Well, not really requesting yet, but just very broadly raising the question: Is there willingness to build libtorrent using Google's Bazel?

There are two main dependencies of libtorrent: boost and openssl.

I have Bazelised the build of boost and openssl under Ubuntu 16.04 with no or minimal change to the official repositories of boost (source files unchanged), openssl (renamed a directory), and libtorrent (source files unchanged), in my own repository https://github.com/qzmfranklin/torrent (master branch always builds correctly). In my example repo, you can build the libtorrent's client using

bazel build third_party/libtorrent:client

The built binary works. I can use it to download torrents and use the nice terminal-based UI.

Or you can run all the tests with

bazel test third_party/libtorrent/...

Currently 22 tests fails and tests under similations are not converted yet.

These are just some preliminary work towards building libtorrent in Bazel. Would like to know how people think here.

Thanks a lot!

[aldenml]

well, I can tell you that even cmake support gets easy out of sync, IMO not sure if bezel has any real opportunity here, unless it can be generated automatically from any of the current builds

[arvidn]

My impression of bazel is that its main selling point is building software in large organizations, perhaps with a single (logical) source code repository for all dependencies.

I must admit I don't know that much about bazel otherwise, but it seems pretty heavy weight for any casual building.

What are the main selling points of using bazel, over boost-build say?

I understand it may be a bit quicker with large build trees by caching files that have changed with an inotify service, is that right?

[qzmfranklin]

Hi @arvidn and @aldenml ,

Thanks a lot for the replies.

Context-wise, I have been using Bazel at my previous company (founded and headed by ex-Googlers) for a year and a half. Afterwards, I have been using Bazel for all my personal projects. I also Bazelised (or improved the existing Bazel files) the build of many open source C/C++/assembly projects such as openssl, boost, icu4c, benchmark, benchmark, libtorrent (yep, I have done it), c-ares, cctz, jemalloc, libunwind, glog, abseil-cpp. For a few of them, I have already created PRs or issued tickets on their official github repos.

Prior to Bazel, I used a lot of Makefiles, cmake, maven, gradle, xcodebuild, etc.. At one point I even wrote my own build system and designed a context free grammar to write build systems.

All in all, in this prolonged effort of seeking the perfect build system, Bazel really amazed me. The Bazel official website explains why Bazel is so great. Just list a few here:

• Perfect incremental build and test. If you make a one line change, bazel test only builds and runs tests that are affected by your change.
• Very to easy to convert to Bazel. For example, I already have a fully Bazelised (including all tests) fork of libtorrent at https://github.com/qzmfranklin/libtorrent. There is only a single BUILD file for the entire project.
• Expressive query language. You can query almost arbitrary information about the dependency graph using the bazel query language. For example, it allows you to get answers to questions such as 'what targets are affected by my change in foo.cc?'
• Python-like language for writing the BUILD files. I know Python is not the favorite language of everybody. But the programming languages in which you can write build files with other build systems such as cmake (crap), Makefile (crap), boost-build (yaml), gradle (groovy), maven (xml), python is really the best - easy to learn, easy to read, easy to maintain - that is, amongst all the ones named here.
• Great extensibility - one build system for all languages and needs. You can extend the set of build rules that Bazel can handle by writing custom rules in its Python-like language. Bazel started with support for only Java and C++/C. But over time, people have extended support for python, golang, rust, swfit, docker, PHP, Ruby, C#, Erlang, protobuf, grpc, Kubernetes, PythonPex, scala, SWIG, ... As you notice, this list is not limited to programming languages, you can write highly customized and performant rules in Bazel to meet your needs. So that you can combine all your work together under a unified build system.
• An emerging community. Bazel was open sourced about 3 years ago. It was known as the Blaze internally at Google. Over time, it has drawn greater and greater awareness and got lot of attraction. If you Google 'Bazel', there are a lot of active projects and threads and the number is only increasing by the day.

@arvidn you are definitely right that Bazel does perfect incremental builds (with caches). But Bazel not only benefits large, monolithic repos such as the ones inside Google and Facebook. Bazel makes the building and maintenance of small projects super simple too. It is hard to tell the true power using some text interface like this. But I would very much like to provide more evidences if you are interested in any particular aspects.

I have not answered your questions directly. But hopefully what I wrote serves as an answer, of sort :D

Best,

Zhongming

[qzmfranklin]

As an example, here is a PR that adds a Bazel BUILD file that has been recently merged to mainline glog:
google/glog#232

[arvidn]

looking at your build file, the first thing that comes to mind is that it doesn't look like bazel provide abstractions for platforms and compilers. There are a lot of GCC specific command line switches in there. So much that I must assume this doesn't work for msvc (or any other compiler that isn't GCC compatible).

Apart from that, how does bazel know which of those compiler switches must be propagated down to dependencies when they are built, or up to dependents when they are built?

For example, if you build libtorrent with -DBOOST_ASIO_ENABLE_CANCELIO=1, you must also build the client with that switch. Same thing with -std=c++11 (since it affects the ABI).

One reason why this build file is simple, is because it doesn't support most of the build configurations of libtorrent, which may not be very important (to you at least).

That said, I must admit that, for being python, the syntax is unusually compact. Compared to other build systems using general purpose languages. However, it's still a lot noisier than boost-build.

[qzmfranklin]

Hi @arvidn ,

Just want to reiterate, in case I did not make this crystal clear to begin with: I do not intend to use Bazel to replace boost_build or any existing build system that libtorrent has been using. That would be folly. All I want is to add some extra support for building with Bazel to libtorrent. Honestly, it is such a great library. I want to use it. But I also want to use it with Bazel for the benefits I described in the previous comment. Hopefully this helps set up a less closed mental state for this discussion.

OK, now let's begin the discussion. You raised two very fair points:

1. How does Bazel support platform and toolchain specific compiling and linking options?
2. How does Bazel selectively propagate cc_library compiling options to clients depending on it?

Indeed, these questions have already been very well answered by Bazel. My BUILD file used many shortcuts and hard coding because, well, I am just boostrapping this effort. If you would like to more seriously consider the idea of adding Bazel support for libtorrent, the BUILD file must be augmented.

OK, actually answering your questions:
A1: Bazel support platform and compiler specific settings via the config_setting construct. Without going into the weeds, the idea that is relevant to this question is that the you can map the (OS, arch, compiler) tuple into a config_setting object in Bazel BUILD file. Then you can use select to specify compiler flags, linker flags, macros, optional source files, etc. based on the config_setting.

A2: Per the official doc about cc_library and cc_binary, if you pass macros in the copts keyword argument, that macro will not be propagated to its depending targets. However, if you use the defines keyword, then all macros defined in the defines keyword will be passed to the depending targets. As a side note, there is also a similar keyword argument called includes that causes the contained paths to be passed to all depending targets.
This said, if you really have some highly specialized flags that you have to pass around that are not captured by the defines and the includes keyword arguments, you do have to pass them manually to the depending targets. Even though I advocate for Bazel, I cannot ignore its actual limitations.

You kind of touched upon the point of supporting multiple build configurations for libtorrent. Without much knowledge, I, doing a poor man's guess, believe that is more or less a solvable problem with my existing understanding of libtorrent and Bazel. It might not be 100% solvable. But at least I believe that a system that supports, say, 80% use cases still has valuable - feel free to scold me here for being nitpicking :D

How do you think?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.