Skip to content

Commit

Permalink
rustbuild: Document many more parts of the build
Browse files Browse the repository at this point in the history 
This commit expands the bootstrap build system's `README.md` as well as ensuring
that all API documentation is present and up-to-date. Additionally a new
`config.toml.example` file is checked in with commented out versions of all
possible configuration values.
  • Loading branch information
@alexcrichton
alexcrichton committed May 3, 2016
1 parent d80497e commit f72bfe6
Show file tree
Hide file tree
Showing 19 changed files with 726 additions and 47 deletions.
142 changes: 136 additions & 6 deletions src/bootstrap/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Bootstrapping Rust
# rustbuild - Bootstrapping Rust

This is an in-progress README which is targeted at helping to explain how Rust
is bootstrapped and in general some of the technical details of the build
Expand All @@ -8,20 +8,64 @@ system.
> intended to be the primarily used one just yet. The makefiles are currently
> the ones that are still "guaranteed to work" as much as possible at least.
## Using the new build system
## Using rustbuild

When configuring Rust via `./configure`, pass the following to enable building
via this build system:

```
./configure --enable-rustbuild
make
```

## ...
Afterwards the `Makefile` which is generated will have a few commands like
`make check`, `make tidy`, etc. For finer-grained control, the
`bootstrap.py` entry point can be used:

```
python src/bootstrap/bootstrap.py
```

This accepts a number of options like `--stage` and `--step` which can configure
what's actually being done.

## Configuring rustbuild

There are currently two primary methods for configuring the rustbuild build
system. First, the `./configure` options serialized in `config.mk` will be
parsed and read. That is, if any `./configure` options are passed, they'll be
handled naturally.

Next, rustbuild offers a TOML-based configuration system with a `config.toml`
file in the same location as `config.mk`. An example of this configuration can
be found at `src/bootstrap/config.toml.example`, and the configuration file
can also be passed as `--config path/to/config.toml` if the build system is
being invoked manually (via the python script).

## Build stages

The rustbuild build system goes through a few phases to actually build the
compiler. What actually happens when you invoke rustbuild is:

1. The entry point script, `src/bootstrap/bootstrap.py` is run. This script is
responsible for downloading the stage0 compiler/Cargo binaries, and it then
compiles the build system itself (this folder). Finally, it then invokes the
actual `boostrap` binary build system.
2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
sanity checks (compilers exist for example), and then start building the
stage0 artifacts.
3. The stage0 `cargo` downloaded earlier is used to build the standard library
and the compiler, and then these binaries are then copied to the `stage1`
directory. That compiler is then used to generate the stage1 artifacts which
are then copied to the stage2 directory, and then finally the stage2
artifacts are generated using that compiler.

The goal of each stage is to (a) leverage Cargo as much as possible and failing
that (b) leverage Rust as much as possible!

## Directory Layout

This build system houses all output under the `target` directory, which looks
This build system houses all output under the `build` directory, which looks
like this:

```
Expand All @@ -42,6 +86,12 @@ build/
debug/
release/
# Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
dist/
# Temporary directory used for various input/output as part of various stages
tmp/
# Each remaining directory is scoped by the "host" triple of compilation at
# hand.
x86_64-unknown-linux-gnu/
Expand All @@ -50,7 +100,8 @@ build/
# folder is under. The exact layout here will likely depend on the platform,
# and this is also built with CMake so the build system is also likely
# different.
compiler-rt/build/
compiler-rt/
build/
# Output folder for LLVM if it is compiled for this target
llvm/
Expand All @@ -67,6 +118,17 @@ build/
share/
...
# Output folder for all documentation of this target. This is what's filled
# in whenever the `doc` step is run.
doc/
# Output for all compiletest-based test suites
test/
run-pass/
compile-fail/
debuginfo/
...
# Location where the stage0 Cargo and Rust compiler are unpacked. This
# directory is purely an extracted and overlaid tarball of these two (done
# by the bootstrapy python script). In theory the build system does not
Expand All @@ -82,7 +144,9 @@ build/
# invocation. The build system instruments calling Cargo in the right order
# with the right variables to ensure these are filled in correctly.
stageN-std/
stageN-test/
stageN-rustc/
stageN-tools/
# This is a special case of the above directories, **not** filled in via
# Cargo but rather the build system itself. The stage0 compiler already has
Expand All @@ -96,7 +160,7 @@ build/
# Basically this directory is just a temporary artifact use to configure the
# stage0 compiler to ensure that the libstd we just built is used to
# compile the stage1 compiler.
stage0-rustc/lib/
stage0-sysroot/lib/
# These output directories are intended to be standalone working
# implementations of the compiler (corresponding to each stage). The build
Expand All @@ -108,3 +172,69 @@ build/
stage2/
stage3/
```

## Cargo projects

The current build is unfortunately not quite as simple as `cargo build` in a
directory, but rather the compiler is split into three different Cargo projects:

* `src/rustc/std_shim` - a project which builds and compiles libstd
* `src/rustc/test_shim` - a project which builds and compiles libtest
* `src/rustc` - the actual compiler itself

Each "project" has a corresponding Cargo.lock file with all dependencies, and
this means that building the compiler involves running Cargo three times. The
structure here serves two goals:

1. Facilitating dependencies coming from crates.io. These dependencies don't
depend on `std`, so libstd is a separate project compiled ahead of time
before the actual compiler builds.
2. Splitting "host artifacts" from "target artifacts". That is, when building
code for an arbitrary target you don't need the entire compiler, but you'll
end up needing libraries like libtest that depend on std but also want to use
crates.io dependencies. Hence, libtest is split out as its own project that
is sequenced after `std` but before `rustc`. This project is built for all
targets.

There is some loss in build parallelism here because libtest can be compiled in
parallel with a number of rustc artifacts, but in theory the loss isn't too bad!

## Build tools

We've actually got quite a few tools that we use in the compiler's build system
and for testing. To organize these, each tool is a project in `src/tools` with a
corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
independent `Cargo.lock` files) and do not currently explicitly depend on the
compiler or standard library. Compiling each tool is sequenced after the
appropriate libstd/libtest/librustc compile above.

## Extending rustbuild

So you'd like to add a feature to the rustbuild build system or just fix a bug.
Great! One of the major motivational factors for moving away from `make` is that
Rust is in theory much easier to read, modify, and write. If you find anything
excessively confusing, please open an issue on this and we'll try to get it
documented or simplified pronto.

First up, you'll probably want to read over the documentation above as that'll
give you a high level overview of what rustbuild is doing. You also probably
want to play around a bit yourself by just getting it up and running before you
dive too much into the actual build system itself.

After that, each module in rustbuild should have enough documentation to keep
you up and running. Some general areas that you may be interested in modifying
are:

* Adding a new build tool? Take a look at `build/step.rs` for examples of other
tools, as well as `build/mod.rs`.
* Adding a new compiler crate? Look no further! Adding crates can be done by
adding a new directory with `Cargo.toml` followed by configuring all
`Cargo.toml` files accordingly.
* Adding a new dependency from crates.io? We're still working on that, so hold
off on that for now.
* Adding a new configuration option? Take a look at `build/config.rs` or perhaps
`build/flags.rs` and then modify the build elsewhere to read that option.
* Adding a sanity check? Take a look at `build/sanity.rs`.

If you have any questions feel free to reach out on `#rust-internals` on IRC or
open an issue in the bug tracker!
23 changes: 23 additions & 0 deletions src/bootstrap/build/cc.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,29 @@
// option. This file may not be copied, modified, or distributed
// except according to those terms.

//! C-compiler probing and detection.
//!
//! This module will fill out the `cc` and `cxx` maps of `Build` by looking for
//! C and C++ compilers for each target configured. A compiler is found through
//! a number of vectors (in order of precedence)
//!
//! 1. Configuration via `target.$target.cc` in `config.toml`.
//! 2. Configuration via `target.$target.android-ndk` in `config.toml`, if
//! applicable
//! 3. Special logic to probe on OpenBSD
//! 4. The `CC_$target` environment variable.
//! 5. The `CC` environment variable.
//! 6. "cc"
//!
//! Some of this logic is implemented here, but much of it is farmed out to the
//! `gcc` crate itself, so we end up having the same fallbacks as there.
//! Similar logic is then used to find a C++ compiler, just some s/cc/c++/ is
//! used.
//!
//! It is intended that after this module has run no C/C++ compiler will
//! ever be probed for. Instead the compilers found here will be used for
//! everything.
use std::process::Command;

use build_helper::{cc2ar, output};
Expand Down
21 changes: 18 additions & 3 deletions src/bootstrap/build/channel.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,13 @@
// option. This file may not be copied, modified, or distributed
// except according to those terms.

use std::env;
//! Build configuration for Rust's release channels.
//!
//! Implements the stable/beta/nightly channel distinctions by setting various
//! flags like the `unstable_features`, calculating variables like `release` and
//! `package_vers`, and otherwise indicating to the compiler what it should
//! print out as part of its version information.
use std::fs::{self, File};
use std::io::prelude::*;
use std::process::Command;
Expand All @@ -19,6 +25,9 @@ use md5;
use build::Build;

pub fn collect(build: &mut Build) {
// Currently the canonical source for the release number (e.g. 1.10.0) and
// the prerelease version (e.g. `.1`) is in `mk/main.mk`. We "parse" that
// here to learn about those numbers.
let mut main_mk = String::new();
t!(t!(File::open(build.src.join("mk/main.mk"))).read_to_string(&mut main_mk));
let mut release_num = "";
Expand All @@ -32,7 +41,8 @@ pub fn collect(build: &mut Build) {
}
}

// FIXME: this is duplicating makefile logic
// Depending on the channel, passed in `./configure --release-channel`,
// determine various properties of the build.
match &build.config.channel[..] {
"stable" => {
build.release = release_num.to_string();
Expand All @@ -58,6 +68,8 @@ pub fn collect(build: &mut Build) {
}
build.version = build.release.clone();

// If we have a git directory, add in some various SHA information of what
// commit this compiler was compiled from.
if fs::metadata(build.src.join(".git")).is_ok() {
let ver_date = output(Command::new("git").current_dir(&build.src)
.arg("log").arg("-1")
Expand All @@ -80,11 +92,14 @@ pub fn collect(build: &mut Build) {
build.short_ver_hash = Some(short_ver_hash);
}

// Calculate this compiler's bootstrap key, which is currently defined as
// the first 8 characters of the md5 of the release string.
let key = md5::compute(build.release.as_bytes());
build.bootstrap_key = format!("{:02x}{:02x}{:02x}{:02x}",
key[0], key[1], key[2], key[3]);
env::set_var("RUSTC_BOOTSTRAP_KEY", &build.bootstrap_key);

// Slurp up the stage0 bootstrap key as we're bootstrapping from an
// otherwise stable compiler.
let mut s = String::new();
t!(t!(File::open(build.src.join("src/stage0.txt"))).read_to_string(&mut s));
if let Some(line) = s.lines().find(|l| l.starts_with("rustc_key")) {
Expand Down
Loading

0 comments on commit f72bfe6

Please sign in to comment.