summaryrefslogtreecommitdiffstats
path: root/src/bootstrap/README.md
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-17 12:02:58 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-17 12:02:58 +0000
commit698f8c2f01ea549d77d7dc3338a12e04c11057b9 (patch)
tree173a775858bd501c378080a10dca74132f05bc50 /src/bootstrap/README.md
parentInitial commit. (diff)
downloadrustc-698f8c2f01ea549d77d7dc3338a12e04c11057b9.tar.xz
rustc-698f8c2f01ea549d77d7dc3338a12e04c11057b9.zip
Adding upstream version 1.64.0+dfsg1.upstream/1.64.0+dfsg1
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'src/bootstrap/README.md')
-rw-r--r--src/bootstrap/README.md333
1 files changed, 333 insertions, 0 deletions
diff --git a/src/bootstrap/README.md b/src/bootstrap/README.md
new file mode 100644
index 000000000..a2e596bf4
--- /dev/null
+++ b/src/bootstrap/README.md
@@ -0,0 +1,333 @@
+# rustbuild - Bootstrapping Rust
+
+This is an in-progress README which is targeted at helping to explain how Rust
+is bootstrapped and in general some of the technical details of the build
+system.
+
+## Using rustbuild
+
+The rustbuild build system has a primary entry point, a top level `x.py` script:
+
+```sh
+$ python ./x.py build
+```
+
+Note that if you're on Unix you should be able to execute the script directly:
+
+```sh
+$ ./x.py build
+```
+
+The script accepts commands, flags, and arguments to determine what to do:
+
+* `build` - a general purpose command for compiling code. Alone `build` will
+ bootstrap the entire compiler, and otherwise arguments passed indicate what to
+ build. For example:
+
+ ```
+ # build the whole compiler
+ ./x.py build --stage 2
+
+ # build the stage1 compiler
+ ./x.py build
+
+ # build stage0 libstd
+ ./x.py build --stage 0 library/std
+
+ # build a particular crate in stage0
+ ./x.py build --stage 0 library/test
+ ```
+
+ If files are dirty that would normally be rebuilt from stage 0, that can be
+ overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
+ that belong to stage n or earlier:
+
+ ```
+ # build stage 1, keeping old build products for stage 0
+ ./x.py build --keep-stage 0
+ ```
+
+* `test` - a command for executing unit tests. Like the `build` command this
+ will execute the entire test suite by default, and otherwise it can be used to
+ select which test suite is run:
+
+ ```
+ # run all unit tests
+ ./x.py test
+
+ # execute tool tests
+ ./x.py test tidy
+
+ # execute the UI test suite
+ ./x.py test src/test/ui
+
+ # execute only some tests in the UI test suite
+ ./x.py test src/test/ui --test-args substring-of-test-name
+
+ # execute tests in the standard library in stage0
+ ./x.py test --stage 0 library/std
+
+ # execute tests in the core and standard library in stage0,
+ # without running doc tests (thus avoid depending on building the compiler)
+ ./x.py test --stage 0 --no-doc library/core library/std
+
+ # execute all doc tests
+ ./x.py test src/doc
+ ```
+
+* `doc` - a command for building documentation. Like above can take arguments
+ for what to document.
+
+## Configuring rustbuild
+
+There are currently two methods for configuring the rustbuild build system.
+
+First, rustbuild offers a TOML-based configuration system with a `config.toml`
+file. An example of this configuration can be found at `config.toml.example`,
+and the configuration file can also be passed as `--config path/to/config.toml`
+if the build system is being invoked manually (via the python script).
+
+Next, the `./configure` options serialized in `config.mk` will be
+parsed and read. That is, if any `./configure` options are passed, they'll be
+handled naturally. `./configure` should almost never be used for local
+installations, and is primarily useful for CI. Prefer to customize behavior
+using `config.toml`.
+
+Finally, rustbuild makes use of the [cc-rs crate] which has [its own
+method][env-vars] of configuring C compilers and C flags via environment
+variables.
+
+[cc-rs crate]: https://github.com/alexcrichton/cc-rs
+[env-vars]: https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
+
+## Build stages
+
+The rustbuild build system goes through a few phases to actually build the
+compiler. What actually happens when you invoke rustbuild is:
+
+1. The entry point script, `x.py` is run. This script is
+ responsible for downloading the stage0 compiler/Cargo binaries, and it then
+ compiles the build system itself (this folder). Finally, it then invokes the
+ actual `bootstrap` binary build system.
+2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
+ sanity checks (compilers exist for example), and then start building the
+ stage0 artifacts.
+3. The stage0 `cargo` downloaded earlier is used to build the standard library
+ and the compiler, and then these binaries are then copied to the `stage1`
+ directory. That compiler is then used to generate the stage1 artifacts which
+ are then copied to the stage2 directory, and then finally the stage2
+ artifacts are generated using that compiler.
+
+The goal of each stage is to (a) leverage Cargo as much as possible and failing
+that (b) leverage Rust as much as possible!
+
+## Incremental builds
+
+You can configure rustbuild to use incremental compilation with the
+`--incremental` flag:
+
+```sh
+$ ./x.py build --incremental
+```
+
+The `--incremental` flag will store incremental compilation artifacts
+in `build/<host>/stage0-incremental`. Note that we only use incremental
+compilation for the stage0 -> stage1 compilation -- this is because
+the stage1 compiler is changing, and we don't try to cache and reuse
+incremental artifacts across different versions of the compiler.
+
+You can always drop the `--incremental` to build as normal (but you
+will still be using the local nightly as your bootstrap).
+
+## Directory Layout
+
+This build system houses all output under the `build` directory, which looks
+like this:
+
+```sh
+# Root folder of all output. Everything is scoped underneath here
+build/
+
+ # Location where the stage0 compiler downloads are all cached. This directory
+ # only contains the tarballs themselves as they're extracted elsewhere.
+ cache/
+ 2015-12-19/
+ 2016-01-15/
+ 2016-01-21/
+ ...
+
+ # Output directory for building this build system itself. The stage0
+ # cargo/rustc are used to build the build system into this location.
+ bootstrap/
+ debug/
+ release/
+
+ # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
+ dist/
+
+ # Temporary directory used for various input/output as part of various stages
+ tmp/
+
+ # Each remaining directory is scoped by the "host" triple of compilation at
+ # hand.
+ x86_64-unknown-linux-gnu/
+
+ # The build artifacts for the `compiler-rt` library for the target this
+ # folder is under. The exact layout here will likely depend on the platform,
+ # and this is also built with CMake so the build system is also likely
+ # different.
+ compiler-rt/
+ build/
+
+ # Output folder for LLVM if it is compiled for this target
+ llvm/
+
+ # build folder (e.g. the platform-specific build system). Like with
+ # compiler-rt this is compiled with CMake
+ build/
+
+ # Installation of LLVM. Note that we run the equivalent of 'make install'
+ # for LLVM to setup these folders.
+ bin/
+ lib/
+ include/
+ share/
+ ...
+
+ # Output folder for all documentation of this target. This is what's filled
+ # in whenever the `doc` step is run.
+ doc/
+
+ # Output for all compiletest-based test suites
+ test/
+ ui/
+ debuginfo/
+ ...
+
+ # Location where the stage0 Cargo and Rust compiler are unpacked. This
+ # directory is purely an extracted and overlaid tarball of these two (done
+ # by the bootstrapy python script). In theory the build system does not
+ # modify anything under this directory afterwards.
+ stage0/
+
+ # These to build directories are the cargo output directories for builds of
+ # the standard library and compiler, respectively. Internally these may also
+ # have other target directories, which represent artifacts being compiled
+ # from the host to the specified target.
+ #
+ # Essentially, each of these directories is filled in by one `cargo`
+ # invocation. The build system instruments calling Cargo in the right order
+ # with the right variables to ensure these are filled in correctly.
+ stageN-std/
+ stageN-test/
+ stageN-rustc/
+ stageN-tools/
+
+ # This is a special case of the above directories, **not** filled in via
+ # Cargo but rather the build system itself. The stage0 compiler already has
+ # a set of target libraries for its own host triple (in its own sysroot)
+ # inside of stage0/. When we run the stage0 compiler to bootstrap more
+ # things, however, we don't want to use any of these libraries (as those are
+ # the ones that we're building). So essentially, when the stage1 compiler is
+ # being compiled (e.g. after libstd has been built), *this* is used as the
+ # sysroot for the stage0 compiler being run.
+ #
+ # Basically this directory is just a temporary artifact use to configure the
+ # stage0 compiler to ensure that the libstd we just built is used to
+ # compile the stage1 compiler.
+ stage0-sysroot/lib/
+
+ # These output directories are intended to be standalone working
+ # implementations of the compiler (corresponding to each stage). The build
+ # system will link (using hard links) output from stageN-{std,rustc} into
+ # each of these directories.
+ #
+ # In theory there is no extra build output in these directories.
+ stage1/
+ stage2/
+ stage3/
+```
+
+## Cargo projects
+
+The current build is unfortunately not quite as simple as `cargo build` in a
+directory, but rather the compiler is split into three different Cargo projects:
+
+* `library/std` - the standard library
+* `library/test` - testing support, depends on libstd
+* `compiler/rustc` - the actual compiler itself
+
+Each "project" has a corresponding Cargo.lock file with all dependencies, and
+this means that building the compiler involves running Cargo three times. The
+structure here serves two goals:
+
+1. Facilitating dependencies coming from crates.io. These dependencies don't
+ depend on `std`, so libstd is a separate project compiled ahead of time
+ before the actual compiler builds.
+2. Splitting "host artifacts" from "target artifacts". That is, when building
+ code for an arbitrary target you don't need the entire compiler, but you'll
+ end up needing libraries like libtest that depend on std but also want to use
+ crates.io dependencies. Hence, libtest is split out as its own project that
+ is sequenced after `std` but before `rustc`. This project is built for all
+ targets.
+
+There is some loss in build parallelism here because libtest can be compiled in
+parallel with a number of rustc artifacts, but in theory the loss isn't too bad!
+
+## Build tools
+
+We've actually got quite a few tools that we use in the compiler's build system
+and for testing. To organize these, each tool is a project in `src/tools` with a
+corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
+independent `Cargo.lock` files) and do not currently explicitly depend on the
+compiler or standard library. Compiling each tool is sequenced after the
+appropriate libstd/libtest/librustc compile above.
+
+## Extending rustbuild
+
+So you'd like to add a feature to the rustbuild build system or just fix a bug.
+Great! One of the major motivational factors for moving away from `make` is that
+Rust is in theory much easier to read, modify, and write. If you find anything
+excessively confusing, please open an issue on this and we'll try to get it
+documented or simplified pronto.
+
+First up, you'll probably want to read over the documentation above as that'll
+give you a high level overview of what rustbuild is doing. You also probably
+want to play around a bit yourself by just getting it up and running before you
+dive too much into the actual build system itself.
+
+After that, each module in rustbuild should have enough documentation to keep
+you up and running. Some general areas that you may be interested in modifying
+are:
+
+* Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of
+ other tools.
+* Adding a new compiler crate? Look no further! Adding crates can be done by
+ adding a new directory with `Cargo.toml` followed by configuring all
+ `Cargo.toml` files accordingly.
+* Adding a new dependency from crates.io? This should just work inside the
+ compiler artifacts stage (everything other than libtest and libstd).
+* Adding a new configuration option? You'll want to modify `bootstrap/flags.rs`
+ for command line flags and then `bootstrap/config.rs` to copy the flags to the
+ `Config` struct.
+* Adding a sanity check? Take a look at `bootstrap/sanity.rs`.
+
+If you make a major change, please remember to:
+
++ Update `VERSION` in `src/bootstrap/main.rs`.
+* Update `changelog-seen = N` in `config.toml.example`.
+* Add an entry in `src/bootstrap/CHANGELOG.md`.
+
+A 'major change' includes
+
+* A new option or
+* A change in the default options.
+
+Changes that do not affect contributors to the compiler or users
+building rustc from source don't need an update to `VERSION`.
+
+If you have any questions feel free to reach out on the `#t-infra` channel in
+the [Rust Zulip server][rust-zulip] or ask on internals.rust-lang.org. When
+you encounter bugs, please file issues on the rust-lang/rust issue tracker.
+
+[rust-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra