Adding upstream version 110.0.1.upstream/110.0.1upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 194 insertions, 0 deletions

diff --git a/build/docs/sccache-dist.rst b/build/docs/sccache-dist.rst
new file mode 100644
index 0000000000..8087c0c017
--- /dev/null
+++ b/build/docs/sccache-dist.rst
@@ -0,0 +1,194 @@
+.. _sccache_dist:
+
+==================================
+Distributed sccache (sccache-dist)
+==================================
+
+`sccache <https://github.com/mozilla/sccache>`_ is a ccache-like tool written in
+Rust by Mozilla.
+
+The steps for setting up your machine as an sccache-dist server are detailed below.
+
+In addition to improved security properties, distributed sccache offers
+distribution and caching of rust compilation, so it should be an improvement
+above and beyond what we see with icecc. Build servers run on Linux and
+distributing builds is currently supported from Linux, macOS, and Windows.
+
+
+Steps for distributing a build as an sccache-dist client
+========================================================
+
+Start by following the instructions at https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-client
+to configure your sccache distributed client.
+*NOTE* If you're distributing from Linux a toolchain will be packaged
+automatically and provided to the build server. If you're distributing from
+Windows or macOS, start by using the cross-toolchains provided by
+``./mach bootstrap`` rather than attempting to use ``icecc-create-env``.
+sccache 0.2.12 or above is recommended, and the auth section of your config
+must read::
+
+    [dist.auth]
+    type = "mozilla"
+
+* If you're compiling from a macOS client, there are a handful of additional
+  considerations outlined here:
+  https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#considerations-when-distributing-from-macos.
+
+  Run ``./mach bootstrap`` to download prebuilt toolchains to
+  ``~/.mozbuild/clang-dist-toolchain.tar.xz`` and
+  ``~/.mozbuild/rustc-dist-toolchain.tar.xz``. This is an example of the paths
+  that should be added to your client config to specify toolchains to build on
+  macOS, located at ``~/Library/Application Support/Mozilla.sccache/config``::
+
+    [[dist.toolchains]]
+    type = "path_override"
+    compiler_executable = "/path/to/home/.rustup/toolchains/stable-x86_64-apple-darwin/bin/rustc"
+    archive = "/path/to/home/.mozbuild/rustc-dist-toolchain.tar.xz"
+    archive_compiler_executable = "/builds/worker/toolchains/rustc/bin/rustc"
+
+    [[dist.toolchains]]
+    type = "path_override"
+    compiler_executable = "/path/to/home/.mozbuild/clang/bin/clang"
+    archive = "/path/to/home/.mozbuild/clang-dist-toolchain.tar.xz"
+    archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
+
+    [[dist.toolchains]]
+    type = "path_override"
+    compiler_executable = "/path/to/home/.mozbuild/clang/bin/clang++"
+    archive = "/path/to/home/.mozbuild/clang-dist-toolchain.tar.xz"
+    archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
+
+  Note that the version of ``rustc`` found in ``rustc-dist-toolchain.tar.xz``
+  must match the version of ``rustc`` used locally. The distributed archive
+  will contain the version of ``rustc`` used by automation builds, which may
+  lag behind stable for a few days after Rust releases, which is specified by
+  the task definition in
+  `this file <https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ci/toolchain/dist-toolchains.yml>`_.
+  For instance, to specify 1.37.0 rather than the current stable, run
+  ``rustup toolchain add 1.37.0`` and point to
+  ``/path/to/home/.rustup/toolchains/1.37.0-x86_64-apple-darwin/bin/rustc`` in your
+  client config.
+
+  The build system currently requires an explicit target to be passed with
+  ``HOST_CFLAGS`` and ``HOST_CXXFLAGS`` e.g.::
+
+    export HOST_CFLAGS="--target=x86_64-apple-darwin16.0.0"
+    export HOST_CXXFLAGS="--target=x86_64-apple-darwin16.0.0"
+
+* Compiling from a Windows client is supported but hasn't seen as much testing
+  as other platforms. The following example mozconfig can be used as a guide::
+
+    ac_add_options CCACHE="C:/Users/<USER>/.mozbuild/sccache/sccache.exe"
+
+    export CC="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
+    export CXX="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
+    export HOST_CC="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
+    export HOST_CXX="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
+
+  The client config should be located at
+  ``~/AppData/Roaming/Mozilla/sccache/config/config``, and as on macOS custom
+  toolchains should be obtained with ``./mach bootstrap`` and specified in the
+  client config, for example::
+
+    [[dist.toolchains]]
+    type = "path_override"
+    compiler_executable = "C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe"
+    archive = "C:/Users/<USER>/.mozbuild/clang-dist-toolchain.tar.xz"
+    archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
+
+    [[dist.toolchains]]
+    type = "path_override"
+    compiler_executable = "C:/Users/<USER>/.rustup/toolchains/stable-x86_64-pc-windows-msvc/bin/rustc.exe"
+    archive = "C:/Users/<USER>/.mozbuild/rustc-dist-toolchain.tar.xz"
+    archive_compiler_executable = "/builds/worker/toolchains/rustc/bin/rustc"
+
+* Add the following to your mozconfig::
+
+    ac_add_options CCACHE=/path/to/home/.mozbuild/sccache/sccache
+
+  If you're compiling from a macOS client, you might need some additional configuration::
+
+    # Set the target flag to Darwin
+    export CFLAGS="--target=x86_64-apple-darwin16.0.0"
+    export CXXFLAGS="--target=x86_64-apple-darwin16.0.0"
+    export HOST_CFLAGS="--target=x86_64-apple-darwin16.0.0"
+    export HOST_CXXFLAGS="--target=x86_64-apple-darwin16.0.0"
+
+    # Specify the macOS SDK to use
+    ac_add_options --with-macos-sdk=/path/to/MacOSX-SDKs/MacOSX10.12.sdk
+
+  You can get the right macOS SDK by downloading an old version of XCode from
+  `developer.apple.com <https://developer.apple.com>`_ and unpacking the SDK
+  from it.
+
+* When attempting to get your client running, the output of ``sccache -s`` should
+  be consulted to confirm compilations are being distributed. To receive helpful
+  logging from the local daemon in case they aren't, run
+  ``SCCACHE_NO_DAEMON=1 SCCACHE_START_SERVER=1 SCCACHE_LOG=sccache=trace path/to/sccache``
+  in a terminal window separate from your build prior to building. *NOTE* use
+  ``RUST_LOG`` instead of ``SCCACHE_LOG`` if your build of ``sccache`` does not
+  include `pull request 822
+  <https://github.com/mozilla/sccache/pull/822>`_. (``sccache`` binaries from
+  ``mach bootstrap`` do include this PR.)
+
+* Run ``./mach build -j<value>`` with an appropriately large ``<value>``.
+  ``sccache --dist-status`` should provide the number of cores available to you
+  (or a message if you're not connected). In the future this will be integrated
+  with the build system to automatically select an appropriate value.
+
+This should be enough to distribute your build and replace your use of icecc.
+Bear in mind there may be a few speedbumps, and please ensure your version of
+sccache is current before investigating further. Please see the common questions
+section below and ask for help if anything is preventing you from using it over
+email (dev-builds), on slack in #sccache, or in #build on irc.
+
+Steps for setting up a server
+=============================
+
+Build servers must run linux and use bubblewrap 0.3.0+ for sandboxing of compile
+processes. This requires a kernel 4.6 or greater, so Ubuntu 18+, RHEL 8, or
+similar.
+
+* Run ``./mach bootstrap`` or
+  ``./mach artifact toolchain --from-build linux64-sccache`` to acquire a recent
+  version of ``sccache-dist``. Please use a ``sccache-dist`` binary acquired in
+  this fashion to ensure compatibility with statically linked dependencies.
+
+* The instructions at https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-build-server
+  should contain everything else required to configure and run the server.
+
+  *NOTE* Port 10500 will be used by convention for builders.
+  Please use port 10500 in the ``public_addr`` section of your builder config.
+
+  Extra logging may be helpful when setting up a server. To enable logging,
+  run your server with
+  ``sudo env SCCACHE_LOG=sccache=trace ~/.mozbuild/sccache/sccache-dist server --config ~/.config/sccache/server.conf``
+  (or similar). *NOTE* ``sudo`` *must* come before setting environment variables
+  for this to work. *NOTE* use ``RUST_LOG`` instead of ``SCCACHE_LOG`` if your
+  build of ``sccache`` does not include `pull request 822
+  <https://github.com/mozilla/sccache/pull/822>`_. (``sccache`` binaries from
+  ``mach bootstrap`` do include this PR.)
+
+
+Common questions/considerations
+===============================
+
+* My build is still slow: scache-dist can only do so much with parts of the
+  build that aren't able to be parallelized. To start debugging a slow build,
+  ensure the "Successful distributed compilations" line in the output of
+  ``sccache -s`` dominates other counts. For a full build, at least a 2-3x
+  improvement should be observed.
+
+* My build output is incomprehensible due to a flood of warnings: clang will
+  treat some warnings differently when it's fed preprocessed code in a separate
+  invocation (preprocessing occurs locally with sccache-dist). Adding
+  ``rewrite_includes_only = true`` to the ``dist`` section of your client config
+  will improve this; however, setting this will cause build failures with a
+  commonly deployed version of ``glibc``. This option will default to ``true``
+  once the fix is more widely available. Details of this fix can be found in
+  `this patch <https://sourceware.org/ml/libc-alpha/2019-11/msg00431.html>`_.
+
+* My build fails with a message about incompatible versions of rustc between
+  dependent crates: if you're using a custom toolchain check that the version
+  of rustc in your ``rustc-dist-toolchain.tar.xz`` is the same as the version
+  you're running locally.