From 6bf0a5cb5034a7e684dcc3500e841785237ce2dd Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sun, 7 Apr 2024 19:32:43 +0200 Subject: Adding upstream version 1:115.7.0. Signed-off-by: Daniel Baumann --- build/docs/sparse.rst | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 build/docs/sparse.rst (limited to 'build/docs/sparse.rst') diff --git a/build/docs/sparse.rst b/build/docs/sparse.rst new file mode 100644 index 0000000000..6dcf548334 --- /dev/null +++ b/build/docs/sparse.rst @@ -0,0 +1,157 @@ +.. _build_sparse: + +================ +Sparse Checkouts +================ + +The Firefox repository is large: over 230,000 files. That many files +can put a lot of strain on machines, tools, and processes. + +Some version control tools have the ability to only populate a +working directory / checkout with a subset of files in the repository. +This is called *sparse checkout*. + +Various tools in the Firefox repository are configured to work +when a sparse checkout is being used. + +Sparse Checkouts in Mercurial +============================= + +Mercurial 4.3 introduced **experimental** support for sparse checkouts +in the official distribution (a Facebook-authored extension has +implemented the feature as a 3rd party extension for years). + +To enable sparse checkout support in Mercurial, enable the ``sparse`` +extension:: + + [extensions] + sparse = + +The *sparseness* of the working directory is managed using +``hg debugsparse``. Run ``hg help debugsparse`` and ``hg help -e sparse`` +for more info on the feature. + +When a *sparse config* is enabled, the working directory only contains +files matching that config. You cannot ``hg add`` or ``hg remove`` files +outside the *sparse config*. + +.. warning:: + + Sparse support in Mercurial 4.3 does not have any backwards + compatibility guarantees. Expect things to change. Scripting against + commands or relying on behavior is strongly discouraged. + +In-Tree Sparse Profiles +======================= + +Mercurial supports defining the sparse config using files under version +control. These are called *sparse profiles*. + +Essentially, the sparse profiles are managed just like any other file in +the repository. When you ``hg update``, the sparse configuration is +evaluated against the sparse profile at the revision being updated to. +From an end-user perspective, you just need to *activate* a profile once +and files will be added or removed as appropriate whenever the versioned +profile file updates. + +In the Firefox repository, the ``build/sparse-profiles`` directory +contains Mercurial *sparse profiles* files. + +Each *sparse profile* essentially defines a list of file patterns +(see ``hg help patterns``) to include or exclude. See +``hg help -e sparse`` for more. + +Mach Support for Sparse Checkouts +================================= + +``mach`` detects when a sparse checkout is being used and its +behavior may vary to accommodate this. + +By default it is a fatal error if ``mach`` can't load one of the +``mach_commands.py`` files it was told to. But if a sparse checkout +is being used, ``mach`` assumes that file isn't part of the sparse +checkout and to ignore missing file errors. This means that +running ``mach`` inside a sparse checkout will only have access +to the commands defined in files in the sparse checkout. + +Sparse Checkouts in Automation +============================== + +``hg robustcheckout`` (the extension/command used to perform clones +and working directory operations in automation) supports sparse checkout. +However, it has a number of limitations over Mercurial's default sparse +checkout implementation: + +* Only supports 1 profile at a time +* Does not support non-profile sparse configs +* Does not allow transitioning from a non-sparse to sparse checkout or + vice-versa + +These restrictions ensure that any sparse working directory populated by +``hg robustcheckout`` is as consistent and robust as possible. + +``run-task`` (the low-level script for *bootstrapping* tasks in +automation) has support for sparse checkouts. + +TaskGraph tasks using ``run-task`` can specify a ``sparse-profile`` +attribute in YAML (or in code) to denote the sparse profile file to +use. e.g.:: + + run: + using: run-command + command: + sparse-profile: taskgraph + +This automagically results in ``run-task`` and ``hg robustcheckout`` +using the sparse profile defined in ``build/sparse-profiles/``. + +Pros and Cons of Sparse Checkouts +================================= + +The benefits of sparse checkout are that it makes the repository appear +to be smaller. This means: + +* Less time performing working directory operations -> faster version + control operations +* Fewer files to consult -> faster operations +* Working directories only contain what is needed -> easier to understand + what everything does + +Fewer files in the working directory also contributes to disadvantages: + +* Searching may not yield hits because a file isn't in the sparse + checkout. e.g. a *global* search and replace may not actually be + *global* after all. +* Tools performing filesystem walking or path globbing (e.g. + ``**/*.js``) may fail to find files because they don't exist. +* Various tools and processes make assumptions that all files in the + repository are always available. + +There can also be problems caused by mixing sparse and non-sparse +checkouts. For example, if a process in automation is using sparse +and a local developer is not using sparse, things may work for the +local developer but fail in automation (because a file isn't included +in the sparse configuration and not available to automation. +Furthermore, if environments aren't using exactly the same sparse +configuration, differences can contribute to varying behavior. + +When Should Sparse Checkouts Be Used? +===================================== + +Developers are discouraged from using sparse checkouts for local work +until tools for handling sparse checkouts have improved. In particular, +Mercurial's support for sparse is still experimental and various Firefox +tools make assumptions that all files are available. Developers should +use sparse checkout at their own risk. + +The use of sparse checkouts in automation is a performance versus +robustness trade-off. Use of sparse checkouts will make automation +faster because machines will only have to manage a few thousand files +in a checkout instead of a few hundred thousand. This can potentially +translate to minutes saved per machine day. At the scale of thousands +of machines, the savings can be significant. But adopting sparse +checkouts will open up new avenues for failures. (See section above.) +If a process is isolated (in terms of file access) and well-understood, +sparse checkout can likely be leveraged with little risk. But if a +process is doing things like walking the filesystem and performing +lots of wildcard matching, the dangers are higher. -- cgit v1.2.3