From 43a97878ce14b72f0981164f87f2e35e14151312 Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sun, 7 Apr 2024 11:22:09 +0200 Subject: Adding upstream version 110.0.1. Signed-off-by: Daniel Baumann --- build/docs/mozbuild-files.rst | 176 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 build/docs/mozbuild-files.rst (limited to 'build/docs/mozbuild-files.rst') diff --git a/build/docs/mozbuild-files.rst b/build/docs/mozbuild-files.rst new file mode 100644 index 0000000000..9d69404732 --- /dev/null +++ b/build/docs/mozbuild-files.rst @@ -0,0 +1,176 @@ +.. _mozbuild-files: + +=============== +moz.build Files +=============== + +``moz.build`` files are the mechanism by which tree metadata (notably +the build configuration) is defined. + +Directories in the tree contain ``moz.build`` files which declare +functionality for their respective part of the tree. This includes +things such as the list of C++ files to compile, where to find tests, +etc. + +``moz.build`` files are actually Python scripts. However, their +execution is governed by special rules. This is explained below. + +moz.build Python Sandbox +======================== + +As mentioned above, ``moz.build`` files are Python scripts. However, +they are executed in a special Python *sandbox* that significantly +changes and limits the execution environment. The environment is so +different, it's doubtful most ``moz.build`` files would execute without +error if executed by a vanilla Python interpreter (e.g. ``python +moz.build``. + +The following properties make execution of ``moz.build`` files special: + +1. The execution environment exposes a limited subset of Python. +2. There is a special set of global symbols and an enforced naming + convention of symbols. +3. Some symbols are inherited from previously-executed ``moz.build`` + files. + +The limited subset of Python is actually an extremely limited subset. +Only a few symbols from ``__builtin__`` are exposed. These include +``True``, ``False``, ``None``, ``sorted``, ``int``, and ``set``. Global +functions like ``import``, ``print``, and ``open`` aren't available. +Without these, ``moz.build`` files can do very little. *This is by design*. + +The execution sandbox treats all ``UPPERCASE`` variables specially. Any +``UPPERCASE`` variable must be known to the sandbox before the script +executes. Any attempt to read or write to an unknown ``UPPERCASE`` +variable will result in an exception being raised. Furthermore, the +types of all ``UPPERCASE`` variables is strictly enforced. Attempts to +assign an incompatible type to an ``UPPERCASE`` variable will result in +an exception being raised. + +The strictness of behavior with ``UPPERCASE`` variables is a very +intentional design decision. By ensuring strict behavior, any operation +involving an ``UPPERCASE`` variable is guaranteed to have well-defined +side-effects. Previously, when the build configuration was defined in +``Makefiles``, assignments to variables that did nothing would go +unnoticed. ``moz.build`` files fix this problem by eliminating the +potential for false promises. + +After a ``moz.build`` file has completed execution, only the +``UPPERCASE`` variables are used to retrieve state. + +The set of variables and functions available to the Python sandbox is +defined by the :py:mod:`mozbuild.frontend.context` module. The +data structures in this module are consumed by the +:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct +the sandbox. There are tests to ensure that the set of symbols exposed +to an empty sandbox are all defined in the ``context`` module. +This module also contains documentation for each symbol, so nothing can +sneak into the sandbox without being explicitly defined and documented. + +Reading and Traversing moz.build Files +====================================== + +The process for reading ``moz.build`` files roughly consists of: + +1. Start at the root ``moz.build`` (``/moz.build``). +2. Evaluate the ``moz.build`` file in a new sandbox. +3. Emit the main *context* and any *sub-contexts* from the executed + sandbox. +4. Extract a set of ``moz.build`` files to execute next. +5. For each additional ``moz.build`` file, goto #2 and repeat until all + referenced files have executed. + +From the perspective of the consumer, the output of reading is a stream +of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each +``Context`` defines a particular aspect of data. Consumers iterate over +these objects and do something with the data inside. Each object is +essentially a dictionary of all the ``UPPERCASE`` variables populated +during its execution. + +.. note:: + + Historically, there was only one ``context`` per ``moz.build`` file. + As the number of things tracked by ``moz.build`` files grew and more + and more complex processing was desired, it was necessary to split these + contexts into multiple logical parts. It is now common to emit + multiple contexts per ``moz.build`` file. + +Build System Reading Mode +------------------------- + +The traditional mode of evaluation of ``moz.build`` files is what's +called *build system traversal mode.* In this mode, the ``CONFIG`` +variable in each ``moz.build`` sandbox is populated from data coming +from ``config.status``, which is produced by ``configure``. + +During evaluation, ``moz.build`` files often make decisions conditional +on the state of the build configuration. e.g. *only compile foo.cpp if +feature X is enabled*. + +In this mode, traversal of ``moz.build`` files is governed by variables +like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child +directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build`` +file and ``foo/moz.build`` would be evaluated. + +.. _mozbuild_fs_reading_mode: + +Filesystem Reading Mode +----------------------- + +There is an alternative reading mode that doesn't involve the build +system and doesn't use ``DIRS`` variables to control traversal into +child directories. This mode is called *filesystem reading mode*. + +In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty +object. Accessing all but a few special variables will return an empty +value. This means that nearly all ``if CONFIG['FOO']:`` branches will +not be taken. + +Instead of using content from within the evaluated ``moz.build`` +file to drive traversal into subsequent ``moz.build`` files, the set +of files to evaluate is controlled by the thing doing the reading. + +A single ``moz.build`` file is not guaranteed to be executable in +isolation. Instead, we must evaluate all *parent* ``moz.build`` files +first. For example, in order to evaluate ``/foo/moz.build``, one must +execute ``/moz.build`` and have its state influence the execution of +``/foo/moz.build``. + +Filesystem reading mode is utilized to power the +:ref:`mozbuild_files_metadata` feature. + +Technical Details +----------------- + +The code for reading ``moz.build`` files lives in +:py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results +(:py:class:`mozbuild.frontend.context.Context`) are passed into +:py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined +in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a +domain-specific component of tree metadata. e.g. there will be separate +classes that represent a JavaScript file vs a compiled C++ file or test +manifests. This means downstream consumers of this data can filter on class +types to only consume what they are interested in. + +There is no well-defined mapping between ``moz.build`` file instances +and the number of :py:mod:`mozbuild.frontend.data` classes derived from +each. Depending on the content of the ``moz.build`` file, there may be 1 +object derived or 100. + +The purpose of the ``emitter`` layer between low-level sandbox execution +and metadata representation is to facilitate a unified normalization and +verification step. There are multiple downstream consumers of the +``moz.build``-derived data and many will perform the same actions. This +logic can be complicated, so we have a component dedicated to it. + +:py:class:`mozbuild.frontend.reader.BuildReader`` and +:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a +stream-based API courtesy of generators. When you hook them up properly, +the :py:mod:`mozbuild.frontend.data` classes are emitted before all +``moz.build`` files have been read. This means that downstream errors +are raised soon after sandbox execution. + +Lots of the code for evaluating Python sandboxes is applicable to +non-Mozilla systems. In theory, it could be extracted into a standalone +and generic package. However, until there is a need, there will +likely be some tightly coupled bits. -- cgit v1.2.3