Adding upstream version 110.0.1.upstream/110.0.1upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 228 insertions, 0 deletions

diff --git a/python/docs/index.rst b/python/docs/index.rst
new file mode 100644
index 0000000000..2d1167a135
--- /dev/null
+++ b/python/docs/index.rst
@@ -0,0 +1,228 @@
+=================================
+Using third-party Python packages
+=================================
+
+Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these
+are vendored in ``third_party/python``, while others are installed at runtime via ``pip``.
+
+The dependencies of Mach itself can be found at ``python/sites/mach.txt``. Mach commands
+may have additional dependencies which are specified at ``python/sites/<site>.txt``.
+
+For example, the following Mach command would have its 3rd-party dependencies declared at
+``python/sites/foo.txt``.
+
+.. code:: python
+
+    @Command(
+        "foo-it",
+        virtualenv_name="foo",
+    )
+    # ...
+    def foo_it_command():
+        import specific_dependency
+
+The format of ``<site>_virtualenv_requirements.txt`` files are documented further in the
+:py:class:`~mach.requirements.MachEnvRequirements` class.
+
+Adding a Python package
+=======================
+
+There's two ways of using 3rd-party Python dependencies:
+
+* :ref:`pip install the packages <python-pip-install>`. Python dependencies with native code must
+  be installed using ``pip``. This is the recommended technique for adding new Python dependencies.
+* :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach
+  core logic or of building Firefox itself must be vendored.
+
+.. note::
+
+    For dependencies that meet both restrictions (dependency of Mach/build, *and* has
+    native code), see the :ref:`mach-and-build-native-dependencies` section below.
+
+.. _python-pip-install:
+
+``pip install`` the package
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To add a ``pip install``-d package dependency, add it to your site's
+``python/sites/<site>.txt`` manifest file:
+
+.. code::
+
+    ...
+    pypi:new-package==<version>
+    ...
+
+.. note::
+
+    Some tasks are not permitted to use external resources, and for those we can
+    publish packages to an internal PyPI mirror.
+    See `how to upload to internal PyPI <https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi>`_
+    for more details.
+
+.. _python-vendor:
+
+Vendoring Python packages
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To vendor a Python package, add it to ``third_party/python/requirements.in``
+and then run ``mach vendor python``. This will update the tree of pinned
+dependencies in ``third_party/python/requirements.txt`` and download them all
+into the ``third_party/python`` directory.
+
+Next, add that package and any new transitive dependencies (you'll see them added in
+``third_party/python/requirements.txt``) to the associated site's dependency manifest in
+``python/sites/<site>.txt``:
+
+.. code::
+
+    ...
+    vendored:third_party/python/new-package
+    vendored:third_party/python/new-package-dependency-foo
+    vendored:third_party/python/new-package-dependency-bar
+    ...
+
+.. note::
+
+    The following policy applies to **ALL** vendored packages:
+
+    * Vendored PyPI libraries **MUST NOT** be modified
+    * Vendored libraries **SHOULD** be released copies of libraries available on
+      PyPI.
+
+      * When considering manually vendoring a package, discuss the situation with
+        the ``#build`` team to ensure that other, more maintainable options are exhausted.
+
+.. note::
+
+    We require that it is possible to build Firefox using only a checkout of the source,
+    without depending on a package index. This ensures that building Firefox is
+    deterministic and dependable, avoids packages from changing out from under us,
+    and means we’re not affected when 3rd party services are offline. We don't want a
+    DoS against PyPI or a random package maintainer removing an old tarball to delay
+    a Firefox chemspill. Therefore, packages required by Mach core logic or for building
+    Firefox itself must be vendored.
+
+.. _mach-and-build-native-dependencies:
+
+Mach/Build Native 3rd-party Dependencies
+========================================
+
+There are cases where Firefox is built without being able to ``pip install``, but where
+native 3rd party Python dependencies enable optional functionality. This can't be solved
+by vendoring the platform-specific libraries, as then each one would have to be stored
+multiple times in-tree according to how many platforms we wish to support.
+
+Instead, this is solved by pre-installing such native packages onto the host system
+in advance, then having Mach attempt to use such packages directly from the system.
+This feature is only viable in very specific environments, as the system Python packages
+have to be compatible with Mach's vendored packages.
+
+.. note:
+
+    All of these native build-specific dependencies **MUST** be optional requirements
+    as to support the "no strings attached" builds that only use vendored packages.
+
+To control this behaviour, the ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment
+variable can be used:
+
+.. list-table:: ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
+    :header-rows: 1
+
+    * - ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
+      - Behaviour
+    * - ``"pip"``
+      - Mach will ``pip install`` all needed dependencies from PyPI at runtime into a Python
+        virtual environment that's reused in future Mach invocations.
+    * - ``"none"``
+      - Mach will perform the build using only vendored packages. No Python virtual environment
+        will be created for Mach.
+    * - ``"system"``
+      - Mach will use the host system's Python packages as part of doing the build. This option
+        allows the usage of native Python packages without leaning on a ``pip install`` at
+        build-time. This is generally slower because the system Python packages have to
+        be asserted to be compatible with Mach. Additionally, dependency lockfiles are ignored,
+        so there's higher risk of breakage. Finally, as with ``"none"``, no Python virtualenv
+        environment is created for Mach.
+    * - ``<unset>``
+      - Same behaviour as ``"pip"`` if ``MOZ_AUTOMATION`` isn't set. Otherwise, uses
+        the same behaviour as ``"none"``.
+
+There's a couple restrictions here:
+
+* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` only applies to the top-level ``"mach"`` site,
+   the ``"common"`` site and the ``"build"`` site. All other sites will use ``pip install`` at
+   run-time as needed.
+
+* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"`` is not allowed when using any site other
+  than ``"mach"``, ``"common"`` or ``"build"``, because:
+
+  * As described in :ref:`package-compatibility` below, packages used by Mach are still
+    in scope when commands are run, and
+  * The host system is practically guaranteed to be incompatible with commands' dependency
+    lockfiles.
+
+The ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment variable fits into the following use
+cases:
+
+Mozilla CI Builds
+~~~~~~~~~~~~~~~~~
+
+We need access to the native packages of ``zstandard`` and ``psutil`` to extract archives and
+get OS information respectively. Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"``.
+
+Mozilla CI non-Build Tasks
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We generally don't want to create a Mach virtual environment to avoid redundant processing,
+but it's ok to ``pip install`` for specific command sites as needed, so leave
+``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset (``MOZ_AUTOMATION`` implies the default
+behaviour of ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``).
+
+In cases where native packages *are* needed by Mach, use
+``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="pip"``.
+
+Downstream CI Builds
+~~~~~~~~~~~~~~~~~~~~
+
+Sometimes these builds happen in sandboxed, network-less environments, and usually these builds
+don't need any of the behaviour enabled by installing native Python dependencies.
+Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
+
+Gentoo Builds
+~~~~~~~~~~~~~
+
+When installing Firefox via the package manager, Gentoo generally builds it from source rather than
+distributing a compiled binary artifact. Accordingly, users doing a build of Firefox in this
+context don't want stray files created in ``~/.mozbuild`` or unnecessary ``pip install`` calls.
+Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
+
+Firefox Developers
+~~~~~~~~~~~~~~~~~~
+
+Leave ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset so that all Mach commands can be run,
+Python dependency lockfiles are respected, and optional behaviour is enabled by installing
+native packages.
+
+.. _package-compatibility:
+
+Package compatibility
+=====================
+
+Mach requires that all commands' package requirements be compatible with those of Mach itself.
+(This is because functions and state created by Mach are still usable from within the commands, and
+they may still need access to their associated 3rd-party modules).
+
+However, it is OK for Mach commands to have package requirements which are incompatible with each
+other. This allows the flexibility for some Mach commands to depend on modern dependencies while
+other, more mature commands may still only be compatible with a much older version.
+
+.. note::
+
+    Only one version of a package may be vendored at any given time. If two Mach commands need to
+    have conflicting packages, then at least one of them must ``pip install`` the package instead
+    of vendoring.
+
+    If a Mach command's dependency conflicts with a vendored package, and that vendored package
+    isn't needed by Mach itself, then that vendored dependency should be moved from
+    ``python/sites/mach.txt`` to its associated environment.