summaryrefslogtreecommitdiffstats
path: root/python/docs
diff options
context:
space:
mode:
Diffstat (limited to 'python/docs')
-rw-r--r--python/docs/index.rst228
1 files changed, 228 insertions, 0 deletions
diff --git a/python/docs/index.rst b/python/docs/index.rst
new file mode 100644
index 0000000000..0c97270bbc
--- /dev/null
+++ b/python/docs/index.rst
@@ -0,0 +1,228 @@
+=================================
+Using third-party Python packages
+=================================
+
+Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these
+are vendored in ``third_party/python``, while others are installed at runtime via ``pip``.
+
+The dependencies of Mach itself can be found at ``python/sites/mach.txt``. Mach commands
+may have additional dependencies which are specified at ``python/sites/<site>.txt``.
+
+For example, the following Mach command would have its 3rd-party dependencies declared at
+``python/sites/foo.txt``.
+
+.. code:: python
+
+ @Command(
+ "foo-it",
+ virtualenv_name="foo",
+ )
+ # ...
+ def foo_it_command():
+ import specific_dependency
+
+The format of ``<site>.txt`` files are documented further in the
+:py:class:`~mach.requirements.MachEnvRequirements` class.
+
+Adding a Python package
+=======================
+
+There's two ways of using 3rd-party Python dependencies:
+
+* :ref:`pip install the packages <python-pip-install>`. Python dependencies with native code must
+ be installed using ``pip``. This is the recommended technique for adding new Python dependencies.
+* :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach
+ core logic or of building Firefox itself must be vendored.
+
+.. note::
+
+ For dependencies that meet both restrictions (dependency of Mach/build, *and* has
+ native code), see the :ref:`mach-and-build-native-dependencies` section below.
+
+.. _python-pip-install:
+
+``pip install`` the package
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To add a ``pip install``-d package dependency, add it to your site's
+``python/sites/<site>.txt`` manifest file:
+
+.. code:: text
+
+ ...
+ pypi:new-package==<version>
+ ...
+
+.. note::
+
+ Some tasks are not permitted to use external resources, and for those we can
+ publish packages to an internal PyPI mirror.
+ See `how to upload to internal PyPI <https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi>`_
+ for more details.
+
+.. _python-vendor:
+
+Vendoring Python packages
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To vendor a Python package, add it to ``third_party/python/requirements.in``
+and then run ``mach vendor python``. This will update the tree of pinned
+dependencies in ``third_party/python/requirements.txt`` and download them all
+into the ``third_party/python`` directory.
+
+Next, add that package and any new transitive dependencies (you'll see them added in
+``third_party/python/requirements.txt``) to the associated site's dependency manifest in
+``python/sites/<site>.txt``:
+
+.. code:: text
+
+ ...
+ vendored:third_party/python/new-package
+ vendored:third_party/python/new-package-dependency-foo
+ vendored:third_party/python/new-package-dependency-bar
+ ...
+
+.. note::
+
+ The following policy applies to **ALL** vendored packages:
+
+ * Vendored PyPI libraries **MUST NOT** be modified
+ * Vendored libraries **SHOULD** be released copies of libraries available on
+ PyPI.
+
+ * When considering manually vendoring a package, discuss the situation with
+ the ``#build`` team to ensure that other, more maintainable options are exhausted.
+
+.. note::
+
+ We require that it is possible to build Firefox using only a checkout of the source,
+ without depending on a package index. This ensures that building Firefox is
+ deterministic and dependable, avoids packages from changing out from under us,
+ and means we’re not affected when 3rd party services are offline. We don't want a
+ DoS against PyPI or a random package maintainer removing an old tarball to delay
+ a Firefox chemspill. Therefore, packages required by Mach core logic or for building
+ Firefox itself must be vendored.
+
+.. _mach-and-build-native-dependencies:
+
+Mach/Build Native 3rd-party Dependencies
+========================================
+
+There are cases where Firefox is built without being able to ``pip install``, but where
+native 3rd party Python dependencies enable optional functionality. This can't be solved
+by vendoring the platform-specific libraries, as then each one would have to be stored
+multiple times in-tree according to how many platforms we wish to support.
+
+Instead, this is solved by pre-installing such native packages onto the host system
+in advance, then having Mach attempt to use such packages directly from the system.
+This feature is only viable in very specific environments, as the system Python packages
+have to be compatible with Mach's vendored packages.
+
+.. note:
+
+ All of these native build-specific dependencies **MUST** be optional requirements
+ as to support the "no strings attached" builds that only use vendored packages.
+
+To control this behaviour, the ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment
+variable can be used:
+
+.. list-table:: ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
+ :header-rows: 1
+
+ * - ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
+ - Behaviour
+ * - ``"pip"``
+ - Mach will ``pip install`` all needed dependencies from PyPI at runtime into a Python
+ virtual environment that's reused in future Mach invocations.
+ * - ``"none"``
+ - Mach will perform the build using only vendored packages. No Python virtual environment
+ will be created for Mach.
+ * - ``"system"``
+ - Mach will use the host system's Python packages as part of doing the build. This option
+ allows the usage of native Python packages without leaning on a ``pip install`` at
+ build-time. This is generally slower because the system Python packages have to
+ be asserted to be compatible with Mach. Additionally, dependency lockfiles are ignored,
+ so there's higher risk of breakage. Finally, as with ``"none"``, no Python virtualenv
+ environment is created for Mach.
+ * - ``<unset>``
+ - Same behaviour as ``"pip"`` if ``MOZ_AUTOMATION`` isn't set. Otherwise, uses
+ the same behaviour as ``"none"``.
+
+There's a couple restrictions here:
+
+* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` only applies to the top-level ``"mach"`` site,
+ the ``"common"`` site and the ``"build"`` site. All other sites will use ``pip install`` at
+ run-time as needed.
+
+* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"`` is not allowed when using any site other
+ than ``"mach"``, ``"common"`` or ``"build"``, because:
+
+ * As described in :ref:`package-compatibility` below, packages used by Mach are still
+ in scope when commands are run, and
+ * The host system is practically guaranteed to be incompatible with commands' dependency
+ lockfiles.
+
+The ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment variable fits into the following use
+cases:
+
+Mozilla CI Builds
+~~~~~~~~~~~~~~~~~
+
+We need access to the native packages of ``zstandard`` and ``psutil`` to extract archives and
+get OS information respectively. Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"``.
+
+Mozilla CI non-Build Tasks
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We generally don't want to create a Mach virtual environment to avoid redundant processing,
+but it's ok to ``pip install`` for specific command sites as needed, so leave
+``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset (``MOZ_AUTOMATION`` implies the default
+behaviour of ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``).
+
+In cases where native packages *are* needed by Mach, use
+``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="pip"``.
+
+Downstream CI Builds
+~~~~~~~~~~~~~~~~~~~~
+
+Sometimes these builds happen in sandboxed, network-less environments, and usually these builds
+don't need any of the behaviour enabled by installing native Python dependencies.
+Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
+
+Gentoo Builds
+~~~~~~~~~~~~~
+
+When installing Firefox via the package manager, Gentoo generally builds it from source rather than
+distributing a compiled binary artifact. Accordingly, users doing a build of Firefox in this
+context don't want stray files created in ``~/.mozbuild`` or unnecessary ``pip install`` calls.
+Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
+
+Firefox Developers
+~~~~~~~~~~~~~~~~~~
+
+Leave ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset so that all Mach commands can be run,
+Python dependency lockfiles are respected, and optional behaviour is enabled by installing
+native packages.
+
+.. _package-compatibility:
+
+Package compatibility
+=====================
+
+Mach requires that all commands' package requirements be compatible with those of Mach itself.
+(This is because functions and state created by Mach are still usable from within the commands, and
+they may still need access to their associated 3rd-party modules).
+
+However, it is OK for Mach commands to have package requirements which are incompatible with each
+other. This allows the flexibility for some Mach commands to depend on modern dependencies while
+other, more mature commands may still only be compatible with a much older version.
+
+.. note::
+
+ Only one version of a package may be vendored at any given time. If two Mach commands need to
+ have conflicting packages, then at least one of them must ``pip install`` the package instead
+ of vendoring.
+
+ If a Mach command's dependency conflicts with a vendored package, and that vendored package
+ isn't needed by Mach itself, then that vendored dependency should be moved from
+ ``python/sites/mach.txt`` to its associated environment.