1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
|
=================================
Using third-party Python packages
=================================
Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these
are vendored in ``third_party/python``, while others are installed at runtime via ``pip``.
The dependencies of Mach itself can be found at ``python/sites/mach.txt``. Mach commands
may have additional dependencies which are specified at ``python/sites/<site>.txt``.
For example, the following Mach command would have its 3rd-party dependencies declared at
``python/sites/foo.txt``.
.. code:: python
@Command(
"foo-it",
virtualenv_name="foo",
)
# ...
def foo_it_command():
import specific_dependency
The format of ``<site>.txt`` files are documented further in the
:py:class:`~mach.requirements.MachEnvRequirements` class.
Adding a Python package
=======================
There's two ways of using 3rd-party Python dependencies:
* :ref:`pip install the packages <python-pip-install>`. Python dependencies with native code must
be installed using ``pip``. This is the recommended technique for adding new Python dependencies.
* :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach
core logic or of building Firefox itself must be vendored.
.. note::
For dependencies that meet both restrictions (dependency of Mach/build, *and* has
native code), see the :ref:`mach-and-build-native-dependencies` section below.
.. _python-pip-install:
``pip install`` the package
~~~~~~~~~~~~~~~~~~~~~~~~~~~
To add a ``pip install``-d package dependency, add it to your site's
``python/sites/<site>.txt`` manifest file:
.. code:: text
...
pypi:new-package==<version>
...
.. note::
Some tasks are not permitted to use external resources, and for those we can
publish packages to an internal PyPI mirror.
See `how to upload to internal PyPI <https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi>`_
for more details.
.. _python-vendor:
Vendoring Python packages
~~~~~~~~~~~~~~~~~~~~~~~~~
To vendor a Python package, add it to ``third_party/python/requirements.in``
and then run ``mach vendor python``. This will update the tree of pinned
dependencies in ``third_party/python/requirements.txt`` and download them all
into the ``third_party/python`` directory.
Next, add that package and any new transitive dependencies (you'll see them added in
``third_party/python/requirements.txt``) to the associated site's dependency manifest in
``python/sites/<site>.txt``:
.. code:: text
...
vendored:third_party/python/new-package
vendored:third_party/python/new-package-dependency-foo
vendored:third_party/python/new-package-dependency-bar
...
.. note::
The following policy applies to **ALL** vendored packages:
* Vendored PyPI libraries **MUST NOT** be modified
* Vendored libraries **SHOULD** be released copies of libraries available on
PyPI.
* When considering manually vendoring a package, discuss the situation with
the ``#build`` team to ensure that other, more maintainable options are exhausted.
.. note::
We require that it is possible to build Firefox using only a checkout of the source,
without depending on a package index. This ensures that building Firefox is
deterministic and dependable, avoids packages from changing out from under us,
and means we’re not affected when 3rd party services are offline. We don't want a
DoS against PyPI or a random package maintainer removing an old tarball to delay
a Firefox chemspill. Therefore, packages required by Mach core logic or for building
Firefox itself must be vendored.
.. _mach-and-build-native-dependencies:
Mach/Build Native 3rd-party Dependencies
========================================
There are cases where Firefox is built without being able to ``pip install``, but where
native 3rd party Python dependencies enable optional functionality. This can't be solved
by vendoring the platform-specific libraries, as then each one would have to be stored
multiple times in-tree according to how many platforms we wish to support.
Instead, this is solved by pre-installing such native packages onto the host system
in advance, then having Mach attempt to use such packages directly from the system.
This feature is only viable in very specific environments, as the system Python packages
have to be compatible with Mach's vendored packages.
.. note:
All of these native build-specific dependencies **MUST** be optional requirements
as to support the "no strings attached" builds that only use vendored packages.
To control this behaviour, the ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment
variable can be used:
.. list-table:: ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
:header-rows: 1
* - ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
- Behaviour
* - ``"pip"``
- Mach will ``pip install`` all needed dependencies from PyPI at runtime into a Python
virtual environment that's reused in future Mach invocations.
* - ``"none"``
- Mach will perform the build using only vendored packages. No Python virtual environment
will be created for Mach.
* - ``"system"``
- Mach will use the host system's Python packages as part of doing the build. This option
allows the usage of native Python packages without leaning on a ``pip install`` at
build-time. This is generally slower because the system Python packages have to
be asserted to be compatible with Mach. Additionally, dependency lockfiles are ignored,
so there's higher risk of breakage. Finally, as with ``"none"``, no Python virtualenv
environment is created for Mach.
* - ``<unset>``
- Same behaviour as ``"pip"`` if ``MOZ_AUTOMATION`` isn't set. Otherwise, uses
the same behaviour as ``"none"``.
There's a couple restrictions here:
* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` only applies to the top-level ``"mach"`` site,
the ``"common"`` site and the ``"build"`` site. All other sites will use ``pip install`` at
run-time as needed.
* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"`` is not allowed when using any site other
than ``"mach"``, ``"common"`` or ``"build"``, because:
* As described in :ref:`package-compatibility` below, packages used by Mach are still
in scope when commands are run, and
* The host system is practically guaranteed to be incompatible with commands' dependency
lockfiles.
The ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment variable fits into the following use
cases:
Mozilla CI Builds
~~~~~~~~~~~~~~~~~
We need access to the native packages of ``zstandard`` and ``psutil`` to extract archives and
get OS information respectively. Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"``.
Mozilla CI non-Build Tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~
We generally don't want to create a Mach virtual environment to avoid redundant processing,
but it's ok to ``pip install`` for specific command sites as needed, so leave
``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset (``MOZ_AUTOMATION`` implies the default
behaviour of ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``).
In cases where native packages *are* needed by Mach, use
``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="pip"``.
Downstream CI Builds
~~~~~~~~~~~~~~~~~~~~
Sometimes these builds happen in sandboxed, network-less environments, and usually these builds
don't need any of the behaviour enabled by installing native Python dependencies.
Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
Gentoo Builds
~~~~~~~~~~~~~
When installing Firefox via the package manager, Gentoo generally builds it from source rather than
distributing a compiled binary artifact. Accordingly, users doing a build of Firefox in this
context don't want stray files created in ``~/.mozbuild`` or unnecessary ``pip install`` calls.
Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
Firefox Developers
~~~~~~~~~~~~~~~~~~
Leave ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset so that all Mach commands can be run,
Python dependency lockfiles are respected, and optional behaviour is enabled by installing
native packages.
.. _package-compatibility:
Package compatibility
=====================
Mach requires that all commands' package requirements be compatible with those of Mach itself.
(This is because functions and state created by Mach are still usable from within the commands, and
they may still need access to their associated 3rd-party modules).
However, it is OK for Mach commands to have package requirements which are incompatible with each
other. This allows the flexibility for some Mach commands to depend on modern dependencies while
other, more mature commands may still only be compatible with a much older version.
.. note::
Only one version of a package may be vendored at any given time. If two Mach commands need to
have conflicting packages, then at least one of them must ``pip install`` the package instead
of vendoring.
If a Mach command's dependency conflicts with a vendored package, and that vendored package
isn't needed by Mach itself, then that vendored dependency should be moved from
``python/sites/mach.txt`` to its associated environment.
|
