summaryrefslogtreecommitdiffstats
path: root/taskcluster/docs/attributes.rst
blob: 31ddecf3a29b37430e73ac431938636613f00fc6 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
===============
Task Attributes
===============

Tasks can be filtered, for example to support "try" pushes which only perform a
subset of the task graph or to link dependent tasks.  This filtering is the
difference between a full task graph and a target task graph.

Filtering takes place on the basis of attributes.  Each task has a dictionary
of attributes and filters over those attributes can be expressed in Python.  A
task may not have a value for every attribute.

The attributes, and acceptable values, are defined here.  In general, attribute
names and values are the short, lower-case form, with underscores.

kind
====

A task's ``kind`` attribute gives the name of the kind that generated it, e.g.,
``build`` or ``spidermonkey``.

run_on_projects
===============

The projects where this task should be in the target task set.  This is how
requirements like "only run this on autoland" get implemented.

.. note::

    Please use this configuration. Running a job for all projects can quickly add up
    in term of cost while not providing any value for some projects.

`run-on-projects` can use either aliases or project names.

These are the aliases:

 * `integration` -- integration repository (autoland)
 * `trunk` -- integration repository plus mozilla-central
 * `release` -- release repositories (beta, release, esr) including mozilla-central
 * `all` -- everywhere (the default)

Project names are the repositories.  They can be:

* `autoland`
* `mozilla-central`
* `mozilla-beta`
* `mozilla-release`
* `mozilla-esr115`
* ... A partial list can be found in taskcluster/gecko_taskgraph/util/attributes.py

For try, this attribute applies only if ``-p all`` is specified.  All jobs can
be specified by name regardless of ``run_on_projects``.

If ``run_on_projects`` is set to an empty list, then the task will not run
anywhere, unless its build platform is specified explicitly in try syntax.


.. note::

    As `try` pushes don't use filter_for_projects by design, there isn't a way
    to define that a task will run on `try`.


.. note::

    A given task `[taskA]` may not respect `run-on-projects` if there another task
    `[taskB]` which is scheduled to run (such as via run-on-projects) which depends it
    `[taskA]`. Because by nature of `TaskB` running we must run `TaskA`.

    See `bug 1640603 <https://bugzilla.mozilla.org/show_bug.cgi?id=1640603#c5>`_ as example.

run_on_hg_branches
==================

On a given project, the mercurial branch where this task should be in the target
task set.  This is how requirements like "only run this RELBRANCH" get implemented.
These are either the regular expression of a branch (e.g.: ``GECKOVIEW_\d+_RELBRANCH``)
or the following alias:

 * `all` -- everywhere (the default)

Like ``run_on_projects``, the same behavior applies if it is set to an empty list.

task_duplicates
===============

This is used to indicate that we want multiple copies of the task created.
This feature is used to track down intermittent job failures.

If this value is set to N, the task-creation machinery will create a total of N
copies of the task.  Only the first copy will be included in the taskgraph
output artifacts, although all tasks will be contained in the same taskGroup.

While most attributes are considered read-only, target task methods may alter
this attribute of tasks they include in the target set.

build_platform
==============

The build platform defines the platform for which the binary was built.  It is
set for both build and test jobs, although test jobs may have a different
``test_platform``.

build_type
==========

The type of build being performed.  This is a subdivision of ``build_platform``,
used for different kinds of builds that target the same platform.  Values are

 * ``debug``
 * ``opt``

test_platform
=============

The test platform defines the platform on which tests are run.  It is only
defined for test jobs and may differ from ``build_platform`` when the same binary
is tested on several platforms (for example, on several versions of Windows).
This applies for both talos and unit tests.

Unlike build_platform, the test platform is represented in a slash-separated
format, e.g., ``linux64/opt``.

unittest_suite
==============

This is the unit test suite being run in a unit test task.  For example,
``mochitest`` or ``cppunittest``.

unittest_category
=================

This is the high-level category of test the suite corresponds to. This is
usually the test harness used to run the suite.

unittest_try_name
=================

This is the name used to refer to a unit test via try syntax.  It
may not match ``unittest_suite``.

unittest_variant
================

The configuration variant the test suite is running with. If set, this usually
means the tests are running with a special pref enabled. These are defined in
``taskgraph.transforms.test.TEST_VARIANTS``.

talos_try_name
==============

This is the name used to refer to a talos job via try syntax.

raptor_try_name
===============

This is the name used to refer to a raptor job via try syntax.

job_try_name
============

This is the name used to refer to a "job" via try syntax (``-j``).  Note that for
some kinds, ``-j`` also matches against ``build_platform``.

test_chunk
==========

This is the chunk number of a chunked test suite. Note that this is a string!

test_manifests
==============

A list of the test manifests that run in this task.

image_name
==========

For the ``docker_image`` kind, this attribute contains the docker image name.

nightly
=======

Signals whether the task is part of a nightly graph. Useful when filtering
out nightly tasks from full task set at target stage.

shippable
=========
Signals whether the task is considered "shippable", that it should get signed and is ok to
be used for nightlies or releases.

all_locales
===========

For the ``l10n`` and ``shippable-l10n`` kinds, this attribute contains the list
of relevant locales for the platform.

all_locales_with_changesets
===========================

Contains a dict of l10n changesets, mapped by locales (same as in ``all_locales``).

l10n_chunk
==========
For the ``l10n`` and ``shippable-l10n`` kinds, this attribute contains the chunk
number of the job. Note that this is a string!

chunk_locales
=============
For the ``l10n`` and ``shippable-l10n`` kinds, this attribute contains an array of
the individual locales this chunk is responsible for processing.

locale
======
For jobs that operate on only one locale, we set the attribute ``locale`` to the
specific locale involved. Currently this is only in l10n versions of the
``beetmover`` and ``balrog`` kinds.

signed
======
Signals that the output of this task contains signed artifacts.

stub-installer
==============
Signals to the build system that this build is expected to have a stub installer
present, and informs followon tasks to expect it.

repackage_type
==============
This is the type of repackage. Can be ``repackage`` or
``repackage_signing``.

fetch-artifact
==============

For fetch jobs, this is the path to the artifact for that fetch operation.

fetch-alias
===========
An alias that can be used instead of the real fetch job name in fetch
stanzas for jobs.

toolchain-artifact
==================
For toolchain jobs, this is the path to the artifact for that toolchain.

toolchain-alias
===============
An alias that can be used instead of the real toolchain job name in fetch
stanzas for jobs.

toolchain-env
=============
Extra environment variables that will be set on the worker when fetching this
toolchain.

toolchain-command
=================
An extra attribute used to communicate to the build system bootstrap code the
command used to build the toolchain. This is useful because the commands are
formatted differently depending on the worker type, sometimes inconveniently
to parse afterwards.

always_target
=============

Tasks with this attribute will be included in the ``target_task_graph`` if
``parameters["tasks_for"]`` is ``hg-push``, regardless of any target task
filtering that occurs. When a task is included in this manner (i.e it otherwise
would have been filtered out), it will be considered for optimization even if
the ``optimize_target_tasks`` parameter is False.

This is meant to be used for tasks which a developer would almost always want to
run. Typically these tasks will be short running and have a high risk of causing
a backout. For example ``lint`` or ``python-unittest`` tasks.

shipping_product
================
For release promotion jobs, this is the product we are shipping.

shipping_phase
==============
For release promotion jobs, this is the shipping phase (build, promote, push, ship).
During the build phase, we build and sign shippable builds. During the promote phase,
we generate l10n repacks and push to the candidates directory. During the push phase,
we push to the releases directory. During the ship phase, we update bouncer, push to
Google Play, version bump, mark as shipped in ship-it.

Using the "snowman model", we depend on previous graphs if they're defined. So if we
ask for a ``push`` (the head of the snowman) and point at the body and base, we only
build the head. If we don't point at the body and base, we build the whole snowman
(build, promote, push).

artifact_prefix
===============
Most taskcluster artifacts are public, so we've hardcoded ``public/build`` in a
lot of places. To support private artifacts, we've moved this to the
``artifact_prefix`` attribute. It will default to ``public/build`` but will be
overridable per-task.

artifact_map
===============
For beetmover jobs, this indicates which yaml file should be used to
generate the upstream artifacts and payload instructions to the task.

release_artifacts
=================
A set of artifacts that are candidates for downstream release tasks to operate
on.

batch
=====
Used by `perftest` to indicates that a task can be run as a batch.


enable-full-crashsymbols
========================
In automation, full crashsymbol package generation is normally disabled.  For
build kinds where the full crashsymbols should be enabled, set this attribute
to True. The full symbol packages will then be generated and uploaded on
release branches and on try.

skip-upload-crashsymbols
========================
Shippable/nightly builds are normally required to set enable-full-crashsymbols,
but in some limited corner cases (universal builds), that is not wanted, because
the symbols are uploaded independently already.

upload-generated-sources
========================
generated-sources are normally only uploaded to S3 for shippable/nightly
builds.  This attributes turns that on for other builds such as macosx
before unification.

cron
====
Indicates that a task is meant to be run via cron tasks, and should not be run
on push.

cached_task
===========
Some tasks generate artifacts that are cached between pushes. This is a
dictionary with the type and name of the cache, and the unique string used to
identify the current version of the artifacts. See :py:mod:`taskgraph.util.cached_task`.

.. code:: yaml

   cached_task:
       digest: 66dfc2204600b48d92a049b6a18b83972bb9a92f9504c06608a9c20eb4c9d8ae
       name: debian7-base
       type: docker-images.v2

eager_indexes
=============
A list of strings of indexes to populate before the task ever completes. Some tasks (e.g. cached tasks) we
want to exist in the index before they even run/complete. Our current use is to allow us to depend on an
unfinished cached task in future pushes. This avoids extra overhead from multiple tasks running, and
can allow us to have our results in just a bit earlier.

required_signoffs
=================
A list of release signoffs that this kind requires, should the release also
require these signoffs. For example, ``mar-signing`` signoffs may be required
by some releases in the future; for any releases that require ``mar-signing``
signoffs, the kinds that also require that signoff are marked with this
attribute.

update-channel
==============
The update channel the build is configured to use.

mar-channel-id
==============
The mar-channel-id the build is configured to use.

accepted-mar-channel-ids
========================
The mar-channel-ids this build will accept updates to. It should usually be the same as
the value mar_channel_id.  If more than one ID is needed, then you should use a
comma separated list of values.

openh264_rev
============
Only used for openh264 plugin builds, used to signify the revision (and thus inform artifact name) of the given build.

code-review
===========
If a task set this boolean attribute to `true`, it will be processed by the code
review bot, the task will ran for every new Phabricator diff.
Any supported and detected issue will be automatically reported on the
Phabricator revision.

resource-monitor
================
If a task set this boolean attribute to `true`, it will collect CPU, memory, and
- if available - Disk and Network IO by running the resource-monitor utility,
provided through fetches.

retrigger
=========
Whether the task can be retriggered, or if it needs to be re-run.

disable-push-apk
================
Some GeckoView-only Android tasks produce APKs that shouldn't be
pushed to the Google Play Store.  Set this to ``true`` to disable
pushing.

disable-build-signing
=====================
Some GeckoView-only tasks produce APKs, but not APKs that should be
signed.  Set this to ``true`` to disable APK signing.

enable-build-signing
====================
We enable build-signing for ``shippable``, ``nightly``, and ``enable-build-signing`` tasks.

run-visual-metrics
==================
If set to true, will run the visual metrics task on the provided
video files.

skip-verify-test-packaging
==========================
If set to true, this task will not be checked to see that
MOZ_AUTOMATION_PACKAGE_TESTS is set correctly based on whether or not the task
has dependent tests. This should only be used in very unique situations, such
as Windows AArch64 builds that copy test packages between build tasks.

geckodriver
===========
If non-empty, declares that the (toolchain) task is a `geckodriver`
task that produces a binary that should be signed.

rebuild-on-release
==================
If true, the digest for this task will also depend on if the branch is a
release branch.  This will cause tasks like toolchains to be rebuilt as they
move from e.g. autoland to mozilla-central.

local-toolchain
===============
This toolchain is used for local development, so should be built on trunk, even
if it does not have any in-graph consumers.

artifact-build
==============

This build is an artifact build.

This deliberately excludes builds that are implemented using the artifact build
machinery, but are not primarily intended to short-circuit build time. In
particular the Windows aarch64 builds are not marked this way.

maven_packages
===============
List of maven packages produced by the build.

supports-artifact-builds
========================

If false, the task requires a compiled build and will not work with artifact builds.

primary-kind-dependency
=======================

For tasks that were derived from a group of dependencies, this attribute
denotes which dependent kind is the `primary one`_.

Typically this is set by the ``taskgraph.transforms.from_deps`` transforms.

snap_test_type
==============

For Snap tests tasks, used to disambiguate task label

.. _primary one: https://taskcluster-taskgraph.readthedocs.io/en/latest/reference/transforms/from_deps.html#primary-kind