1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
Taskgraph Overview
==================
Taskgraph builds a directed acyclic graph, where each node in the graph
represents a task, and each edge represents the dependency relationships
between them.
See Taskgraph's `graph generation documentation`_ for more details.
Decision Task
-------------
The decision task is the first task created when a new graph begins. It is
responsible for creating the rest of the task graph.
The decision task for pushes is defined in-tree, in ``.taskcluster.yml``. That
task description invokes ``mach taskcluster decision`` with some metadata about
the push. That mach command determines the required graph of tasks to submit,
then calls the Taskcluster API to create the tasks.
.. note::
``mach taskgraph decision`` is *not* meant to be invoked by humans.
Instead, follow `this guide`_ (prepending ``mach`` to all commands) to
invoke Taskgraph locally without submitting anything to Taskcluster.
.. _kinds:
Kinds
-----
Generation starts with "kinds". These are yaml files that denote groupings of
tasks that are loosely related to one another. For example, in Gecko there are
``build`` and ``test`` kinds. Each kind has its own directory under
`taskcluster/ci`_ which contains a ``kind.yml`` file.
For more information on kinds, see Taskgraph's `kind documentation`_. For a
list of available kinds in ``mozilla-central``, see the :doc:`kinds reference
</taskcluster/kinds>`.
Loader
------
Next, a "loader" is responsible for parsing each ``kind.yml`` file and turning
it into an initial set of tasks. Loaders will always parse kinds in an ordering
satisfying the ``kind-dependencies`` key.
See Taskgraph's `loader documentation`_ for more details.
.. _transforms:
Transforms
----------
After the initial set of tasks are loaded, transformations are applied to each
task. Transforms are Python functions that take a generator of tasks as input,
and yields a generator of tasks as output. Possibly modifying, adding or removing
tasks along the way.
See Taskgrpah's `transforms documentation`_ for more details on transforms, or
the :doc:`transforms section </taskcluster/transforms/index>` for information
on some of the transforms available in ``mozilla-central``.
Target Tasks
------------
After transformation is finished, the `target_tasks`_ module filters out any
tasks that aren't applicable to the current :doc:`parameter set
</taskcluster/parameters>`.
Optimization
------------
After the "target graph" is generated, an optimization process looks to remove
or replace unnecessary tasks in the graph. For instance, a task may be removed
if the push doesn't modify any of the files that could affect it.
See Taskgraph's `optimization documentation`_ for more details, or the
:doc:`optimization strategies <optimization/index>` available in
``mozilla-central``.
Task Parameterization
---------------------
A few components of tasks are only known at the very end of the decision task
-- just before the ``queue.createTask`` call is made. These are specified
using simple parameterized values, as follows:
``{"relative-datestamp": "certain number of seconds/hours/days/years"}``
Objects of this form will be replaced with an offset from the current time
just before the ``queue.createTask`` call is made. For example, an
artifact expiration might be specified as ``{"relative-datestamp": "1
year"}``.
``{"task-reference": "string containing <dep-name>"}``
The task definition may contain "task references" of this form. These will
be replaced during the optimization step, with the appropriate taskId for
the named dependency substituted for ``<dep-name>`` in the string.
Additionally, `decision` and `self` can be used a dependency names to refer
to the decision task, and the task itself. Multiple labels may be
substituted in a single string, and ``<<>`` can be used to escape a literal
``<``.
``{"artifact-reference": "..<dep-name/artifact/name>.."}``
Similar to a ``task-reference``, but this substitutes a URL to the queue's
``getLatestArtifact`` API method (for which a GET will redirect to the
artifact itself).
.. _taskgraph-graph-config:
Graph Configuration
-------------------
There are several configuration settings that are pertain to the entire
taskgraph. These are specified in :file:`config.yml` at the root of the
taskgraph configuration (typically :file:`taskcluster/ci/`). The available
settings are documented inline in `taskcluster/gecko_taskgraph/config.py
<https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/config.py>`_.
.. _taskgraph-trust-domain:
Action Tasks
------------
Action Tasks are tasks which perform an action based on a manual trigger (e.g
clicking a button in Treeherder). Actions are how it is possible to retrigger
or "Add New Jobs".
For more information, see Taskgraph's `actions documentation`_.
.. _graph generation documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/task-graphs.html
.. _this guide: https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/run-locally.html
.. _taskcluster/ci: https://searchfox.org/mozilla-central/source/taskcluster/ci
.. _kind documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/kind.html
.. _loader documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/loading.html
.. _transforms documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/transforms.html
.. _target_tasks:
.. _optimization documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/optimization.html
.. _actions documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/create-actions.html
|
