diff options
Diffstat (limited to 'taskcluster/docs/transforms')
| -rw-r--r-- | taskcluster/docs/transforms/index.rst | 22 | ||||
| -rw-r--r-- | taskcluster/docs/transforms/job.rst | 42 | ||||
| -rw-r--r-- | taskcluster/docs/transforms/signing.rst | 15 | ||||
| -rw-r--r-- | taskcluster/docs/transforms/task.rst | 45 | ||||
| -rw-r--r-- | taskcluster/docs/transforms/test.rst | 15 |
5 files changed, 139 insertions, 0 deletions
diff --git a/taskcluster/docs/transforms/index.rst b/taskcluster/docs/transforms/index.rst new file mode 100644 index 0000000000..641f1abbc4 --- /dev/null +++ b/taskcluster/docs/transforms/index.rst @@ -0,0 +1,22 @@ +Transforms +========== + +`Transforms`_ are Python functions that can modify (or sometimes add and +remove) tasks from the graph. This section describes some of the transform +functions that are available in ``mozilla-central``. + +.. toctree:: + + signing + test + job + task + +Other Transforms +---------------- + +There are tons of transforms that are not documented here. Consult the source +files under `taskcluster/gecko_taskgraph/transforms`_ which provide lots of +additional detail, both in the code itself and in comments and docstrings. + +.. _taskcluster/gecko_taskgraph/transforms: https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/transforms diff --git a/taskcluster/docs/transforms/job.rst b/taskcluster/docs/transforms/job.rst new file mode 100644 index 0000000000..160402dd36 --- /dev/null +++ b/taskcluster/docs/transforms/job.rst @@ -0,0 +1,42 @@ +Job Transforms +============== + +.. note:: + + These transforms are currently duplicated by standalone Taskgraph + and will likely be refactored / removed at a later date. + +A job description says what to run in the task. It is a combination of a +``run`` section and all of the fields from a task description. The run section +has a ``using`` property that defines how this task should be run; for example, +``mozharness`` to run a mozharness script, or ``mach`` to run a mach command. +The remainder of the run section is specific to the run-using implementation. + +The effect of a job description is to say "run this thing on this worker". The +job description must contain enough information about the worker to identify +the workerType and the implementation (docker-worker, generic-worker, etc.). +Alternatively, job descriptions can specify the ``platforms`` field in +conjunction with the ``by-platform`` key to specify multiple workerTypes and +implementations. Any other task-description information is passed along +verbatim, although it is augmented by the run-using implementation. + +The run-using implementations are all located in +``taskcluster/gecko_taskgraph/transforms/job``, along with the schemas for their +implementations. Those well-commented source files are the canonical +documentation for what constitutes a job description, and should be considered +part of the documentation. + +following ``run-using`` are available + + * ``hazard`` + * ``mach`` + * ``mozharness`` + * ``mozharness-test`` + * ``run-task`` + * ``spidermonkey`` or ``spidermonkey-package`` + * ``debian-package`` + * ``ubuntu-package`` + * ``toolchain-script`` + * ``always-optimized`` + * ``fetch-url`` + * ``python-test`` diff --git a/taskcluster/docs/transforms/signing.rst b/taskcluster/docs/transforms/signing.rst new file mode 100644 index 0000000000..52b13765bf --- /dev/null +++ b/taskcluster/docs/transforms/signing.rst @@ -0,0 +1,15 @@ +Signing Transforms +================== + +Signing kinds are passed a single dependent job (from its kind dependency) to act +on. + +The transforms in ``taskcluster/gecko_taskgraph/transforms/signing.py`` implement +this common functionality. They expect a "signing description", and produce a +task definition. The schema for a signing description is defined at the top of +``signing.py``, with copious comments. + +In particular you define a set of upstream artifact urls (that point at the +dependent task) and can optionally provide a dependent name (defaults to build) +for use in ``task-reference``/``artifact-reference``. You also need to provide +the signing formats to use. diff --git a/taskcluster/docs/transforms/task.rst b/taskcluster/docs/transforms/task.rst new file mode 100644 index 0000000000..f19b4a7e15 --- /dev/null +++ b/taskcluster/docs/transforms/task.rst @@ -0,0 +1,45 @@ +Task Transforms +=============== + +.. note:: + + These transforms are currently duplicated by standalone Taskgraph + and will likely be refactored / removed at a later date. + +Every kind needs to create tasks, and all of those tasks have some things in +common. They all run on one of a small set of worker implementations, each +with their own idiosyncrasies. And they all report to TreeHerder in a similar +way. + +The transforms in ``taskcluster/gecko_taskgraph/transforms/task.py`` implement +this common functionality. They expect a "task description", and produce a +task definition. The schema for a task description is defined at the top of +``task.py``, with copious comments. Go forth and read it now! + +In general, the task-description transforms handle functionality that is common +to all Gecko tasks. While the schema is the definitive reference, the +functionality includes: + +* TreeHerder metadata + +* Build index routes + +* Information about the projects on which this task should run + +* Optimizations + +* Defaults for ``expires-after`` and and ``deadline-after``, based on project + +* Worker configuration + +The parts of the task description that are specific to a worker implementation +are isolated in a ``task_description['worker']`` object which has an +``implementation`` property naming the worker implementation. Each worker +implementation has its own section of the schema describing the fields it +expects. Thus the transforms that produce a task description must be aware of +the worker implementation to be used, but need not be aware of the details of +its payload format. + +The ``task.py`` file also contains a dictionary mapping treeherder groups to +group names using an internal list of group names. Feel free to add additional +groups to this list as necessary. diff --git a/taskcluster/docs/transforms/test.rst b/taskcluster/docs/transforms/test.rst new file mode 100644 index 0000000000..901aaa7cc1 --- /dev/null +++ b/taskcluster/docs/transforms/test.rst @@ -0,0 +1,15 @@ +Test Transforms +=============== + +Test descriptions specify how to run a unittest or talos run. They aim to +describe this abstractly, although in many cases the unique nature of +invocation on different platforms leaves a lot of specific behavior in the test +description, divided by ``by-test-platform``. + +Test descriptions are validated to conform to the schema in +``taskcluster/gecko_taskgraph/transforms/test/__init__.py``. This schema is +extensively documented and is a the primary reference for anyone modifying +tests. + +The output of ``tests.py`` is a task description. Test dependencies are +produced in the form of a dictionary mapping dependency name to task label. |