summaryrefslogtreecommitdiffstats
path: root/taskcluster/docs/transforms
diff options
context:
space:
mode:
Diffstat (limited to 'taskcluster/docs/transforms')
-rw-r--r--taskcluster/docs/transforms/index.rst22
-rw-r--r--taskcluster/docs/transforms/job.rst42
-rw-r--r--taskcluster/docs/transforms/signing.rst15
-rw-r--r--taskcluster/docs/transforms/task.rst45
-rw-r--r--taskcluster/docs/transforms/test.rst15
5 files changed, 139 insertions, 0 deletions
diff --git a/taskcluster/docs/transforms/index.rst b/taskcluster/docs/transforms/index.rst
new file mode 100644
index 0000000000..641f1abbc4
--- /dev/null
+++ b/taskcluster/docs/transforms/index.rst
@@ -0,0 +1,22 @@
+Transforms
+==========
+
+`Transforms`_ are Python functions that can modify (or sometimes add and
+remove) tasks from the graph. This section describes some of the transform
+functions that are available in ``mozilla-central``.
+
+.. toctree::
+
+ signing
+ test
+ job
+ task
+
+Other Transforms
+----------------
+
+There are tons of transforms that are not documented here. Consult the source
+files under `taskcluster/gecko_taskgraph/transforms`_ which provide lots of
+additional detail, both in the code itself and in comments and docstrings.
+
+.. _taskcluster/gecko_taskgraph/transforms: https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/transforms
diff --git a/taskcluster/docs/transforms/job.rst b/taskcluster/docs/transforms/job.rst
new file mode 100644
index 0000000000..160402dd36
--- /dev/null
+++ b/taskcluster/docs/transforms/job.rst
@@ -0,0 +1,42 @@
+Job Transforms
+==============
+
+.. note::
+
+ These transforms are currently duplicated by standalone Taskgraph
+ and will likely be refactored / removed at a later date.
+
+A job description says what to run in the task. It is a combination of a
+``run`` section and all of the fields from a task description. The run section
+has a ``using`` property that defines how this task should be run; for example,
+``mozharness`` to run a mozharness script, or ``mach`` to run a mach command.
+The remainder of the run section is specific to the run-using implementation.
+
+The effect of a job description is to say "run this thing on this worker". The
+job description must contain enough information about the worker to identify
+the workerType and the implementation (docker-worker, generic-worker, etc.).
+Alternatively, job descriptions can specify the ``platforms`` field in
+conjunction with the ``by-platform`` key to specify multiple workerTypes and
+implementations. Any other task-description information is passed along
+verbatim, although it is augmented by the run-using implementation.
+
+The run-using implementations are all located in
+``taskcluster/gecko_taskgraph/transforms/job``, along with the schemas for their
+implementations. Those well-commented source files are the canonical
+documentation for what constitutes a job description, and should be considered
+part of the documentation.
+
+following ``run-using`` are available
+
+ * ``hazard``
+ * ``mach``
+ * ``mozharness``
+ * ``mozharness-test``
+ * ``run-task``
+ * ``spidermonkey`` or ``spidermonkey-package``
+ * ``debian-package``
+ * ``ubuntu-package``
+ * ``toolchain-script``
+ * ``always-optimized``
+ * ``fetch-url``
+ * ``python-test``
diff --git a/taskcluster/docs/transforms/signing.rst b/taskcluster/docs/transforms/signing.rst
new file mode 100644
index 0000000000..52b13765bf
--- /dev/null
+++ b/taskcluster/docs/transforms/signing.rst
@@ -0,0 +1,15 @@
+Signing Transforms
+==================
+
+Signing kinds are passed a single dependent job (from its kind dependency) to act
+on.
+
+The transforms in ``taskcluster/gecko_taskgraph/transforms/signing.py`` implement
+this common functionality. They expect a "signing description", and produce a
+task definition. The schema for a signing description is defined at the top of
+``signing.py``, with copious comments.
+
+In particular you define a set of upstream artifact urls (that point at the
+dependent task) and can optionally provide a dependent name (defaults to build)
+for use in ``task-reference``/``artifact-reference``. You also need to provide
+the signing formats to use.
diff --git a/taskcluster/docs/transforms/task.rst b/taskcluster/docs/transforms/task.rst
new file mode 100644
index 0000000000..f19b4a7e15
--- /dev/null
+++ b/taskcluster/docs/transforms/task.rst
@@ -0,0 +1,45 @@
+Task Transforms
+===============
+
+.. note::
+
+ These transforms are currently duplicated by standalone Taskgraph
+ and will likely be refactored / removed at a later date.
+
+Every kind needs to create tasks, and all of those tasks have some things in
+common. They all run on one of a small set of worker implementations, each
+with their own idiosyncrasies. And they all report to TreeHerder in a similar
+way.
+
+The transforms in ``taskcluster/gecko_taskgraph/transforms/task.py`` implement
+this common functionality. They expect a "task description", and produce a
+task definition. The schema for a task description is defined at the top of
+``task.py``, with copious comments. Go forth and read it now!
+
+In general, the task-description transforms handle functionality that is common
+to all Gecko tasks. While the schema is the definitive reference, the
+functionality includes:
+
+* TreeHerder metadata
+
+* Build index routes
+
+* Information about the projects on which this task should run
+
+* Optimizations
+
+* Defaults for ``expires-after`` and and ``deadline-after``, based on project
+
+* Worker configuration
+
+The parts of the task description that are specific to a worker implementation
+are isolated in a ``task_description['worker']`` object which has an
+``implementation`` property naming the worker implementation. Each worker
+implementation has its own section of the schema describing the fields it
+expects. Thus the transforms that produce a task description must be aware of
+the worker implementation to be used, but need not be aware of the details of
+its payload format.
+
+The ``task.py`` file also contains a dictionary mapping treeherder groups to
+group names using an internal list of group names. Feel free to add additional
+groups to this list as necessary.
diff --git a/taskcluster/docs/transforms/test.rst b/taskcluster/docs/transforms/test.rst
new file mode 100644
index 0000000000..901aaa7cc1
--- /dev/null
+++ b/taskcluster/docs/transforms/test.rst
@@ -0,0 +1,15 @@
+Test Transforms
+===============
+
+Test descriptions specify how to run a unittest or talos run. They aim to
+describe this abstractly, although in many cases the unique nature of
+invocation on different platforms leaves a lot of specific behavior in the test
+description, divided by ``by-test-platform``.
+
+Test descriptions are validated to conform to the schema in
+``taskcluster/gecko_taskgraph/transforms/test/__init__.py``. This schema is
+extensively documented and is a the primary reference for anyone modifying
+tests.
+
+The output of ``tests.py`` is a task description. Test dependencies are
+produced in the form of a dictionary mapping dependency name to task label.