diff options
Diffstat (limited to '')
| -rw-r--r-- | taskcluster/docs/kinds/test.rst | 161 |
1 files changed, 161 insertions, 0 deletions
diff --git a/taskcluster/docs/kinds/test.rst b/taskcluster/docs/kinds/test.rst new file mode 100644 index 0000000000..20d3635c5c --- /dev/null +++ b/taskcluster/docs/kinds/test.rst @@ -0,0 +1,161 @@ +Test Kind +========= + +The ``test`` kind defines both desktop and mobile tests for builds. Each YAML +file referenced in ``kind.yml`` defines the full set of tests for the +associated suite. + +The process of generating tests goes like this, based on a set of YAML files +named in ``kind.yml``: + + * For each build task, determine the related test platforms based on the build + platform. For example, a Windows 2010 build might be tested on Windows 7 + and Windows 10. Each test platform specifies "test sets" indicating which + tests to run. This is configured in the file named + ``test-platforms.yml``. + + * Each test set is expanded to a list of tests to run. This is configured in + the file named by ``test-sets.yml``. A platform may specify several test + sets, in which case the union of those sets is used. + + * Each named test is looked up in the file named by ``tests.yml`` to find a + test description. This test description indicates what the test does, how + it is reported to treeherder, and how to perform the test, all in a + platform-independent fashion. + + * Each test description is converted into one or more tasks. This is + performed by a sequence of transforms defined in the ``transforms`` key in + ``kind.yml``. See :ref:`transforms` for more information. + + * The resulting tasks become a part of the task graph. + +.. important:: + + This process generates *all* test jobs, regardless of tree or try syntax. + It is up to a later stages of the task-graph generation (the target set and + optimization) to select the tests that will actually be performed. + +Variants +-------- + +Sometimes we want to run the same tests under a different Firefox context, +usually this means with a pref set. The concept of ``variants`` was invented to +handle this use case. A variant is a stanza of configuration that can be merged +into each test definition. Variants are defined in the `variants.yml`_ file. +See this file for an up to date list of active variants and the pref(s) they +set. + +Each variant must conform to the +:py:data:`~gecko_taskgraph.transforms.test.variant_description_schema`: + +* **description** (required) - A description explaining what the variant is for. +* **component** (required) - The name of the component that owns the variant. It + should be formatted as ``PRODUCT``::``COMPONENT``. +* **expiration** (required) - The date when the variant will be expired (maximum + 6 months). +* **suffix** (required) - A suffix to apply to the task label and treeherder symbol. +* **when** - A `json-e`_ expression that must evaluate to ``true`` for the variant + to be applied. The ``task`` definition is passed in as context. +* **replace** - A dictionary that will overwrite keys in the task definition. +* **merge** - A dictionary that will be merged into the task definition using + the :py:func:`~gecko_taskgraph.util.templates.merge` function. + +.. note:: + + Exceptions can be requested to have a variant without expiration (using + "never") if this is a shipped mode we support. Teams should contact the CI + team to discuss this before submitting a patch if they think their variant + qualifies. All exceptions will require director approval. + + +Defining Variants +~~~~~~~~~~~~~~~~~ + +Variants can be defined in the test YAML files using the ``variants`` key. E.g: + +.. code-block:: yaml + + example-suite: + variants: + - foo + - bar + +This will split the task into three. The original task, the task with the +config from the variant named 'foo' merged in and the task with the config from +the variant named 'bar' merged in. + + +Composite Variants +~~~~~~~~~~~~~~~~~~ + +Sometimes we want to run tasks with multiple variants enabled at once. This can +be achieved with "composite variants". Composite variants are simply two or +more variant names joined with the ``+`` sign. Using the previous example, if +we wanted to run both the ``foo`` and ``bar`` variants together, we could do: + +.. code-block:: yaml + + example-suite: + variants: + - foo+bar + +This will first merge or replace the config of ``foo`` into the task, followed +by the config of ``bar``. Care should be taken if both variants are replacing +the same keys. The last variant's configuration will be the one that gets used. + + +Expired Variants +~~~~~~~~~~~~~~~~ +Ideally, when a variant is not needed anymore, it should be dropped (even if it +has not expired). If you need to extend the expiration date, you can submit a +patch to modify the expiration date in the `variants.yml`_ file. Variants will +not be scheduled to run after the expiration date. + +If an expired variant is not dropped, the triage owner of the component will be +notified. If the expired variant persists for an extended period, the autonag +bot will escalate to notify the manager and director of the triage owner. Once +at that point, we will submit a patch to remove the variant from Taskcluster and +manifest conditions pending the triage owner / manager to review. + +Please subscribe to alerts from `firefox-ci <https://groups.google.com/a/mozilla.com/g/firefox-ci>` +group in order to be aware of changes to the CI, scheduling, or the policy. + +.. _variants.yml: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/variants.yml +.. _json-e: https://json-e.js.org/ + + +Setting +------- + +A test ``setting`` is the set of conditions under which a test is running. +Aside from the chunk number, a ``setting`` uniquely distinguishes a task from +another that is running the same set of tests. There are three types of inputs +that make up a ``setting``: + +1. Platform - Bits of information that describe the underlying platform the + test is running on. This includes things like the operating system and + version, CPU architecture, etc. + +2. Build - Bits of information that describe the build being tested. This + includes things like the build type and which build attributes (like + ``asan``, ``ccov``, etc) are enabled. + +3. Runtime - Bits of information that describe the configured state of Firefox. + This includes things like prefs and environment variables. Note that tasks + should only set runtime configuration via the variants system (see + `Variants`_). + +Test ``settings`` are available in the ``task.extra.test-setting`` object in +all test tasks. They are defined by the +:py:func:`~gecko_taskgraph.transforms.test.set_test_setting` transform +function. + +The full schema is defined in the +:py:data:`~gecko_taskgraph.transforms.test.test_setting_description_schema`. + +Setting Hash +~~~~~~~~~~~~ + +In addition to the three top-level objects, there is also a ``_hash`` key which +contains a hash of the rest of the setting object. This is a convenient way for +consumers to group or compare tasks that run under the same setting. |