Adding upstream version 124.0.1.upstream/124.0.1

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 130 insertions, 0 deletions

diff --git a/testing/docs/tests-for-new-config/index.rst b/testing/docs/tests-for-new-config/index.rst
new file mode 100644
index 0000000000..32fdf62b7f
--- /dev/null
+++ b/testing/docs/tests-for-new-config/index.rst
@@ -0,0 +1,130 @@
+Turning on Firefox tests for a new configuration
+================================================
+
+You are ready to go with turning on Firefox tests for a new config.  Once you
+get to this stage, you will have seen a try push with all the tests running
+(many not green) to verify some tests pass and there are enough machines
+available to run tests.
+
+For the purpose of this document, assume you are tasked with upgrading Windows
+10 OS from version 1803 -> 1903. To simplify this we can call this `windows_1903`,
+and we need to:
+
+ * create meta bug
+ * push to try
+ * run skip-fails
+ * repeat 2 more times
+ * land changes and turn on tests
+ * turn on run only failures
+
+If you are running this manually or on configs/tests that are not supported with
+`./mach try --new-test-config`, then please follow the steps `here <manual.html>`__
+
+
+Create Meta Bug
+---------------
+
+This is a simple step where you create a meta bug to track the failures associated
+with the tests you are greening up.  If this is a test suite (i.e. `devtools`), it
+is ok to have a meta bug just for the test suite and the new platform.
+
+All bugs related to tests skipped or failing will be blocking this meta bug.
+
+Push to Try Server
+------------------
+
+Now that you have a configuration setup and machines available via try server, it
+is time to run try.  If you are migrating mochitest or xpcshell, then you can do:
+
+  ``./mach try fuzzy --no-artifact --full --rebuild 10 --new-task-config -q 'test-windows10-64-1903 mochitest-browser-chrome !ccov !ship !browsertime !talos !asan'``
+
+This will run many tests (thanks to --full and --rebuild 10), but will give plenty
+of useful data.
+
+In the scenario you are migrating tests such as:
+ * performance
+ * web-platform-tests
+ * reftest / crashtest / jsreftest
+ * mochitest-webgl (has a different process for test skipping)
+ * cppunittest / gtest / junit
+ * marionette / firefox-ui / telemetry
+
+ then please follow the steps `here <manual.html>`__
+
+ If you are migrating to a small machine pool, it is best to avoid `--rebuild 10` and
+ instead do `--rebuild 3`.  Likewise please limit your jobs to be the specific test
+ suite and variant.  The size of a worker pool is shown at the Workers page of the
+ Taskcluster instance.
+
+Run skip-fails
+--------------
+
+When the try push is completed it is time to run skip-fails.  Skip-fails will
+look at all the test results and automatically create a set of local changes
+with skip-if conditions to green up the tests faster.
+
+``./mach manifest skip-fails --b bugzilla.mozilla.org -m <meta_bug_id> --turbo "https://treeherder.mozilla.org/jobs?repo=try&revision=<rev>"``
+
+Please input the proper `meta_bug_id` and `rev` into the above command.
+
+The first time running this, you will need to get a `bugzilla api key <https://bugzilla.mozilla.org/userprefs.cgi?tab=apikey>`__.  copy
+this key and add it to your `~/.config/python-bugzilla/bugzilla-rc` file:
+
+.. code-block:: none
+
+  cat bugzillarc
+  [DEFAULT]
+  url = https://bugzilla.mozilla.org
+  [bugzilla.mozilla.org]
+  api_key = <key>
+
+When the command finishes, you will have new bugs created that are blocking the
+meta bug.  In addition you will have many changes to manifests adding skip-if
+conditions.  For tests than fail 40% of the time or for entire manifests that
+take >20 minutes to run on opt or >40 minutes on debug.
+
+You will need to create a commit (or `--amend` your previous commit if this is round 2 or 3):
+
+``hg commit -m "Bug <meta_bug_id> - Green up tests for <suite> on <platform>"``
+
+
+Repeat 2 More Times
+-------------------
+
+In 3 rounds this should be complete and ready to submit for review and turn on
+the new tests.
+
+There will be additional failures, those will follow the normal process of
+intermittents.
+
+
+Land Changes and Turn on Tests
+------------------------------
+
+After you have a green test run, it is time to land the patches.  There could
+be changes needed to the taskgraph in order to add the new hardware type and
+duplicate tests to run on both the old and the new, or create a new variant and
+denote which tests to run on that variant.
+
+Using our example of ``windows_1903``, this would be a new worker type that
+would require these edits:
+
+ * `transforms/tests.py <https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#97>`__ (duplicate windows 10 entries)
+ * `test-platforms.py <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#229>`__ (copy windows10 debug/opt/shippable/asan entries and make win10_1903)
+ * `test-sets.py <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#293>`__ (ideally you need nothing, otherwise copy ``windows-tests`` and edit the test list)
+
+In general this should allow you to have tests scheduled with no custom flags
+in try server and all of these will be scheduled by default on
+``mozilla-central``, ``autoland``, and ``release-branches``.
+
+Turn on Run Only Failures
+-------------------------
+
+Now that we have tests running regularly, the next step is to take all the
+disabled tests and run them in the special failures job.
+
+We have a basic framework created, but for every test harness (i.e. xpcshell,
+mochitest-gpu, browser-chrome, devtools, web-platform-tests, crashtest, etc.),
+there will need to be a corresponding tier-3 job that is created.
+
+TODO: point to examples of how to add this after we get our first jobs running.