Adding upstream version 115.7.0esr.upstream/115.7.0esrupstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 375 insertions, 0 deletions

diff --git a/testing/web-platform/docs/index.rst b/testing/web-platform/docs/index.rst
new file mode 100644
index 0000000000..7f2d263113
--- /dev/null
+++ b/testing/web-platform/docs/index.rst
@@ -0,0 +1,375 @@
+web-platform-tests
+==================
+
+`web-platform-tests <http://web-platform-tests.org>`_ is a
+cross-browser testsuite. Writing tests as web-platform-tests helps
+ensure that browsers all implement the same behaviour for web-platform
+features.
+
+Upstream Documentation
+----------------------
+
+This documentation covers the integration of web-platform-tests into
+the Firefox tree. For documentation on writing tests, see
+`web-platform-tests.org <http://web-platform-tests.org>`_. In particular
+the following documents cover common test-writing needs:
+
+* `Javascript Tests (testharness.js)
+  <https://web-platform-tests.org/writing-tests/testharness.html>`_
+
+  - `testharness.js API
+    <https://web-platform-tests.org/writing-tests/testharness-api.html>`_
+
+  - `testdriver.js API
+    <https://web-platform-tests.org/writing-tests/testdriver.html>`_ -
+    features for writing tests that require special privileges
+    e.g. user-initiated input events.
+
+  - `Message Channels
+    <https://web-platform-tests.org/writing-tests/channels.html>`_ -
+    features for communicating between different globals (including
+    those in different browsing context groups / processes).
+
+* `Reftests <https://web-platform-tests.org/writing-tests/reftests.html>`_
+
+* `Crashtests <https://web-platform-tests.org/writing-tests/crashtest.html>`_
+
+* `Server features
+  <https://web-platform-tests.org/writing-tests/server-features.html>`_ -
+  e.g. multiple origins, substitutions, server-side Python scripts.
+
+Running Tests
+-------------
+
+Tests can be run using ``mach``::
+
+    mach wpt
+
+To run only certain tests, pass these as additional arguments to the
+command. For example to include all tests in the dom directory::
+
+    mach wpt testing/web-platform/tests/dom
+
+Tests may also be passed by id; this is the path plus any query or
+fragment part of a URL and is suitable for copying directly from logs
+e.g. on treeherder::
+
+    mach wpt /web-nfc/idlharness.https.window.html
+
+A single file can produce multiple tests, so passing test ids rather
+than paths is sometimes necessary to run exactly one test.
+
+The testsuite contains a mix of various test types including
+Javascript (``testharness``) tests, reftests and wdspec tests. To limit
+the type of tests that get run, use ``--test-type=<type>`` e.g.
+``--test-type=reftest`` for reftests.
+
+Note that if only a single testharness test is run the browser will
+stay open by default (matching the behaviour of mochitest). To prevent
+this pass ``--no-pause-after-test`` to ``mach wpt``.
+
+When the source tree is configured for building android, tests will
+also be run on Android, by default using a local emulator.
+
+Running Tests In Other Browsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+web-platform-tests is cross browser, and the runner is compatible with
+multiple browsers. Therefore it's possible to check the behaviour of
+tests in other browsers. By default Chrome, Edge and Servo are
+supported. In order to run the tests in these browsers use the
+``--product`` argument to wptrunner::
+
+    mach wpt --product chrome dom/historical.html
+
+By default these browsers run without expectation metadata, but it can
+be added in the ``testing/web-platform/products/<product>``
+directory. To run with the same metadata as for Firefox (so that
+differences are reported as unexpected results), pass ``--meta
+testing/web-platform/meta`` to the mach command.
+
+Results from the upstream CI for many browser, including Chrome and
+Safari, are available on `wpt.fyi <https://wpt.fyi>`_. There is also a
+`gecko dashboard <https://jgraham.github.io/wptdash/>`_ which by default
+shows tests that are failing in Gecko but not in Chrome and Safari,
+organised by bug component, based on the wpt.fyi data.
+
+Directories
+-----------
+
+Under ``testing/web-platform`` are the following directories:
+
+``tests/``
+  An automatically-updated import of the web-platform-tests
+  repository. Any changes to this directory are automatically
+  converted into pull requests against the upstream repository, and
+  merged if they pass CI.
+
+``meta/``
+  Gecko-specific metadata including expected test results
+  and configuration e.g. prefs that are set when running the
+  test. This is explained in the following section.
+
+``mozilla/tests``
+  Tests that will not be upstreamed and may
+  make use of Mozilla-specific features. They can access
+  the ``SpecialPowers`` APIs.
+
+``mozilla/meta``
+  Metadata for the Mozilla-specific tests.
+
+Metadata
+--------
+
+In order to separate out the shared testsuite from Firefox-specific
+metadata about the tests, all the metadata is stored in separate
+ini-like files in the ``meta/`` sub-directory.
+
+There is one metadata file per test file with associated
+gecko-specific data. The metadata file of a test has the same relative
+path as the test file and has the the suffix ``.ini`` e.g. for the
+test in ``testing/web-platform/tests/example/example.html``, the
+corresponding metadata file is
+``testing/web-platform/meta/example/example.html.ini``.
+
+The format of these files is similar to `ini` files, but with a couple
+of important differences; sections can be nested using indentation,
+and only `:` is permitted as a key-value separator. For example::
+
+    [filename.html]
+        [Subtest 1 name]
+            key: value
+
+        [Subtest 2 name]
+            key: [list, value]
+
+For cases where a single file generates multiple tests (e.g. variants
+or ``.any.js`` tests), the metadata file has one top-level section for
+each test, for example::
+
+    [test.any.html]
+        [Subtest name]
+            key: value
+
+    [test.any.worker.html]
+        [Subtest name]
+            key: other-value
+
+Values can be made conditional using a Python-like conditional syntax::
+
+    [filename.html]
+        key:
+            if os == "linux": linux-value
+            default-value
+
+The available variables for the conditions are those provided by
+`mozinfo
+<https://firefox-source-docs.mozilla.org/mozbase/mozinfo.html>`_, plus
+some additional `wpt-specific values
+<https://searchfox.org/mozilla-central/search?q=def%20run_info_extras&path=testing%2Fweb-platform%2Ftests%2Ftools%2Fwptrunner%2Fwptrunner%2Fbrowsers%2Ffirefox.py&case=false&regexp=false>`_.
+
+For more information on manifest files, see the `wptrunner
+documentation
+<https://web-platform-tests.org/tools/wptrunner/docs/expectation.html>`_
+
+
+Expectation Data
+~~~~~~~~~~~~~~~~
+
+All tests that don't pass in our CI have expectation data stored in
+the metadata file, under the key ``expected``. For example the
+expectation file for a test with one failing subtest and one erroring
+subtest might look like::
+
+    [filename.html]
+        [Subtest name for failing test]
+            expected: FAIL
+
+        [Subtest name for erroring test]
+            expected: ERROR
+
+Expectations can be made configuration-specific using the conditional syntax::
+
+    [filename.html]
+        expected:
+            if os == "linux" and bits == 32: TIMEOUT
+            if os == "win": ERROR
+            FAIL
+
+Tests that are intermittent may be marked with multiple statuses using
+a list of possibilities e.g. for a test that usually passes, but
+intermittently fails::
+
+    [filename.html]
+        [Subtest name for intermittent test]
+            expected: [PASS, FAIL]
+
+
+Auto-generating Expectation Data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After changing some code it may be necessary to update the expectation
+data for the relevant tests. This can of course be done manually, but
+tools are available to automate much of the process.
+
+First one must run the tests that have changed status, and save the
+raw log output to a file::
+
+    mach wpt /url/of/test.html --log-wptreport wptreport.json
+
+Then the ``wpt-update`` command may be run using this log data to update
+the expectation files::
+
+    mach wpt-update wptreport.json
+
+CI runs also produce ``wptreport.json`` files that can be downloaded
+as artifacts. When tests are run across multiple platforms, and all
+the wptreport files are processed together, the tooling will set the
+appropriate conditions for any platform-specific results::
+
+    mach wpt-update logs/*.json
+
+For complete runs the ``--full`` flag will cause metadata to be
+removed when a) the test was updated and b) there is a condition that
+didn't match any of the configuration in the input files.
+
+When tests are run more than once ``--update-intermittent`` flag will
+cause conflicting results to be marked as intermittents (otherwise the
+data is not updated in the case of conflicts).
+
+Disabling Tests
+~~~~~~~~~~~~~~~
+
+Tests are disabled using the same manifest files used to set
+expectation values. For example, if a test is unstable on Windows, it
+can be disabled using an ini file with the contents::
+
+    [filename.html]
+        disabled:
+            if os == "win": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567
+
+For intermittents it's generally preferable to give the test multiple
+expectations rather than disable it.
+
+Fuzzy Reftests
+~~~~~~~~~~~~~~
+
+Reftests where the test doesn't exactly match the reference can be
+marked as fuzzy. If the difference is inherent to the test, it should
+be encoded in a `meta element
+<https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching>`_,
+but where it's a Gecko-specific difference it can be added to the
+metadata file, using the same syntax::
+
+    [filename.html]
+        fuzzy: maxDifference=10-15;totalPixels=200-300
+
+In this case we expect between 200 and 300 pixels, inclusive, to be
+different, and the maximum difference in any RGB colour channel to be
+between 10 and 15.
+
+
+Enabling Prefs
+~~~~~~~~~~~~~~
+
+Some tests require specific prefs to be enabled before running. These
+prefs can be set in the expectation data using a ``prefs`` key with a
+comma-separated list of ``pref.name:value`` items::
+
+    [filename.html]
+        prefs: [dom.serviceWorkers.enabled:true,
+                dom.serviceWorkers.exemptFromPerDomainMax:true,
+                dom.caches.enabled:true]
+
+Disabling Leak Checks
+~~~~~~~~~~~~~~~~~~~~~
+
+When a test is imported that leaks, it may be necessary to temporarily
+disable leak checking for that test in order to allow the import to
+proceed. This works in basically the same way as disabling a test, but
+with the key ``leaks``::
+
+    [filename.html]
+        leaks:
+            if os == "linux": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567
+
+Per-Directory Metadata
+~~~~~~~~~~~~~~~~~~~~~~
+
+Occasionally it is useful to set metadata for an entire directory of
+tests e.g. to disable then all, or to enable prefs for every test. In
+that case it is possible to create a ``__dir__.ini`` file in the
+metadata directory corresponding to the tests for which you want to
+set this metadata e.g. to disable all the tests in
+``tests/feature/unsupported/``, one might create
+``meta/feature/unsupported/__dir__.ini`` with the contents::
+
+    disabled: Feature is unsupported
+
+Settings set in this way are inherited into sub-directories. It is
+possible to unset a value that has been set in a parent using the
+special token ``@Reset`` (usually used with prefs), or to force a value
+to true or false using ``@True`` and ``@False``.  For example to enable
+the tests in ``meta/feature/unsupported/subfeature-supported`` one might
+create an ini file
+``meta/feature/unsupported/subfeature-supported/__dir__.ini`` like::
+
+    disabled: @False
+
+Setting Metadata for Release Branches
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Run info properties can be used to set metadata for release branches
+that differs from nightly (e.g. for when a feature depends on prefs
+that are only set on nightly), for example::
+
+    [filename.html]
+      expected:
+        if release_or_beta: FAIL
+
+Note that in general the automatic metadata update will work better if
+the nonstandard configuration is used explicitly in the conditional,
+and placed at the top of the set of conditions, i.e. the following
+would cause problems later::
+
+    [filename.html]
+      expected:
+        if nightly_build: PASS
+        FAIL
+
+This is because on import the automatic metadata updates are run
+against the results of nightly builds, and we remove any existing
+conditions that match all the input runs to avoid building up stale
+configuration options.
+
+Test Manifest
+-------------
+
+web-platform-tests use a large auto-generated JSON file as their
+manifest. This stores data about the type of tests, their references,
+if any, and their timeout, gathered by inspecting the filenames and
+the contents of the test files. It is not necessary to manually add
+new tests to the manifest; it is automatically kept up to date when
+running `mach wpt`.
+
+Synchronization with Upstream
+-----------------------------
+
+Tests are automatically synchronized with upstream using the `wpt-sync
+bot <https://github.com/mozilla/wpt-sync>`_. This performs the following tasks:
+
+* Creates upstream PRs for changes in
+  ``testing/web-platform/tests`` once they land on autoland, and
+  automatically merges them after they reach mozilla-central.
+
+* Runs merged upstream PRs through gecko CI to generate updated
+  expectation metadata.
+
+* Updates the copy of web-platform-tests in the gecko tree with
+  changes from upstream, and the expectation metadata required to make
+  CI jobs pass.
+
+The nature of a two-way sync means that occasional merge conflicts and
+other problems. If something isn't in sync with upstream in the way
+you expect, please ask on `#interop
+<https://chat.mozilla.org/#/room/#interop:mozilla.org>`_ on matrix.