Adding upstream version 115.7.0esr.upstream/115.7.0esrupstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 338 insertions, 0 deletions

diff --git a/testing/docs/automated-testing/index.rst b/testing/docs/automated-testing/index.rst
new file mode 100644
index 0000000000..8fb9a022ef
--- /dev/null
+++ b/testing/docs/automated-testing/index.rst
@@ -0,0 +1,338 @@
+Automated Testing
+=================
+
+You've just written a feature and (hopefully!) want to test it. Or you've
+decided that an existing feature doesn't have enough tests and want to contribute
+some. But where do you start? You've looked around and found references to things
+like "xpcshell" or "web-platform-tests" or "talos". What code, features or
+platforms do they all test? Where do their feature sets overlap? In short, where
+should your new tests go? This document is a starting point for those who want
+to start to learn about Mozilla's automated testing tools and procedures. Below
+you'll find a short summary of each framework we use, and some questions to help
+you pick the framework(s) you need for your purposes.
+
+If you still have questions, ask on `Matrix <https://wiki.mozilla.org/Matrix>`__
+or on the relevant bug.
+
+Firefox Production
+------------------
+These tests are found within the `mozilla-central <https://hg.mozilla.org/mozilla-central>`__
+tree, along with the product code.
+
+They are run when a changeset is pushed
+to `mozilla-central <https://hg.mozilla.org/mozilla-central>`__,
+`autoland <https://hg.mozilla.org/integration/autoland/>`__, or
+`try </tools/try/index.html>`_, with the results showing up on
+`Treeherder <https://treeherder.mozilla.org/>`__. Not all tests will be run on
+every changeset; alogrithms are put in place to run the most likely failures,
+with all tests being run on a regular basis.
+
+They can also be run on local builds.
+Note: Most of the mobile tests run on emulators, but some of the tests
+(notably, performance tests) run on hardware devices.
+We try to avoid running mobile tests on hardware devices unnecessarily.
+In Treeherder, tests with names that start with "hw-" run on hardware.
+
+Linting
+~~~~~~~
+
+Lint tests help to ensure better quality, less error-prone code by
+analysing the code with a linter.
+
+
+.. csv-table:: Linters
+   :header-rows: 1
+
+   "Treeherder Symbol", "Name", "Platforms", "What is Tested"
+   "``ES``", "`ESLint </code-quality/lint/linters/eslint.html>`__", "All", "JavaScript is analyzed for correctness."
+   "``ES-build``", "`eslint-build </code-quality/lint/linters/eslint.html#eslint-build-es-b>`_", "All", "Extended javascript analysis that uses build artifacts."
+   "``mocha(EPM)``", "`ESLint-plugin-mozilla </code-quality/lint/linters/eslint-plugin-mozilla.html>`__", "Desktop", "The ESLint plugin rules."
+   "``f8``", "`flake8 </code-quality/lint/linters/flake8.html>`__", "All", "Python analyzed for style and correctness."
+   "``stylelint``", "`Stylelint </code-quality/lint/linters/stylelint.html>`__", "All", "CSS is analyzed for correctness."
+   "``W``", "`wpt lint </web-platform/index.html>`__", "Desktop", "web-platform-tests analyzed for style and manifest correctness"
+   "``WR(tidy)``", "`WebRender servo-tidy </testing/webrender/index.html>`__", "Desktop", "Code in gfx/wr is run through servo-tidy."
+   "``A``", "`Spotless </code-quality/lint/linters/android-format.html>`_", "Android", "Java is analyzed for style and correctness."
+
+.. _Functional_testing:
+
+Functional testing
+~~~~~~~~~~~~~~~~~~
+
+.. csv-table:: Automated Test Suites
+   :header-rows: 2
+
+   "Treeherder Symbol", "Name", "Platform", "Process", "Environment", "", "Privilege", "What is Tested"
+   "", "", "", "", "Shell", "Browser Profile", "",
+   "``R(J)``", "JS Reftest", "Desktop", "N/A", "JSShell", "N/A", "N/A", "The JavaScript engine's implementation of the JavaScript language."
+   "``R(C)``", "`Crashtest </web-platform/index.html>`__", "All", "Child", "Content", "Yes", "Low", "That pages load without crashing, asserting, or leaking."
+   "``R(R)``", "`Reftest </web-platform/index.html>`__", "All", "Child", "Content", "Yes", "Low", "That pages are rendered (and thus also layed out) correctly."
+   "``GTest``", "`GTest </gtest/index.html>`__", "All", "N/A", "Terminal", "N/A", "N/A", "Code that is not exposed to JavaScript."
+   "``X``", "`xpcshell </testing/xpcshell/index.html>`__", "All", "Parent, Allow", "XPCShell", "Allow", "High", "Low-level code exposed to JavaScript, such as XPCOM components."
+   "``M(a11y)``", "Accessibility (mochitest-a11y)", "Desktop", "Child", "Content", "Yes", "?", "Accessibility interfaces."
+   "``M(1), M(2), M(...)``", "`Mochitest plain </testing/mochitest-plain/index.html>`__", "All", "Child", "Content", "Yes", "Low, Allow", "Features exposed to JavaScript in web content, like DOM and other Web APIs, where the APIs do not require elevated permissions to test."
+   "``M(c1/c2/c3)``", "`Mochitest chrome </testing/chrome-tests/index.html>`__", "All", "Child, Allow", "Content", "Yes", "High", "Code requiring UI or JavaScript interactions with privileged objects."
+   "``M(bc)``", "`Mochitest browser-chrome </testing/mochitest-plain/index.html>`__", "All", "Parent, Allow", "Browser", "Yes", "High", "How the browser UI interacts with itself and with content."
+   "``M(remote)``", "Mochitest Remote Protocol", "All", "Parent, Allow", "Browser", "Yes", "High", "Firefox Remote Protocol (Implements parts of Chrome dev-tools protocol). Based on Mochitest browser-chrome."
+   "``SM(...), SM(pkg)``", "`SpiderMonkey automation <https://wiki.mozilla.org/Javascript:Automation_Builds>`__", "Desktop", "N/A", "JSShell", "N/A", "Low", "SpiderMonkey engine shell tests and JSAPI tests."
+   "``W``", "`web-platform-tests </web-platform/index.html>`__", "Desktop", "Child", "Content", "Yes", "Low", "Standardized features exposed to ECMAScript in web content; tests are shared with other vendors."
+   "``Wr``", "`web-platform-tests </web-platform/writing-tests/reftests.html>`__", "All", "Child", "Content", "Yes", "Low", "Layout and graphic correctness for standardized features; tests are shared with other vendors."
+   "``Mn``", "`Marionette </testing/marionette/Testing.html>`__", "Desktop", "?", "Content, Browser", "?", "High", "Large out-of-process function integration tests and tests that do communication with multiple remote Gecko processes."
+   "``Fxfn``", "`Firefox UI Tests </remote/Testing.html#puppeteer-tests>`__", "Desktop", "?", "Content, Browser", "Yes", "High", "Integration tests with a focus on the user interface and localization."
+   "``tt(c)``", "`telemetry-tests-client </toolkit/components/telemetry/internals/tests.html>`__", "Desktop", "N/A", "Content, Browser", "Yes", "High", "Integration tests for the Firefox Telemetry client."
+   "``TV``", "`Test Verification (test-verify) </testing/test-verification/index.html>`__", "All", "Depends on test harness", "?", "?", "?", "Uses other test harnesses - mochitest, reftest, xpcshell - to perform extra testing on new/modified tests."
+   "``TVw``", "`Test Verification for wpt (test-verify-wpt) </testing/test-verification/index.html>`__", "Desktop", "Child", "?", "?", "?", "Uses wpt test harnesses to perform extra testing on new/modified web-platform tests."
+   "``WR(wrench)``", "`WebRender standalone tests </testing/webrender/index.html>`__", "All", "N/A", "Terminal", "N/A", "N/A", "WebRender rust code (as a standalone module, with Gecko integration)."
+
+Note: there are preference-based variations of the previous testing suites.
+For example, mochitests on Treeherder can have ``gli``, ``swr``, ``spi``,
+``nofis``, ``a11y-checks``, ``spi-nw-1proc``, and many others. Another
+example is GTest, which can use ``GTest-1proc``. To learn more about
+these variations, you can mouse hover over these items to read a
+description of what these abbreviations mean.
+
+.. _Table_key:
+
+Table key
+^^^^^^^^^
+
+Symbol
+   Abbreviation for the test suite used by
+   `Treeherder <https://treeherder.mozilla.org/>`__. The first letter
+   generally indicates which of the test harnesses is used to execute
+   the test. The letter in parentheses identifies the actual test suite.
+Name
+   Common name used when referring to the test suite.
+File type
+   When adding a new test, you will generally create a file of this type
+   in the source tree and then declare it in a manifest or makefile.
+Platform
+   Most test suites are supported only on a subset of the available
+   plaforms and operating systems. Unless otherwise noted:
+
+   -  **Desktop** tests run on Windows, Mac OS X, and Linux.
+   -  **Mobile** tests run on Android emulators or remotely on Android
+      devices.
+
+Process
+   -  When **Parent** is indicated, the test file will always run in the
+      parent process, even when the browser is running in Electrolysis
+      (e10s) mode.
+   -  When **Child** is indicated, the test file will run in the child
+      process when the browser is running in Electrolysis (e10s) mode.
+   -  The **Allow** label indicates that the test has access to
+      mechanisms to run code in the other process.
+
+Environment
+   -  The **JSShell** and **XPCShell** environments are limited
+      JavaScript execution environments with no windows or user
+      interface (note however that XPCShell tests on Android are run
+      within a browser window.)
+   -  The **Content** indication means that the test is run inside a
+      content page loaded by a browser window.
+   -  The **Browser** indication means that the test is loaded in the
+      JavaScript context of a browser XUL window.
+   -  The **Browser Profile** column indicates whether a browser profile
+      is loaded when the test starts. The **Allow** label means that the
+      test can optionally load a profile using a special command.
+
+Privilege
+   Indicates whether the tests normally run with low (content) or high
+   (chrome) JavaScript privileges. The **Allow** label means that the
+   test can optionally run code in a privileged environment using a
+   special command.
+
+.. _Performance_testing:
+
+Performance testing
+~~~~~~~~~~~~~~~~~~~
+
+There are many test harnesses used to test performance.
+`For more information on the various performance harnesses,
+check out the perf docs. </testing/perfdocs>`_
+
+
+.. _So_which_should_I_use:
+
+So which should I use?
+----------------------
+
+Generally, you should pick the lowest-level framework that you can. If
+you are testing JavaScript but don't need a window, use XPCShell or even
+JSShell. If you're testing page layout, try to use
+`web-platform-test reftest.
+<https://web-platform-tests.org/writing-tests/reftests.html>`_
+The advantage in lower level testing is that you don't drag in a lot of
+other components that might have their own problems, so you can home in
+quickly on any bugs in what you are specifically testing.
+
+Here's a series of questions to ask about your work when you want to
+write some tests.
+
+.. _Is_it_low-level_code:
+
+Is it low-level code?
+~~~~~~~~~~~~~~~~~~~~~
+
+If the functionality is exposed to JavaScript, and you don't need a
+window, consider `XPCShell </testing/xpcshell/index.html>`__. If not,
+you'll probably have to use `GTest </gtest/index.html>`__, which can
+test pretty much anything. In general, this should be your
+last option for a new test, unless you have to test something that is
+not exposed to JavaScript.
+
+.. _Does_it_cause_a_crash:
+
+Does it cause a crash?
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you've found pages that crash Firefox, add a
+`crashtest </web-platform/index.html>`__ to
+make sure future versions don't experience this crash (assertion or
+leak) again. Note that this may lead to more tests once the core
+problem is found.
+
+.. _Is_it_a_layoutgraphics_feature:
+
+Is it a layout/graphics feature?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`Reftest </layout/Reftest.html#writing-tests>`__ is your best bet, if possible.
+
+.. _Do_you_need_to_verify_performance:
+
+Do you need to verify performance?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`Use an appropriate performance test suite from this list </testing/perfdocs>`_.
+
+.. _Are_you_testing_UI_features:
+
+Are you testing UI features?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Try one of the flavors of
+`mochitest </testing/mochitest-plain/index.html>`__, or
+`Marionette </docs/Marionette>`__ if the application also needs to be
+restarted, or tested with localized builds.
+
+.. _Are_you_testing_MobileAndroid:
+
+Are you testing Mobile/Android?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are testing GeckoView, you will need to need to use
+`JUnit integration tests
+</mobile/android/geckoview/contributor/junit.html#testing-overview>`__.
+
+There are some specific features that
+`Mochitest </testing/mochitest-plain/index.html>`__ or
+`Reftest </layout/Reftest.html>`__ can cover. Browser-chrome tests do not run on
+Android. If you want to test performance, `Raptor </testing/perfdocs/raptor.html>`__ will
+be a good choice.
+
+
+.. _Are_you_doing_none_of_the_above:
+
+Are you doing none of the above?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  To get your tests running in continuous integration, try
+   `web-platform-tests </web-platform/index.html>`_, or
+   `Mochitest </testing/mochitest-plain/index.html>`__, or,
+   if higher privileges are required, try
+   `Mochitest browser chrome tests </testing/mochitest-plain/index.html>`__.
+-  For Desktop Firefox, or if you just want to see the future of Gecko
+   testing, look into the on-going
+   `Marionette </testing/marionette/Testing.html#harness-tests>`__ project.
+
+.. _Need_to_get_more_data_out_of_your_tests:
+
+Need to get more data out of your tests?
+----------------------------------------
+
+Most test jobs now expose an environment variable named
+``$MOZ_UPLOAD_DIR``. If this variable is set during automated test runs,
+you can drop additional files into this directory, and they will be
+uploaded to a web server when the test finishes. The URLs to retrieve
+the files will be output in the test log.
+
+.. _Need_to_set_preferences_for_test-suites:
+
+Need to set preferences for test-suites?
+----------------------------------------
+
+First ask yourself if these prefs need to be enabled for all tests or
+just a subset of tests (e.g to enable a feature).
+
+.. _Setting_prefs_that_only_apply_to_certain_tests:
+
+Setting prefs that only apply to certain tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the answer is the latter, try to set the pref as local to the tests
+that need it as possible. Here are some options:
+
+-  If the test runs in chrome scope (e.g mochitest chrome or
+   browser-chrome), you can use
+   `Services.prefs
+   <https://searchfox.org/mozilla-central/source/modules/libpref/nsIPrefBranch.idl>`__
+   to set the prefs in your test's setup function. Be sure to reset the
+   pref back to its original value during teardown!
+
+-  Mochitest plain tests can use
+   `SpecialPowers
+   <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest/SpecialPowers>`__
+   to set prefs.
+
+-  All variants of mochitest can set prefs in their manifests. For
+   example, to set a pref for all tests in a manifest:
+
+   ::
+
+      [DEFAULT]
+      prefs =
+        my.awesome.pref=foo,
+        my.other.awesome.pref=bar,
+      [test_foo.js]
+      [test_bar.js]
+
+-  All variants of reftest can also set prefs in their
+   `manifests </layout/Reftest.html>`__.
+
+-  All variants of web-platform-tests can also `set prefs in their
+   manifests </web-platform/index.html#enabling-prefs>`__.
+
+.. _Setting_prefs_that_apply_to_the_entire_suite:
+
+Setting prefs that apply to the entire suite
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Most test suites define prefs in user.js files that live under
+`testing/profiles
+<https://searchfox.org/mozilla-central/source/testing/profiles>`__.
+Each directory is a profile that contains a ``user.js`` file with a
+number of prefs defined in it. Test suites will then merge one or more
+of these basic profiles into their own profile at runtime. To see which
+profiles apply to which test suites, you can inspect
+`testing/profiles/profiles.json
+<https://searchfox.org/mozilla-central/source/testing/profiles/profiles.json>`__.
+Profiles at the beginning of the list get overridden by profiles at the
+end of the list.
+
+Because this system makes it hard to get an overall view of which
+profiles are set for any given test suite, a handy ``profile`` utility
+was created:
+
+::
+
+   $ cd testing/profiles
+   $ ./profile -- --help
+   usage: profile [-h] {diff,sort,show,rm} ...
+   $ ./profile show mochitest          # prints all prefs that will be set in mochitest
+   $ ./profile diff mochitest reftest  # prints differences between the mochitest and reftest suites
+
+.. container:: blockIndicator note
+
+   **Note:** JS engine tests do not use testing/profiles yet, instead
+   `set prefs
+   here <https://searchfox.org/mozilla-central/source/js/src/tests/user.js>`__.