diff options
Diffstat (limited to 'testing/docs/automated-testing')
| -rw-r--r-- | testing/docs/automated-testing/index.rst | 348 | ||||
| -rw-r--r-- | testing/docs/automated-testing/manifest-sandbox.rst | 103 |
2 files changed, 451 insertions, 0 deletions
diff --git a/testing/docs/automated-testing/index.rst b/testing/docs/automated-testing/index.rst new file mode 100644 index 0000000000..632b3acf59 --- /dev/null +++ b/testing/docs/automated-testing/index.rst @@ -0,0 +1,348 @@ +Automated Testing +================= + +You've just written a feature and (hopefully!) want to test it. Or you've +decided that an existing feature doesn't have enough tests and want to contribute +some. But where do you start? You've looked around and found references to things +like "xpcshell" or "web-platform-tests" or "talos". What code, features or +platforms do they all test? Where do their feature sets overlap? In short, where +should your new tests go? This document is a starting point for those who want +to start to learn about Mozilla's automated testing tools and procedures. Below +you'll find a short summary of each framework we use, and some questions to help +you pick the framework(s) you need for your purposes. + +If you still have questions, ask on `Matrix <https://wiki.mozilla.org/Matrix>`__ +or on the relevant bug. + +Firefox Production +------------------ +These tests are found within the `mozilla-central <https://hg.mozilla.org/mozilla-central>`__ +tree, along with the product code. + +They are run when a changeset is pushed +to `mozilla-central <https://hg.mozilla.org/mozilla-central>`__, +`autoland <https://hg.mozilla.org/integration/autoland/>`__, or +`try </tools/try/index.html>`_, with the results showing up on +`Treeherder <https://treeherder.mozilla.org/>`__. Not all tests will be run on +every changeset; alogrithms are put in place to run the most likely failures, +with all tests being run on a regular basis. + +They can also be run on local builds. +Note: Most of the mobile tests run on emulators, but some of the tests +(notably, performance tests) run on hardware devices. +We try to avoid running mobile tests on hardware devices unnecessarily. +In Treeherder, tests with names that start with "hw-" run on hardware. + +Linting +~~~~~~~ + +Lint tests help to ensure better quality, less error-prone code by +analysing the code with a linter. + + +.. csv-table:: Linters + :header-rows: 1 + + "Treeherder Symbol", "Name", "Platforms", "What is Tested" + "``ES``", "`ESLint </code-quality/lint/linters/eslint.html>`__", "All", "JavaScript is analyzed for correctness." + "``ES-build``", "`eslint-build </code-quality/lint/linters/eslint.html#eslint-build-es-b>`_", "All", "Extended javascript analysis that uses build artifacts." + "``mocha(EPM)``", "`ESLint-plugin-mozilla </code-quality/lint/linters/eslint-plugin-mozilla.html>`__", "Desktop", "The ESLint plugin rules." + "``f8``", "`flake8 </code-quality/lint/linters/flake8.html>`__", "All", "Python analyzed for style and correctness." + "``stylelint``", "`Stylelint </code-quality/lint/linters/stylelint.html>`__", "All", "CSS is analyzed for correctness." + "``W``", "`wpt lint </web-platform/index.html>`__", "Desktop", "web-platform-tests analyzed for style and manifest correctness" + "``WR(tidy)``", "`WebRender servo-tidy </testing/webrender/index.html>`__", "Desktop", "Code in gfx/wr is run through servo-tidy." + "``A``", "`Spotless </code-quality/lint/linters/android-format.html>`_", "Android", "Java is analyzed for style and correctness." + +.. _Functional_testing: + +Functional testing +~~~~~~~~~~~~~~~~~~ + +.. csv-table:: Automated Test Suites + :header-rows: 2 + + "Treeherder Symbol", "Name", "Platform", "Process", "Environment", "", "Privilege", "What is Tested" + "", "", "", "", "Shell", "Browser Profile", "", + "``R(J)``", "JS Reftest", "Desktop", "N/A", "JSShell", "N/A", "N/A", "The JavaScript engine's implementation of the JavaScript language." + "``R(C)``", "`Crashtest </web-platform/index.html>`__", "All", "Child", "Content", "Yes", "Low", "That pages load without crashing, asserting, or leaking." + "``R(R)``", "`Reftest </web-platform/index.html>`__", "All", "Child", "Content", "Yes", "Low", "That pages are rendered (and thus also layed out) correctly." + "``GTest``", "`GTest </gtest/index.html>`__", "All", "N/A", "Terminal", "N/A", "N/A", "Code that is not exposed to JavaScript." + "``X``", "`xpcshell </testing/xpcshell/index.html>`__", "All", "Parent, Allow", "XPCShell", "Allow", "High", "Low-level code exposed to JavaScript, such as XPCOM components." + "``M(a11y)``", "Accessibility (mochitest-a11y)", "Desktop", "Child", "Content", "Yes", "?", "Accessibility interfaces." + "``M(1), M(2), M(...)``", "`Mochitest plain </testing/mochitest-plain/index.html>`__", "All", "Child", "Content", "Yes", "Low, Allow", "Features exposed to JavaScript in web content, like DOM and other Web APIs, where the APIs do not require elevated permissions to test." + "``M(c1/c2/c3)``", "`Mochitest chrome </testing/chrome-tests/index.html>`__", "All", "Child, Allow", "Content", "Yes", "High", "Code requiring UI or JavaScript interactions with privileged objects." + "``M(bc)``", "`Mochitest browser-chrome </testing/mochitest-plain/index.html>`__", "All", "Parent, Allow", "Browser", "Yes", "High", "How the browser UI interacts with itself and with content." + "``M(remote)``", "Mochitest Remote Protocol", "All", "Parent, Allow", "Browser", "Yes", "High", "Firefox Remote Protocol (Implements parts of Chrome dev-tools protocol). Based on Mochitest browser-chrome." + "``SM(...), SM(pkg)``", "`SpiderMonkey automation <https://wiki.mozilla.org/Javascript:Automation_Builds>`__", "Desktop", "N/A", "JSShell", "N/A", "Low", "SpiderMonkey engine shell tests and JSAPI tests." + "``W``", "`web-platform-tests </web-platform/index.html>`__", "Desktop", "Child", "Content", "Yes", "Low", "Standardized features exposed to ECMAScript in web content; tests are shared with other vendors." + "``Wr``", "`web-platform-tests </web-platform/writing-tests/reftests.html>`__", "All", "Child", "Content", "Yes", "Low", "Layout and graphic correctness for standardized features; tests are shared with other vendors." + "``Mn``", "`Marionette </testing/marionette/Testing.html>`__", "Desktop", "?", "Content, Browser", "?", "High", "Large out-of-process function integration tests and tests that do communication with multiple remote Gecko processes." + "``Fxfn``", "`Firefox UI Tests </remote/Testing.html#puppeteer-tests>`__", "Desktop", "?", "Content, Browser", "Yes", "High", "Integration tests with a focus on the user interface and localization." + "``tt(c)``", "`telemetry-tests-client </toolkit/components/telemetry/internals/tests.html>`__", "Desktop", "N/A", "Content, Browser", "Yes", "High", "Integration tests for the Firefox Telemetry client." + "``TV``", "`Test Verification (test-verify) </testing/test-verification/index.html>`__", "All", "Depends on test harness", "?", "?", "?", "Uses other test harnesses - mochitest, reftest, xpcshell - to perform extra testing on new/modified tests." + "``TVw``", "`Test Verification for wpt (test-verify-wpt) </testing/test-verification/index.html>`__", "Desktop", "Child", "?", "?", "?", "Uses wpt test harnesses to perform extra testing on new/modified web-platform tests." + "``WR(wrench)``", "`WebRender standalone tests </testing/webrender/index.html>`__", "All", "N/A", "Terminal", "N/A", "N/A", "WebRender rust code (as a standalone module, with Gecko integration)." + +Note: there are preference-based variations of the previous testing suites. +For example, mochitests on Treeherder can have ``gli``, ``swr``, ``spi``, +``nofis``, ``a11y-checks``, ``spi-nw-1proc``, and many others. Another +example is GTest, which can use ``GTest-1proc``. To learn more about +these variations, you can mouse hover over these items to read a +description of what these abbreviations mean. + +.. _Table_key: + +Table key +^^^^^^^^^ + +Symbol + Abbreviation for the test suite used by + `Treeherder <https://treeherder.mozilla.org/>`__. The first letter + generally indicates which of the test harnesses is used to execute + the test. The letter in parentheses identifies the actual test suite. +Name + Common name used when referring to the test suite. +File type + When adding a new test, you will generally create a file of this type + in the source tree and then declare it in a manifest or makefile. +Platform + Most test suites are supported only on a subset of the available + plaforms and operating systems. Unless otherwise noted: + + - **Desktop** tests run on Windows, Mac OS X, and Linux. + - **Mobile** tests run on Android emulators or remotely on Android + devices. + +Process + - When **Parent** is indicated, the test file will always run in the + parent process, even when the browser is running in Electrolysis + (e10s) mode. + - When **Child** is indicated, the test file will run in the child + process when the browser is running in Electrolysis (e10s) mode. + - The **Allow** label indicates that the test has access to + mechanisms to run code in the other process. + +Environment + - The **JSShell** and **XPCShell** environments are limited + JavaScript execution environments with no windows or user + interface (note however that XPCShell tests on Android are run + within a browser window.) + - The **Content** indication means that the test is run inside a + content page loaded by a browser window. + - The **Browser** indication means that the test is loaded in the + JavaScript context of a browser XUL window. + - The **Browser Profile** column indicates whether a browser profile + is loaded when the test starts. The **Allow** label means that the + test can optionally load a profile using a special command. + +Privilege + Indicates whether the tests normally run with low (content) or high + (chrome) JavaScript privileges. The **Allow** label means that the + test can optionally run code in a privileged environment using a + special command. + +.. _Performance_testing: + +Performance testing +~~~~~~~~~~~~~~~~~~~ + +There are many test harnesses used to test performance. +`For more information on the various performance harnesses, +check out the perf docs. </testing/perfdocs>`_ + + +.. _So_which_should_I_use: + +So which should I use? +---------------------- + +Generally, you should pick the lowest-level framework that you can. If +you are testing JavaScript but don't need a window, use XPCShell or even +JSShell. If you're testing page layout, try to use +`web-platform-test reftest. +<https://web-platform-tests.org/writing-tests/reftests.html>`_ +The advantage in lower level testing is that you don't drag in a lot of +other components that might have their own problems, so you can home in +quickly on any bugs in what you are specifically testing. + +Here's a series of questions to ask about your work when you want to +write some tests. + +.. _Is_it_low-level_code: + +Is it low-level code? +~~~~~~~~~~~~~~~~~~~~~ + +If the functionality is exposed to JavaScript, and you don't need a +window, consider `XPCShell </testing/xpcshell/index.html>`__. If not, +you'll probably have to use `GTest </gtest/index.html>`__, which can +test pretty much anything. In general, this should be your +last option for a new test, unless you have to test something that is +not exposed to JavaScript. + +.. _Does_it_cause_a_crash: + +Does it cause a crash? +~~~~~~~~~~~~~~~~~~~~~~ + +If you've found pages that crash Firefox, add a +`crashtest </web-platform/index.html>`__ to +make sure future versions don't experience this crash (assertion or +leak) again. Note that this may lead to more tests once the core +problem is found. + +.. _Is_it_a_layoutgraphics_feature: + +Is it a layout/graphics feature? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`Reftest </layout/Reftest.html#writing-tests>`__ is your best bet, if possible. + +.. _Do_you_need_to_verify_performance: + +Do you need to verify performance? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`Use an appropriate performance test suite from this list </testing/perfdocs>`_. + +.. _Are_you_testing_UI_features: + +Are you testing UI features? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Try one of the flavors of +`mochitest </testing/mochitest-plain/index.html>`__, or +`Marionette </docs/Marionette>`__ if the application also needs to be +restarted, or tested with localized builds. + +.. _Are_you_testing_MobileAndroid: + +Are you testing Mobile/Android? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are testing GeckoView, you will need to need to use +`JUnit integration tests +</mobile/android/geckoview/contributor/junit.html#testing-overview>`__. + +There are some specific features that +`Mochitest </testing/mochitest-plain/index.html>`__ or +`Reftest </layout/Reftest.html>`__ can cover. Browser-chrome tests do not run on +Android. If you want to test performance, `Raptor </testing/perfdocs/raptor.html>`__ will +be a good choice. + + +.. _Are_you_doing_none_of_the_above: + +Are you doing none of the above? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- To get your tests running in continuous integration, try + `web-platform-tests </web-platform/index.html>`_, or + `Mochitest </testing/mochitest-plain/index.html>`__, or, + if higher privileges are required, try + `Mochitest browser chrome tests </testing/mochitest-plain/index.html>`__. +- For Desktop Firefox, or if you just want to see the future of Gecko + testing, look into the on-going + `Marionette </testing/marionette/Testing.html#harness-tests>`__ project. + +.. _Need_to_get_more_data_out_of_your_tests: + +Need to get more data out of your tests? +---------------------------------------- + +Most test jobs now expose an environment variable named +``$MOZ_UPLOAD_DIR``. If this variable is set during automated test runs, +you can drop additional files into this directory, and they will be +uploaded to a web server when the test finishes. The URLs to retrieve +the files will be output in the test log. + +.. _Need_to_set_preferences_for_test-suites: + +Need to set preferences for test-suites? +---------------------------------------- + +First ask yourself if these prefs need to be enabled for all tests or +just a subset of tests (e.g to enable a feature). + +.. _Setting_prefs_that_only_apply_to_certain_tests: + +Setting prefs that only apply to certain tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the answer is the latter, try to set the pref as local to the tests +that need it as possible. Here are some options: + +- If the test runs in chrome scope (e.g mochitest chrome or + browser-chrome), you can use + `Services.prefs + <https://searchfox.org/mozilla-central/source/modules/libpref/nsIPrefBranch.idl>`__ + to set the prefs in your test's setup function. Be sure to reset the + pref back to its original value during teardown! + +- Mochitest plain tests can use + `SpecialPowers + <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest/SpecialPowers>`__ + to set prefs. + +- All variants of mochitest can set prefs in their manifests. For + example, to set a pref for all tests in a manifest: + + :: + + [DEFAULT] + prefs = + my.awesome.pref=foo, + my.other.awesome.pref=bar, + [test_foo.js] + [test_bar.js] + +- All variants of reftest can also set prefs in their + `manifests </layout/Reftest.html>`__. + +- All variants of web-platform-tests can also `set prefs in their + manifests </web-platform/index.html#enabling-prefs>`__. + +.. _Setting_prefs_that_apply_to_the_entire_suite: + +Setting prefs that apply to the entire suite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Most test suites define prefs in user.js files that live under +`testing/profiles +<https://searchfox.org/mozilla-central/source/testing/profiles>`__. +Each directory is a profile that contains a ``user.js`` file with a +number of prefs defined in it. Test suites will then merge one or more +of these basic profiles into their own profile at runtime. To see which +profiles apply to which test suites, you can inspect +`testing/profiles/profiles.json +<https://searchfox.org/mozilla-central/source/testing/profiles/profiles.json>`__. +Profiles at the beginning of the list get overridden by profiles at the +end of the list. + +Because this system makes it hard to get an overall view of which +profiles are set for any given test suite, a handy ``profile`` utility +was created: + +:: + + $ cd testing/profiles + $ ./profile -- --help + usage: profile [-h] {diff,sort,show,rm} ... + $ ./profile show mochitest # prints all prefs that will be set in mochitest + $ ./profile diff mochitest reftest # prints differences between the mochitest and reftest suites + +.. container:: blockIndicator note + + **Note:** JS engine tests do not use testing/profiles yet, instead + `set prefs + here <https://searchfox.org/mozilla-central/source/js/src/tests/user.js>`__. + +Adding New Context to Skip Conditions +------------------------------------- + +Often when standing up new test configurations, it's necessary to add new keys +that can be used in ``skip-if`` annotations. + +.. toctree:: + + manifest-sandbox diff --git a/testing/docs/automated-testing/manifest-sandbox.rst b/testing/docs/automated-testing/manifest-sandbox.rst new file mode 100644 index 0000000000..19df4ed883 --- /dev/null +++ b/testing/docs/automated-testing/manifest-sandbox.rst @@ -0,0 +1,103 @@ +Adding Context to ``manifestparser`` based Manifests +---------------------------------------------------- + +Suites that use ``manifestparser``, like Mochitest and XPCShell, have test +manifests that denote whether a given test should be skipped or not based +on a set of context. + +Gecko builds generate a ``target.mozinfo.json`` with metadata about the build. +An example ``target.mozinfo.json`` might look like :download:`this +<target.mozinfo.json>`. These keys can then be used ``skip-if`` in test manifests: + +.. code-block:: + + skip-if = e10s && os == 'win' + +In this case, ``e10s`` is a boolean. + +The test will download the build's ``target.mozinfo.json``, then update the +mozinfo dictionary with additional runtime information based on the task or +runtime environment. This logic lives in `mozinfo +<https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/mozinfo/mozinfo/mozinfo.py>`__. + +How to Add a Keyword +~~~~~~~~~~~~~~~~~~~~ + +Where to add the new key depends on what type of information it is. + +1. If the key is a property of the build, you'll need to patch `this file + <https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/mozinfo.py>`_. +2. If the key is a property of the test environment, you'll need to patch + `mozinfo <https://firefox-source-docs.mozilla.org/mozbase/mozinfo.html>`_. +3. If the key is a runtime configuration, for example based on a pref that is + passed in via mach or the task configuration, then you'll need to update the + individual test harnesses. For example, `this location + <https://searchfox.org/mozilla-central/rev/a7e33b7f61e7729e2b1051d2a7a27799f11a5de6/testing/mochitest/runtests.py#3341>`_ + for Mochitest. Currently there is no shared location to set runtime keys + across test harnesses. + +Adding a Context to Reftest Style Manifests +------------------------------------------- + +Reftests and Crashtests use a different kind of manifest, but the general idea +is the same. + +As before, Gecko builds generate a ``target.mozinfo.json`` with metadata about +the build. An example ``target.mozinfo.json`` might look like :download:`this +<target.mozinfo.json>`. This is consumed in the Reftest harness and translated +into keywords that can be used like: + +.. code-block:: + + fuzzyIf(cocoaWidget&&isDebugBuild,1-1,85-88) + +In this case, ``cocoaWidget`` and ``isDebugbuild`` are booleans. + +The test will download the build's ``target.mozinfo.json``, then in addition to +the mozinfo, will query runtime info from the browser to build a sandbox of +keywords. This logic lives in `manifest.sys.mjs +<https://searchfox.org/mozilla-central/source/layout/tools/reftest/manifest.sys.mjs#439>`__. + +How to Add a Keyword +~~~~~~~~~~~~~~~~~~~~ + +Where to add the new key depends on what type of information it is. + +1. If the key is a property of the build, you'll need to patch `this file + <https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/mozinfo.py>`_. +2. If the key is a property of the test environment or a runtime configuration, + then you'll need need to update manifest sandbox. + +For example, for Apple Silicon, we can add an ``apple_silicon`` keyword with a +patch like this: + +.. code-block:: diff + + --- a/layout/tools/reftest/manifest.sys.mjs + +++ b/layout/tools/reftest/manifest.sys.mjs + @@ -572,16 +572,18 @@ function BuildConditionSandbox(aURL) { + + // Set OSX to be the Mac OS X version, as an integer, or undefined + // for other platforms. The integer is formed by 100 times the + // major version plus the minor version, so 1006 for 10.6, 1010 for + // 10.10, etc. + var osxmatch = /Mac OS X (\d+).(\d+)$/.exec(hh.oscpu); + sandbox.OSX = osxmatch ? parseInt(osxmatch[1]) * 100 + parseInt(osxmatch[2]) : undefined; + + + sandbox.apple_silicon = sandbox.cocoaWidget && sandbox.OSX>=11; + + + // Plugins are no longer supported. Don't try to use TestPlugin. + sandbox.haveTestPlugin = false; + + // Set a flag on sandbox if the windows default theme is active + sandbox.windowsDefaultTheme = g.containingWindow.matchMedia("(-moz-windows-default-theme)").matches; + + try { + sandbox.nativeThemePref = !prefs.getBoolPref("widget.disable-native-theme-for-content"); + + +Then to use this: + +.. code-block:: + + fuzzy-if(apple_silicon,1-1,281-281) == frame_above_rules_none.html frame_above_rules_none_ref.html |