diff options
Diffstat (limited to 'testing/docs')
| -rw-r--r-- | testing/docs/mochitest-plain/faq.md | 63 | ||||
| -rw-r--r-- | testing/docs/tests-for-new-config/index.rst | 4 | ||||
| -rw-r--r-- | testing/docs/tests-for-new-config/manual.rst | 4 | ||||
| -rw-r--r-- | testing/docs/xpcshell/index.rst | 50 |
4 files changed, 91 insertions, 30 deletions
diff --git a/testing/docs/mochitest-plain/faq.md b/testing/docs/mochitest-plain/faq.md index a852f135d8..d67f113b63 100644 --- a/testing/docs/mochitest-plain/faq.md +++ b/testing/docs/mochitest-plain/faq.md @@ -321,6 +321,67 @@ The accessibility tests could be run locally with the `--enable-a11y-checks` fla ./mach mochitest --enable-a11y-checks somePath/someTestFile.html ``` -On CI, a11y-checks only run on tier 2 Linux 18.04 x64 WebRender (Opt and Shippable) builds. If you'd like to run only a11y-checks on Try, you can run the ``./mach try fuzzy --full` command with the query `a11y-checks linux !wayland !tsan !asan !ccov !debug !devedition` for all checks. Alternatively, to exclude devtools chrome tests, pass the query `swr-a11y-checks` to `./mach try fuzzy --full`. +On CI, a11y-checks only run on tier 2 Linux 18.04 x64 WebRender (Opt and Shippable) builds. If you'd like to run only a11y-checks on Try, you can run the `./mach try fuzzy --full` command with the query `a11y-checks linux !wayland !tsan !asan !ccov !debug !devedition` for all checks. Alternatively, to exclude devtools chrome tests, pass the query `swr-a11y-checks` to `./mach try fuzzy --full`. If you have questions on the results of a11y-checks and the ways to remediate any issues, reach out to the Accessibility team the [#accessibility room on Matrix](https://matrix.to/#/#accessibility:mozilla.org). + +## How to debug a failing accessibility test (a11y-checks)? + +First, review the failure log to learn which element may be not accessible. For example: + +```TEST-UNEXPECTED-FAIL | path/to/specific/test/browser_test.js | Node is not accessible via accessibility API: id: foo, tagName: div, className: bar``` + +This failure reports that when the `browser_test.js` is running, an `<div id="foo" class="bar">` is clicked and this element may not be accessible. +The next step is to review the code for this element in general to ensure it is built to be accessible: +1. it [has an interactive role](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#825-877) +1. it [is enabled](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#768-781) +1. it [is focusable with the keyboard](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#783-823) +1. it [is labeled](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#879-993) + +When all of the above is true, the element [can be clicked](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#1150-1174). + +After that, check the element [is, in fact, visible](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#995-1009) and accessible when the test is clicking it. For example, when dealing with a pop-up panel, sometimes we may need to wait for the panel to be shown before clicking a button inside it. + +If the issue is not clear, feel free to reach out to the Accessibility team to brainstorm together. + +## My patch is failing a11y-checks. Can I skip them for now? + +In general, we skip the a11y_checks only when there are unexplained intermittents or crashes ([example](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/dom/midi/tests/browser.toml#12)). There are very few tests like that in our repository. + +If there is a failure and the resolution is not clear yet, or if troubleshooting would need a separate patch, ensure a follow-up bug is filed and refer to it while setting `fail-if` in the test manifest ([example](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/browser/components/ion/test/browser/browser.toml#4)). This allows accessibility tests to still be running and reporting in the log on any new UI that might be added in the future, even without a Tier 2 failure flagged. Then, when the issue is fixed and the test is passing, you can remove the `fail-if` notation from the test manifest. If you forget to remove the `fail-if` notation, you would be notified about an (unexpectedly) passing test to remind you to update the manifest. + +## What are the exceptions from a11y_checks? When and how should I implement them? + +Sometimes, a test case is expected to fail the Accessibility Utils tests. An example is when a mochitest clicks a disabled button to confirm that, in fact, nothing happens. In these cases, you can intentionally modify the [default test environment object](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/testing/mochitest/tests/SimpleTest/AccessibilityUtils.js#94-117) to disable specific checks. You can do this by calling `AccessibilityUtils.setEnv` right before the click and then reset it to the original state right after the click (to avoid unintentionally excluding further test cases). + +An `AccessibilityUtils.setEnv` call should generally be preceded by a descriptive comment. This helps to avoid unintentional copy-pasting of this code to a new test case, risking an accessibility regression. Refer to the examples linked in the most common exception cases below for example comments. + +### Clicking on a disabled control or other non-interactive UI to confirm the click event won't come through + +Failure reported as: `Node expected to be enabled but is disabled via the accessibility API`. + +These clicks are not meant to be interactive and their targets are not expected to be accessible or enabled via the Accessibility API. Thus, we add an a11y_checks exception for this test via `AccessibilityUtils.setEnv({ mustBeEnabled: false })` ([example](https://searchfox.org/mozilla-central/rev/eb4700a6be8371fe07053bc066c2d48ba813ce3d/browser/components/extensions/test/browser/browser_ext_menus_capture_secondary_click.js#126-131,133)) + +### Clicking on a non-interactive UI to confirm nothing happens + +Failure reported as: `Node is not accessible via accessibility API`. + +These clicks are not meant to be interactive and their targets are not expected to be accessible or available via the Accessibility API. Thus, we add an a11y_checks exception for this test via `AccessibilityUtils.setEnv({ mustHaveAccessibleRule: false })` ([example](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/toolkit/mozapps/extensions/test/browser/browser_sidebar_categories.js#85-89,91)) + +### Clicking on arbitrary web content and remote documents + +Failure reported as: `Node is not accessible via accessibility API`. + +We do not want to test remote web content because we do not currently support remote documents with a11y_checks. In addition, we recognize that some arbitrary remote content is written to exercise a specific browser function and therefore doesn't need to be complete. Thus, we add an a11y-checks exception for this test via `AccessibilityUtils.setEnv({ mustHaveAccessibleRule: false })` ([example](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/dom/tests/browser/browser_focus_steal_from_chrome_during_mousedown.js#28-32,36)) + +### Clicking on non-interactive content to dismiss a dialog/menupopup/panel + +Failure reported as: `Node is not accessible via accessibility API`. + +Some tests send a click on a non-interactive element (e.g. on a container, on the `<body>` of the page, etc.) to close an opened dialog, menu popup or panel. While this method of interaction is inaccessible to some users, it is acceptable as long as there is an alternative, accessible way to dismiss the popup for users of keyboards and assistive technology; e.g. pressing the `Esc` key or a `X` Close button. In this case, we need to add an a11y_checks exception for this test via `AccessibilityUtils.setEnv({ mustHaveAccessibleRule: false })` while explicitly mentioning in the comment that at least one other way to dismiss it exists which does not rely on a mouse ([example clicking on a <body>](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/dom/events/test/clipboard/head.js#104-110,128) and [example clicking on a spacer](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/toolkit/mozapps/extensions/test/browser/browser_menu_button_accessibility.js#32-37,39)) + +### Non-user-facing test cases that are never expected to be done by an end user (e.g. telemetry, performance, crash tests) + +Failure reported as: `Node is not accessible via accessibility API`. + +Sometimes, we test a behavior that is never expected to be done by a real user; e.g. confirming a crash test patch is working, or clicking on an element to test a telemetry or performance action. As long as we have other tests checking the same UI for accessibility, you can add an a11y_checks exception for this test via `AccessibilityUtils.setEnv({ mustHaveAccessibleRule: false })` while explicitly mentioning in the comment the reason for this exclusion ([example clicking on a <body> for a performance testing](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/devtools/client/debugger/test/mochitest/browser_dbg-javascript-tracer-next-interation.js#51-56,62), [example of a crashtest](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/layout/base/tests/browser_bug1701027-1.js#94-99), [example of a telemetry behavior test](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/browser/components/urlbar/tests/engagementTelemetry/browser/browser_glean_telemetry_abandonment_tips.js#56-62,64,73-79,81), and [example of clicking on a hidden panel to confirm its content was refreshed](https://searchfox.org/mozilla-central/rev/1f5e1875cbfd5d4b1bfa27ca54832f62dd19589e/browser/base/content/test/permissions/browser_site_scoped_permissions.js#49-56,58,99-106,108)). diff --git a/testing/docs/tests-for-new-config/index.rst b/testing/docs/tests-for-new-config/index.rst index 32fdf62b7f..bbd3b2339e 100644 --- a/testing/docs/tests-for-new-config/index.rst +++ b/testing/docs/tests-for-new-config/index.rst @@ -110,8 +110,8 @@ Using our example of ``windows_1903``, this would be a new worker type that would require these edits: * `transforms/tests.py <https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#97>`__ (duplicate windows 10 entries) - * `test-platforms.py <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#229>`__ (copy windows10 debug/opt/shippable/asan entries and make win10_1903) - * `test-sets.py <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#293>`__ (ideally you need nothing, otherwise copy ``windows-tests`` and edit the test list) + * `test-platforms.py <https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/test-platforms.yml#229>`__ (copy windows10 debug/opt/shippable/asan entries and make win10_1903) + * `test-sets.py <https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/test-sets.yml#293>`__ (ideally you need nothing, otherwise copy ``windows-tests`` and edit the test list) In general this should allow you to have tests scheduled with no custom flags in try server and all of these will be scheduled by default on diff --git a/testing/docs/tests-for-new-config/manual.rst b/testing/docs/tests-for-new-config/manual.rst index cf2485251a..246d60111a 100644 --- a/testing/docs/tests-for-new-config/manual.rst +++ b/testing/docs/tests-for-new-config/manual.rst @@ -204,8 +204,8 @@ Using our example of ``windows_1903``, this would be a new worker type that would require these edits: * `transforms/tests.py <https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#97>`__ (duplicate windows 10 entries) - * `test-platforms.py <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#229>`__ (copy windows10 debug/opt/shippable/asan entries and make win10_1903) - * `test-sets.py <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#293>`__ (ideally you need nothing, otherwise copy ``windows-tests`` and edit the test list) + * `test-platforms.py <https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/test-platforms.yml#229>`__ (copy windows10 debug/opt/shippable/asan entries and make win10_1903) + * `test-sets.py <https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/test-sets.yml#293>`__ (ideally you need nothing, otherwise copy ``windows-tests`` and edit the test list) In general this should allow you to have tests scheduled with no custom flags in try server and all of these will be scheduled by default on diff --git a/testing/docs/xpcshell/index.rst b/testing/docs/xpcshell/index.rst index 6fc6d8ba4b..0387cfb924 100644 --- a/testing/docs/xpcshell/index.rst +++ b/testing/docs/xpcshell/index.rst @@ -22,13 +22,13 @@ Creating a new test directory If you need to create a new test directory, then follow the steps here. The test runner needs to know about the existence of the tests and how -to configure them through the use of the ``xpcshell.ini`` manifest file. +to configure them through the use of the ``xpcshell.toml`` manifest file. -First add a ``XPCSHELL_TESTS_MANIFESTS += ['xpcshell.ini']`` declaration -(with the correct relative ``xpcshell.ini`` path) to the ``moz.build`` +First add a ``XPCSHELL_TESTS_MANIFESTS += ['xpcshell.toml']`` declaration +(with the correct relative ``xpcshell.toml`` path) to the ``moz.build`` file located in or above the directory. -Then create an empty ``xpcshell.ini`` file to tell the build system +Then create an empty ``xpcshell.toml`` file to tell the build system about the individual tests, and provide any additional configuration options. @@ -44,7 +44,7 @@ run: $ hg add path/to/test/test_example.js This will automatically create the test file and add it to -``xpcshell.ini``, the second line adds it to your commit. +``xpcshell.toml``, the second line adds it to your commit. The test file contains an empty test which will give you an idea of how to write a test. There are plenty more examples throughout @@ -282,8 +282,8 @@ Multiprocess communication **optionalCallback** or the value to which the promise resolves. -xpcshell.ini manifest ---------------------- +xpcshell.toml manifest +---------------------- The manifest controls what tests are included in a test suite, and the configuration of the tests. It is loaded via the \`moz.build\` property @@ -306,7 +306,7 @@ listed under the ``[DEFAULT]`` section of the manifest. browser/ directory (e.g. additional XPCOM services that live there) ``skip-if`` ``run-if`` ``fail-if`` For this entire test suite, run the tests only if they meet certain - conditions. See `Adding conditions in the xpcshell.ini + conditions. See `Adding conditions in the xpcshell.toml manifest <#adding-conditions-through-the-add-task-or-add-test-function>`__ for how to use these properties. ``support-files`` @@ -320,19 +320,19 @@ listed under the ``[DEFAULT]`` section of the manifest. brackets -Creating a new xpcshell.ini file -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Creating a new xpcshell.toml file +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When creating a new directory and new xpcshell.ini manifest file, the +When creating a new directory and new xpcshell.toml manifest file, the following must be added to a moz.build file near that file in the directory hierarchy: .. code:: bash - XPCSHELL_TESTS_MANIFESTS += ['path/to/xpcshell.ini'] + XPCSHELL_TESTS_MANIFESTS += ['path/to/xpcshell.toml'] Typically, the moz.build containing *XPCSHELL_TESTS_MANIFESTS* is not in -the same directory as *xpcshell.ini*, but rather in a parent directory. +the same directory as *xpcshell.toml*, but rather in a parent directory. Common directory structures look like: .. code:: bash @@ -340,7 +340,7 @@ Common directory structures look like: feature ├──moz.build └──tests/xpcshell - └──xpcshell.ini + └──xpcshell.toml # or @@ -349,7 +349,7 @@ Common directory structures look like: └──tests ├──moz.build └──xpcshell - └──xpcshell.ini + └──xpcshell.toml Test head and support files @@ -357,13 +357,13 @@ Test head and support files Typically in a test suite, similar setup code and dependencies will need to be loaded in across each test. This can be done through the test -head, which is the file declared in the ``xpcshell.ini`` manifest file +head, which is the file declared in the ``xpcshell.toml`` manifest file under the ``head`` property. The file itself is typically called ``head.js``. Any variable declared in the test head will be in the global scope of each test in that test suite. In addition to the test head, other support files can be declared in the -``xpcshell.ini`` manifest file. This is done through the +``xpcshell.toml`` manifest file. This is done through the ``support-files`` declaration. These files will be made available through the url ``resource://test`` plus the name of the file. These files can then be loaded in using the @@ -380,11 +380,11 @@ well, and they will be made available by their filename. ├──module.mjs ├──moz.build ├──test_example.js - └──xpcshell.ini + └──xpcshell.toml -.. code:: ini +.. code:: toml - # xpcshell.ini + # xpcshell.toml [DEFAULT] head = head.js support-files = @@ -586,8 +586,8 @@ on Mac OS X, the test will be skipped on all other platforms. specifics of what the condition is. -Adding conditions in the xpcshell.ini manifest -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Adding conditions in the xpcshell.toml manifest +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sometimes you may want to add conditions to specify that a test should be skipped in certain configurations, or that a test is known to fail on @@ -749,13 +749,13 @@ If you get a message such as: 0:00.62 ERROR Failed to initialize debugging: Error: resource://devtools appears to be inaccessible from the xpcshell environment. This can usually be resolved by adding: firefox-appdir = browser - to the xpcshell.ini manifest. + to the xpcshell.toml manifest. It is possible for this to alter test behevior by triggering additional browser code to run, so check test behavior after making this change. This is typically a test in core code. You can attempt to add that to -the xpcshell.ini, however as it says, it might affect how the test runs +the xpcshell.toml, however as it says, it might affect how the test runs and cause failures. Generally the firefox-appdir should only be left in -xpcshell.ini for tests that are in the browser/ directory, or are +xpcshell.toml for tests that are in the browser/ directory, or are Firefox-only. |