diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-19 01:47:29 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-19 01:47:29 +0000 |
| commit | 0ebf5bdf043a27fd3dfb7f92e0cb63d88954c44d (patch) | |
| tree | a31f07c9bcca9d56ce61e9a1ffd30ef350d513aa /testing/web-platform/tests/docs/writing-tests/manual.md | |
| parent | Initial commit. (diff) | |
| download | firefox-esr-0ebf5bdf043a27fd3dfb7f92e0cb63d88954c44d.tar.xz firefox-esr-0ebf5bdf043a27fd3dfb7f92e0cb63d88954c44d.zip | |
Adding upstream version 115.8.0esr.upstream/115.8.0esr
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'testing/web-platform/tests/docs/writing-tests/manual.md')
| -rw-r--r-- | testing/web-platform/tests/docs/writing-tests/manual.md | 77 |
1 files changed, 77 insertions, 0 deletions
diff --git a/testing/web-platform/tests/docs/writing-tests/manual.md b/testing/web-platform/tests/docs/writing-tests/manual.md new file mode 100644 index 0000000000..122a22b3f3 --- /dev/null +++ b/testing/web-platform/tests/docs/writing-tests/manual.md @@ -0,0 +1,77 @@ +# Manual Tests + +Some testing scenarios are intrinsically difficult to automate and +require a human to run the test and check the pass condition. + +## When to Write Manual Tests + +Whenever possible it's best to write a fully automated test. For a +browser vendor it's possible to run an automated test hundreds of +times a day, but manual tests are likely to be run at most a handful +of times a year (and quite possibly approximately never!). This makes +them significantly less useful for catching regressions than automated +tests. + +However, there are certain scenarios in which this is not yet +possible. For example: + +* Test which require observing animation (e.g., a test for CSS + animation or for video playback), + +* Tests that require interaction with browser security UI (e.g., a + test in which a user refuses a geolocation permissions grant), + +* Tests that require interaction with the underlying OS (e.g., tests + for drag and drop from the desktop onto the browser), + +* Tests that require non-default browser configuration (e.g., images + disabled), and + +* Tests that require interaction with the physical environment (e.g., + tests that the vibration API causes the device to vibrate or that + various sensor APIs respond in the expected way). + +## Requirements for a Manual Test + +Manual tests are distinguished by their filename; all manual tests +have filenames of the form `name-manual.ext` (i.e., a `-manual` suffix +after the main filename but before the extension). + +Manual tests must be +fully +[self-describing](general-guidelines). +It is particularly important for these tests that it is easy to +determine the result from the information provided in the page to the +tester, because a tester may have hundreds of tests to get through and +little understanding of the features that they are testing. As a +result, minimalism is especially a virtue for manual tests. + +A test should have, at a minimum step-by-step instructions for +performing the test, and a clear statement of either the test result +if it can be automatically determined after some setup or how to +otherwise determine the outcome. + +Any information other than this (e.g., quotes from the spec) should be +avoided (though, as always, can be provided in +HTML/CSS/JS/etc. comments). + +## Using testharness.js for Manual Tests + +A convenient way to present the results of a test that can have the +result determined by script after some manual setup steps is to use +testharness.js to determine and present the result. In this case one +must pass `{explicit_timeout: true}` in a call to `setup()` in order +to disable the automatic timeout of the test. For example: + +```html +<!doctype html> +<title>Manual click on button triggers onclick handler</title> +<script src="/resources/testharness.js"></script> +<script src="/resources/testharnessreport.js"></script> +<script> +setup({explicit_timeout: true}) +</script> +<p>Click on the button below. If a "PASS" result appears the test +passes, otherwise it fails</p> +<button onclick="done()">Click Here</button> +``` |