1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
|
# Manual Tests
Some testing scenarios are intrinsically difficult to automate and
require a human to run the test and check the pass condition.
## When to Write Manual Tests
Whenever possible it's best to write a fully automated test. For a
browser vendor it's possible to run an automated test hundreds of
times a day, but manual tests are likely to be run at most a handful
of times a year (and quite possibly approximately never!). This makes
them significantly less useful for catching regressions than automated
tests.
However, there are certain scenarios in which this is not yet
possible. For example:
* Test which require observing animation (e.g., a test for CSS
animation or for video playback),
* Tests that require interaction with browser security UI (e.g., a
test in which a user refuses a geolocation permissions grant),
* Tests that require interaction with the underlying OS (e.g., tests
for drag and drop from the desktop onto the browser),
* Tests that require non-default browser configuration (e.g., images
disabled), and
* Tests that require interaction with the physical environment (e.g.,
tests that the vibration API causes the device to vibrate or that
various sensor APIs respond in the expected way).
## Requirements for a Manual Test
Manual tests are distinguished by their filename; all manual tests
have filenames of the form `name-manual.ext` (i.e., a `-manual` suffix
after the main filename but before the extension).
Manual tests must be
fully
[self-describing](general-guidelines).
It is particularly important for these tests that it is easy to
determine the result from the information provided in the page to the
tester, because a tester may have hundreds of tests to get through and
little understanding of the features that they are testing. As a
result, minimalism is especially a virtue for manual tests.
A test should have, at a minimum step-by-step instructions for
performing the test, and a clear statement of either the test result
if it can be automatically determined after some setup or how to
otherwise determine the outcome.
Any information other than this (e.g., quotes from the spec) should be
avoided (though, as always, can be provided in
HTML/CSS/JS/etc. comments).
## Using testharness.js for Manual Tests
A convenient way to present the results of a test that can have the
result determined by script after some manual setup steps is to use
testharness.js to determine and present the result. In this case one
must pass `{explicit_timeout: true}` in a call to `setup()` in order
to disable the automatic timeout of the test. For example:
```html
<!doctype html>
<title>Manual click on button triggers onclick handler</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
setup({explicit_timeout: true})
</script>
<p>Click on the button below. If a "PASS" result appears the test
passes, otherwise it fails</p>
<button onclick="done()">Click Here</button>
```
|
