301 lines
11 KiB
Markdown
301 lines
11 KiB
Markdown
# Mochitest
|
|
|
|
## DISCLAIMER
|
|
|
|
If you are testing web platform code, prefer using use a [wpt
|
|
test](/web-platform/index.rst) (preferably upstreamable ones).
|
|
|
|
## Introduction
|
|
|
|
Mochitest is an automated testing framework built on top of the
|
|
[MochiKit](https://mochi.github.io/mochikit/) JavaScript libraries.
|
|
|
|
Only things that can be tested using JavaScript (with chrome privileges!) can be
|
|
tested with this framework. Given some creativity, that's actually much more
|
|
than you might first think, but it's not possible to write Mochitest tests to
|
|
directly test a non-scripted C++ component, for example. (Use a compiled-code
|
|
test like [GTest](/gtest/index.rst) to do that.)
|
|
|
|
## Running tests
|
|
|
|
To run a single test (perhaps a new test you just added) or a subset of the
|
|
entire Mochitest suite, pass a path parameter to the `mach` command.
|
|
|
|
For example, to run only the test `test_CrossSiteXHR.html` in the Mozilla source
|
|
tree, you would run this command:
|
|
|
|
```
|
|
./mach test dom/security/test/cors/test_CrossSiteXHR.html
|
|
```
|
|
|
|
To run all the tests in `dom/svg/`, this command would work:
|
|
|
|
```
|
|
./mach test dom/svg/
|
|
```
|
|
|
|
You can also pass a manifest path to run all tests on that manifest:
|
|
|
|
```
|
|
./mach test dom/base/test/mochitest.ini
|
|
```
|
|
|
|
## Running flavors and subsuites
|
|
|
|
Flavors are variations of the default configuration used to run Mochitest. For
|
|
example, a flavor might have a slightly different set of prefs set for it, a
|
|
custom extension installed or even run in a completely different scope.
|
|
|
|
The Mochitest flavors are:
|
|
|
|
* **plain** - The most basic and common Mochitest. They run in content scope,
|
|
but can access certain privileged APIs with SpecialPowers.
|
|
|
|
* **browser** - These often test the browser UI itself and run in browser
|
|
window scope.
|
|
|
|
* **chrome** - These run in chrome scope and are typically used for testing
|
|
privileged JavaScript APIs. More information can be found
|
|
[here](../chrome-tests/index.rst).
|
|
|
|
* **a11y** - These test the accessibility interfaces. They can be found under
|
|
the top `accessible` directory and run in chrome scope. Note that these run
|
|
without e10s / fission.
|
|
|
|
A subsuite is similar to a flavor, except that it has an identical
|
|
configuration. It is just logically separated from the "default" subsuite for
|
|
display purposes. For example, devtools is a subsuite of the browser flavor.
|
|
There is no difference in how these two jobs are run. It exists so that the
|
|
devtools team can easily see and run their tests.
|
|
|
|
**Note**: There are also tags, which are similar to subsuites. Although they
|
|
both are used to logically group related sets of tests, they behave
|
|
differently. For example, applying a subsuite to a test removes that test from
|
|
the default set, whereas, a tag does not remove it.
|
|
|
|
By default, mach finds and runs every test in the given subdirectory no matter
|
|
which flavor or subsuite it belongs to. But sometimes, you might only want to
|
|
run a specific flavor or subsuite. This can be accomplished using the `--flavor`
|
|
(or `-f`) and `--subsuite` options respectively. For example:
|
|
|
|
|
|
```
|
|
./mach mochitest -f plain # runs all plain tests
|
|
./mach mochitest -f browser --subsuite devtools # runs all browser tests in the devtools subsuite
|
|
./mach mochitest -f chrome dom/indexedDB # runs all chrome tests in the dom/indexedDB subdirectory
|
|
```
|
|
|
|
In many cases, it won't be necessary to filter by flavor or subsuite as running
|
|
specific directories will do it implicitly. For example running:
|
|
|
|
```
|
|
./mach mochitest devtools/
|
|
```
|
|
|
|
Is a rough equivalent to running the `devtools` subsuite. There might be
|
|
situations where you might want to run tests that don't belong to any subsuite.
|
|
To do this, use:
|
|
|
|
```
|
|
./mach mochitest --subsuite default
|
|
```
|
|
|
|
## Debugging individual tests
|
|
|
|
If you need to debug an individual test, you could reload the page containing
|
|
the test with the debugger attached. If attaching a debugger before the problem
|
|
shows up is hard (for example, if the browser crashes as the test is loading),
|
|
you can specify a debugger when you run mochitest:
|
|
|
|
```
|
|
./mach mochitest --debugger=gdb ...
|
|
```
|
|
|
|
See also the `--debugger-args` and `--debugger-interactive` arguments. You can
|
|
also use the `--jsdebugger` argument to debug JavaScript.
|
|
|
|
## Finding errors
|
|
|
|
Search for the string `TEST-UNEXPECTED-FAIL` to find unexpected failures. You
|
|
can also search for `SimpleTest FINISHED` to see the final test summary.
|
|
## Logging results
|
|
|
|
The output from a test run can be sent to the console and/or a file (by default
|
|
the results are only displayed in the browser). There are several levels of
|
|
detail to choose from. The levels are `DEBUG`, `INFO`, `WARNING`, `ERROR` and
|
|
`CRITICAL`, where `DEBUG` produces the highest detail (everything), and
|
|
`CRITICAL` produces the least.
|
|
|
|
Mochitest uses structured logging. This means that you can use a set of command
|
|
line arguments to configure the log output. To log to stdout using the mach
|
|
formatter and log to a file in JSON format, you can use `--log-mach=-`
|
|
`--log-raw=mochitest.log`. By default the file logging level for all your
|
|
formatters is `INFO` but you can change this using `--log-mach-level=<level>`.
|
|
|
|
To turn on logging to the console use `--console-level=<level>`.
|
|
|
|
For example, to log test run output with the default (tbpl) formatter to the
|
|
file `~/mochitest.log` at `DEBUG` level detail you would use:
|
|
|
|
```
|
|
./mach mochitest --log-tbpl=~/mochitest.log --log-tbpl-level=DEBUG
|
|
```
|
|
|
|
## Headless mode
|
|
|
|
The tests must run in a focused window, which effectively prevents any other
|
|
user activity on the engaged computer. You can avoid this by using the
|
|
`--headless` argument or `MOZ_HEADLESS=1` environment variable.
|
|
|
|
```
|
|
./mach mochitest --headless ...
|
|
```
|
|
|
|
## Writing tests
|
|
|
|
A Mochitest plain test is simply an HTML or XHTML file that contains some
|
|
JavaScript to test for some condition.
|
|
|
|
### Asynchronous Tests
|
|
|
|
Sometimes tests involve asynchronous patterns, such as waiting for events or
|
|
observers. In these cases, you need to use `add_task`:
|
|
|
|
```js
|
|
add_task(async function my_test() {
|
|
let keypress = new Promise(...);
|
|
// .. simulate keypress
|
|
await keypress;
|
|
// .. run test
|
|
});
|
|
```
|
|
|
|
Use `add_setup()` when asynchronous test task is meant to prepare test for run.
|
|
All setup tasks are executed once in order they appear prior to any test tasks.
|
|
|
|
```js
|
|
add_setup(async () => {
|
|
await clearStorage();
|
|
});
|
|
```
|
|
|
|
Or alternatively, manually call `waitForExplicitFinish` and `finish`:
|
|
|
|
```js
|
|
SimpleTest.waitForExplicitFinish();
|
|
addEventListener("keypress", function() {
|
|
// ... run test ...
|
|
SimpleTest.finish();
|
|
}, false);
|
|
// ... simulate key press ...
|
|
```
|
|
|
|
|
|
If you need more time, `requestLongerTimeout(number)` can be quite useful.
|
|
`requestLongerTimeout()` takes an integer factor that is a multiplier for the
|
|
default 45 seconds timeout. So a factor of 2 means: "Wait for at last 90s
|
|
(2*45s)". This is really useful if you want to pause execution to do a little
|
|
debugging.
|
|
|
|
### Test functions
|
|
|
|
Each test must contain some JavaScript that will run and tell Mochitest whether
|
|
the test has passed or failed. `SimpleTest.js` provides a number of functions
|
|
for the test to use, to communicate the results back to Mochitest. These
|
|
include:
|
|
|
|
|
|
* `ok(expressionThatShouldBeTrue, "Description of the check")` -- tests a value for its truthfulness
|
|
* `is(actualValue, expectedValue, "Description of the check")` -- compares two values (using Object.is)
|
|
* `isnot(actualValue, unexpectedValue, "Description of the check")` -- opposite of is()
|
|
|
|
If you want to include a test for something that currently fails, don't just
|
|
comment it out! Instead, use one of the "todo" equivalents so we notice if it
|
|
suddenly starts passing (at which point the test can be re-enabled):
|
|
|
|
* `todo(falseButShouldBeTrue, "Description of the check")`
|
|
* `todo_is(actualValue, expectedValue, "Description of the check")`
|
|
* `todo_isnot(actualValue, unexpectedValue, "Description of the check")`
|
|
|
|
Tests can call a function `info("Message string")` to write a message to the
|
|
test log.
|
|
|
|
In addition to mochitest assertions, mochitest supports the
|
|
[CommonJS standard assertions](http://wiki.commonjs.org/wiki/Unit_Testing/1.1),
|
|
like [nodejs' assert module](https://nodejs.org/api/assert.html#assert) but
|
|
implemented in `Assert.sys.mjs`. These are auto-imported in the browser flavor, but
|
|
need to be imported manually in other flavors.
|
|
|
|
### Helper functions
|
|
|
|
Right now, useful helpers derived from MochiKit are available in
|
|
[`testing/mochitest/tests/SimpleTest/SimpleTest.js`](https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js).
|
|
|
|
Although all of Mochikit is available at `testing/mochitest/MochiKit`, only
|
|
include files that you require to minimize test load times. Bug 367569 added
|
|
`sendChar`, `sendKey`, and `sendString` helpers.
|
|
These are available in [`testing/mochitest/tests/SimpleTest/EventUtils.js`](https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/EventUtils.js).
|
|
|
|
If you need to access some data files from your Mochitest, you can get an URI
|
|
for them by using `SimpleTest.getTestFileURL("relative/path/to/data.file")`.
|
|
Then you can eventually fetch their content by using `XMLHttpRequest` or so.
|
|
|
|
### Adding tests to the tree
|
|
|
|
`mach addtest` is the preferred way to add a test to the tree:
|
|
|
|
```
|
|
./mach addtest --suite mochitest-{plain,chrome,browser-chrome} path/to/new/test
|
|
```
|
|
|
|
That will add the manifest entry to the relevant manifest (`mochitest.ini`,
|
|
`chrome.ini`, etc. depending on the flavor) to tell the build system about your
|
|
new test, as well as creating the file based on a template.
|
|
|
|
```ini
|
|
[test_new_feature.html]
|
|
```
|
|
|
|
Optionally, you can specify metadata for your test, like whether to skip the
|
|
test on certain platforms:
|
|
|
|
```ini
|
|
[test_new_feature.html]
|
|
skip-if = os == 'win'
|
|
```
|
|
|
|
The [mochitest.ini format](/build/buildsystem/test_manifests.rst), which is
|
|
recognized by the parser, defines a long list of metadata.
|
|
|
|
### Adding a new mochitest.ini or chrome.ini file
|
|
|
|
If a `mochitest.ini` or `chrome.ini` file does not exist in the test directory
|
|
where you want to add a test, add them and update the moz.build file in the
|
|
directory for your test. For example, in `gfx/layers/moz.build`, we add
|
|
these two manifest files:
|
|
|
|
```python
|
|
MOCHITEST_MANIFESTS += ['apz/test/mochitest.ini']
|
|
MOCHITEST_CHROME_MANIFESTS += ['apz/test/chrome.ini']
|
|
```
|
|
|
|
<!-- TODO: This might be outdated.*
|
|
|
|
## Getting Stack Traces
|
|
|
|
|
|
To get stack when Mochitest crashes:
|
|
|
|
* Get a minidump_stackwalk binary for your platform from http://hg.mozilla.org/build/tools/file/tip/breakpad/
|
|
* Set the MINIDUMP_STACKWALK environment variable to point to the absolute path of the binary.
|
|
|
|
If the resulting stack trace doesn't have line numbers, run `mach buildsymbols`
|
|
to generate the requisite symbol files.
|
|
|
|
-->
|
|
|
|
## FAQ
|
|
|
|
See the [Mochitest FAQ page](faq.md) for other features and such that you may
|
|
want to use, such as SSL-enabled tests, custom http headers, async tests, leak
|
|
debugging, prefs...
|