diff options
Diffstat (limited to 'testing/docs/mochitest-plain/index.md')
| -rw-r--r-- | testing/docs/mochitest-plain/index.md | 301 |
1 files changed, 301 insertions, 0 deletions
diff --git a/testing/docs/mochitest-plain/index.md b/testing/docs/mochitest-plain/index.md new file mode 100644 index 0000000000..0bfce6a2fe --- /dev/null +++ b/testing/docs/mochitest-plain/index.md @@ -0,0 +1,301 @@ +# Mochitest + +## DISCLAIMER + +If you are testing web platform code, prefer using use a [wpt +test](/web-platform/index.rst) (preferably upstreamable ones). + +## Introduction + +Mochitest is an automated testing framework built on top of the +[MochiKit](https://mochi.github.io/mochikit/) JavaScript libraries. + +Only things that can be tested using JavaScript (with chrome privileges!) can be +tested with this framework. Given some creativity, that's actually much more +than you might first think, but it's not possible to write Mochitest tests to +directly test a non-scripted C++ component, for example. (Use a compiled-code +test like [GTest](/gtest/index.rst) to do that.) + +## Running tests + +To run a single test (perhaps a new test you just added) or a subset of the +entire Mochitest suite, pass a path parameter to the `mach` command. + +For example, to run only the test `test_CrossSiteXHR.html` in the Mozilla source +tree, you would run this command: + +``` +./mach test dom/security/test/cors/test_CrossSiteXHR.html +``` + +To run all the tests in `dom/svg/`, this command would work: + +``` +./mach test dom/svg/ +``` + +You can also pass a manifest path to run all tests on that manifest: + +``` +./mach test dom/base/test/mochitest.ini +``` + +## Running flavors and subsuites + +Flavors are variations of the default configuration used to run Mochitest. For +example, a flavor might have a slightly different set of prefs set for it, a +custom extension installed or even run in a completely different scope. + +The Mochitest flavors are: + + * **plain** - The most basic and common Mochitest. They run in content scope, + but can access certain privileged APIs with SpecialPowers. + + * **browser** - These often test the browser UI itself and run in browser + window scope. + + * **chrome** - These run in chrome scope and are typically used for testing + privileged JavaScript APIs. More information can be found + [here](../chrome-tests/index.rst). + + * **a11y** - These test the accessibility interfaces. They can be found under + the top `accessible` directory and run in chrome scope. Note that these run + without e10s / fission. + +A subsuite is similar to a flavor, except that it has an identical +configuration. It is just logically separated from the "default" subsuite for +display purposes. For example, devtools is a subsuite of the browser flavor. +There is no difference in how these two jobs are run. It exists so that the +devtools team can easily see and run their tests. + +**Note**: There are also tags, which are similar to subsuites. Although they +both are used to logically group related sets of tests, they behave +differently. For example, applying a subsuite to a test removes that test from +the default set, whereas, a tag does not remove it. + +By default, mach finds and runs every test in the given subdirectory no matter +which flavor or subsuite it belongs to. But sometimes, you might only want to +run a specific flavor or subsuite. This can be accomplished using the `--flavor` +(or `-f`) and `--subsuite` options respectively. For example: + + +``` +./mach mochitest -f plain # runs all plain tests +./mach mochitest -f browser --subsuite devtools # runs all browser tests in the devtools subsuite +./mach mochitest -f chrome dom/indexedDB # runs all chrome tests in the dom/indexedDB subdirectory +``` + +In many cases, it won't be necessary to filter by flavor or subsuite as running +specific directories will do it implicitly. For example running: + +``` +./mach mochitest devtools/ +``` + +Is a rough equivalent to running the `devtools` subsuite. There might be +situations where you might want to run tests that don't belong to any subsuite. +To do this, use: + +``` +./mach mochitest --subsuite default +``` + +## Debugging individual tests + +If you need to debug an individual test, you could reload the page containing +the test with the debugger attached. If attaching a debugger before the problem +shows up is hard (for example, if the browser crashes as the test is loading), +you can specify a debugger when you run mochitest: + +``` +./mach mochitest --debugger=gdb ... +``` + +See also the `--debugger-args` and `--debugger-interactive` arguments. You can +also use the `--jsdebugger` argument to debug JavaScript. + +## Finding errors + +Search for the string `TEST-UNEXPECTED-FAIL` to find unexpected failures. You +can also search for `SimpleTest FINISHED` to see the final test summary. +## Logging results + +The output from a test run can be sent to the console and/or a file (by default +the results are only displayed in the browser). There are several levels of +detail to choose from. The levels are `DEBUG`, `INFO`, `WARNING`, `ERROR` and +`CRITICAL`, where `DEBUG` produces the highest detail (everything), and +`CRITICAL` produces the least. + +Mochitest uses structured logging. This means that you can use a set of command +line arguments to configure the log output. To log to stdout using the mach +formatter and log to a file in JSON format, you can use `--log-mach=-` +`--log-raw=mochitest.log`. By default the file logging level for all your +formatters is `INFO` but you can change this using `--log-mach-level=<level>`. + +To turn on logging to the console use `--console-level=<level>`. + +For example, to log test run output with the default (tbpl) formatter to the +file `~/mochitest.log` at `DEBUG` level detail you would use: + +``` +./mach mochitest --log-tbpl=~/mochitest.log --log-tbpl-level=DEBUG +``` + +## Headless mode + +The tests must run in a focused window, which effectively prevents any other +user activity on the engaged computer. You can avoid this by using the +`--headless` argument or `MOZ_HEADLESS=1` environment variable. + +``` +./mach mochitest --headless ... +``` + +## Writing tests + +A Mochitest plain test is simply an HTML or XHTML file that contains some +JavaScript to test for some condition. + +### Asynchronous Tests + +Sometimes tests involve asynchronous patterns, such as waiting for events or +observers. In these cases, you need to use `add_task`: + +```js +add_task(async function my_test() { + let keypress = new Promise(...); + // .. simulate keypress + await keypress; + // .. run test +}); +``` + +Use `add_setup()` when asynchronous test task is meant to prepare test for run. +All setup tasks are executed once in order they appear prior to any test tasks. + +```js +add_setup(async () => { + await clearStorage(); +}); +``` + +Or alternatively, manually call `waitForExplicitFinish` and `finish`: + +```js +SimpleTest.waitForExplicitFinish(); +addEventListener("keypress", function() { + // ... run test ... + SimpleTest.finish(); +}, false); +// ... simulate key press ... +``` + + +If you need more time, `requestLongerTimeout(number)` can be quite useful. +`requestLongerTimeout()` takes an integer factor that is a multiplier for the +default 45 seconds timeout. So a factor of 2 means: "Wait for at last 90s +(2*45s)". This is really useful if you want to pause execution to do a little +debugging. + +### Test functions + +Each test must contain some JavaScript that will run and tell Mochitest whether +the test has passed or failed. `SimpleTest.js` provides a number of functions +for the test to use, to communicate the results back to Mochitest. These +include: + + + * `ok(expressionThatShouldBeTrue, "Description of the check")` -- tests a value for its truthfulness + * `is(actualValue, expectedValue, "Description of the check")` -- compares two values (using Object.is) + * `isnot(actualValue, unexpectedValue, "Description of the check")` -- opposite of is() + +If you want to include a test for something that currently fails, don't just +comment it out! Instead, use one of the "todo" equivalents so we notice if it +suddenly starts passing (at which point the test can be re-enabled): + + * `todo(falseButShouldBeTrue, "Description of the check")` + * `todo_is(actualValue, expectedValue, "Description of the check")` + * `todo_isnot(actualValue, unexpectedValue, "Description of the check")` + +Tests can call a function `info("Message string")` to write a message to the +test log. + +In addition to mochitest assertions, mochitest supports the +[CommonJS standard assertions](http://wiki.commonjs.org/wiki/Unit_Testing/1.1), +like [nodejs' assert module](https://nodejs.org/api/assert.html#assert) but +implemented in `Assert.sys.mjs`. These are auto-imported in the browser flavor, but +need to be imported manually in other flavors. + +### Helper functions + +Right now, useful helpers derived from MochiKit are available in +[`testing/mochitest/tests/SimpleTest/SimpleTest.js`](https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js). + +Although all of Mochikit is available at `testing/mochitest/MochiKit`, only +include files that you require to minimize test load times. Bug 367569 added +`sendChar`, `sendKey`, and `sendString` helpers. +These are available in [`testing/mochitest/tests/SimpleTest/EventUtils.js`](https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/EventUtils.js). + +If you need to access some data files from your Mochitest, you can get an URI +for them by using `SimpleTest.getTestFileURL("relative/path/to/data.file")`. +Then you can eventually fetch their content by using `XMLHttpRequest` or so. + +### Adding tests to the tree + +`mach addtest` is the preferred way to add a test to the tree: + +``` +./mach addtest --suite mochitest-{plain,chrome,browser-chrome} path/to/new/test +``` + +That will add the manifest entry to the relevant manifest (`mochitest.ini`, +`chrome.ini`, etc. depending on the flavor) to tell the build system about your +new test, as well as creating the file based on a template. + +```ini +[test_new_feature.html] +``` + +Optionally, you can specify metadata for your test, like whether to skip the +test on certain platforms: + +```ini +[test_new_feature.html] +skip-if = os == 'win' +``` + +The [mochitest.ini format](/build/buildsystem/test_manifests.rst), which is +recognized by the parser, defines a long list of metadata. + +### Adding a new mochitest.ini or chrome.ini file + +If a `mochitest.ini` or `chrome.ini` file does not exist in the test directory +where you want to add a test, add them and update the moz.build file in the +directory for your test. For example, in `gfx/layers/moz.build`, we add +these two manifest files: + +```python +MOCHITEST_MANIFESTS += ['apz/test/mochitest.ini'] +MOCHITEST_CHROME_MANIFESTS += ['apz/test/chrome.ini'] +``` + +<!-- TODO: This might be outdated.* + +## Getting Stack Traces + + +To get stack when Mochitest crashes: + + * Get a minidump_stackwalk binary for your platform from http://hg.mozilla.org/build/tools/file/tip/breakpad/ + * Set the MINIDUMP_STACKWALK environment variable to point to the absolute path of the binary. + +If the resulting stack trace doesn't have line numbers, run `mach buildsymbols` +to generate the requisite symbol files. + +--> + +## FAQ + +See the [Mochitest FAQ page](faq.md) for other features and such that you may +want to use, such as SSL-enabled tests, custom http headers, async tests, leak +debugging, prefs... |