1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
|
Testing
=======
We verify and test Marionette in a couple of different ways, using
a combination of unit tests and functional tests. There are three
distinct components that we test:
- the Marionette **server**, using a combination of xpcshell
unit tests and functional tests written in Python spread across
Marionette- and WPT tests;
- the Python **client** is tested with the same body of functional
Marionette tests;
- and the **harness** that backs the Marionette, or `Mn` job on
try, tests is verified using separate mock-styled unit tests.
All these tests can be run by using [mach].
[mach]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach
xpcshell unit tests
-------------------
Marionette has a set of [xpcshell] unit tests located in
_testing/marionette/test/unit. These can be run this way:
% ./mach test testing/marionette/test/unit
Because tests are run in parallel and xpcshell itself is quite
chatty, it can sometimes be useful to run the tests sequentially:
% ./mach test --sequential testing/marionette/test/unit/test_error.js
These unit tests run as part of the `X` jobs on Treeherder.
[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
Marionette functional tests
---------------------------
We also have a set of [functional tests] that make use of the Marionette
Python client. These start a Firefox process and tests the Marionette
protocol input and output, and will appear as `Mn` on Treeherder.
The following command will run all tests locally:
% ./mach marionette-test
But you can also run individual tests:
% ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
In case you want to run the tests with another binary like [Firefox Nightly]:
% ./mach marionette-test --binary /path/to/nightly/firefox TEST
When working on Marionette it is often useful to surface the stdout
from Gecko, which can be achieved using the `--gecko-log` option.
See <Debugging.html> for usage instructions, but the gist is that
you can redirect all Gecko output to stdout:
% ./mach marionette-test --gecko-log - TEST
Our functional integration tests pop up Firefox windows sporadically,
and a helpful tip is to suppress the window can be to use Firefox’
[headless mode]:
% ./mach marionette-test -z TEST
`-z` is an alias for the `--headless` flag and equivalent to setting
the `MOZ_HEADLESS` output variable. In addition to `MOZ_HEADLESS`
there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
controlling the dimensions of the no-op virtual display. This is
similar to using Xvfb(1) which you may know from the X windowing system,
but has the additional benefit of also working on macOS and Windows.
[functional tests]: PythonTests.html
[Firefox Nightly]: https://nightly.mozilla.org/
### Android
Prerequisites:
* You have [built Fennec](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_for_Android_build).
* You can run an Android [emulator](https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#Running_tests_on_the_Android_emulator),
which means you have the AVD you need.
When running tests on Fennec, you can have Marionette runner take care of
starting Fennec and an emulator, as shown below.
% ./mach marionette-test --emulator --app fennec
--avd-home /path/to/.mozbuild/android-device/avd
--emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator
--avd=mozemulator-x86
For Fennec tests, if the appropriate `emulator` command is in your `PATH`, you may omit the `--emulator-binary` argument. See `./mach marionette-test -h`
for additional options.
Alternately, you can start an emulator yourself and have the Marionette runner
start Fennec for you:
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828
To connect to an already-running Fennec in an already running emulator or on a device, you will need to enable Marionette manually by setting the browser preference
`marionette.enabled` set to true in the Fennec profile.
Make sure port 2828 is forwarded:
% adb forward tcp:2828 tcp:2828
If Fennec is already started:
% ./mach marionette-test --app='fennec' --address=127.0.0.1:2828
If Fennec is not already started on the emulator/device, add the `--emulator`
option. Marionette Test Runner will take care of forwarding the port and
starting Fennec with the correct prefs. (You may need to run
`adb forward --remove-all` to allow the runner to start.)
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --startup-timeout=300
If you need to troubleshoot the Marionette connection, the most basic check is
to start Fennec, make sure the `marionette.enabled` browser preference is
true and port 2828 is forwarded, then see if you get any response from
Marionette when you connect manually:
% telnet 127.0.0.1:2828
You should see output like `{"applicationType":"gecko","marionetteProtocol":3}`
[headless mode]: https://developer.mozilla.org/en-US/Firefox/Headless_mode
[geckodriver]: /testing/geckodriver/geckodriver
WPT functional tests
--------------------
Marionette is also indirectly tested through [geckodriver] with WPT
(`Wd` on Treeherder). To run them:
% ./mach wpt testing/web-platform/tests/webdriver
WPT tests conformance to the [WebDriver] standard and uses
[geckodriver]. Together with the Marionette remote protocol in
Gecko, they make up Mozilla’s WebDriver implementation.
This command supports a `--webdriver-arg='-vv'` argument that
enables more detailed logging, as well as `--jsdebugger` for opening
the Browser Toolbox.
A particularly useful trick is to combine this with the [headless
mode] for Firefox we learned about earlier:
% MOZ_HEADLESS=1 ./mach wpt --webdriver-arg='-vv' testing/web-platform/tests/webdriver
[WebDriver]: https://w3c.github.io/webdriver/webdriver-spec.html
Harness tests
-------------
The Marionette harness Python package has a set of mock-styled unit
tests that uses the [pytest] framework. The following command will
run all tests:
% ./mach python-test testing/marionette
To run a specific test specify the full path to the module:
% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
[pytest]: https://docs.pytest.org/en/latest/
One-click loaners
-----------------
Additionally, for debugging hard-to-reproduce test failures in CI,
one-click loaners from <Taskcluster.html> can be particularly useful.
Out-of-tree testing
-------------------
All the above examples show tests running _in-tree_, with a local
checkout of _central_ and a local build of Firefox. It is also
possibly to run the Marionette tests _without_ a local build and
with a downloaded test archive from <Taskcluster.html>.
If you want to run tests from a downloaded test archive, you will
need to download the `target.common.tests.tar.gz` artifact attached to
Treeherder [build jobs] `B` for your system. Extract the archive
and set up the Python Marionette client and harness by executing
the following command in a virtual environment:
% pip install -r config/marionette_requirements.txt
The tests can then be found under
_marionette/tests/testing/marionette/harness/marionette_harness/tests_
and can be executed with the command `marionette`. It supports
the same options as described above for `mach`.
[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build
|
