98 lines
4.4 KiB
Markdown
98 lines
4.4 KiB
Markdown
# Testing with one-click loaners
|
|
|
|
[Taskcluster] is the task execution framework that supports Mozilla's
|
|
continuous integration and release processes.
|
|
|
|
Build and test jobs (like Marionette) are executed across all supported
|
|
platforms, and job results are pushed to [Treeherder] for observation.
|
|
|
|
The best way to debug issues for intermittent test failures of
|
|
Marionette tests for Firefox and Fennec (Android) is to use a
|
|
one-click loaner as provided by Taskcluster. Such a loaner creates
|
|
an interactive task you can interact with via a shell and VNC.
|
|
|
|
To create an interactive task for a Marionette job which is shown
|
|
as failed on Treeherder, select the job, click the ellipse in the lower
|
|
left pane, and choose `Create Interactive Task`.
|
|
|
|
Please note that you need special permissions to actually request
|
|
such a loaner.
|
|
|
|
When the task has been created you will receive an email with the connection
|
|
details. Open the referenced shell and you will be connected via a WebSocket.
|
|
Once that has been done a wizard will automatically launch and
|
|
provide some options. Best here is to choose the second option,
|
|
which will run all the setup steps, installs the Firefox or Fennec
|
|
binary, and then exits.
|
|
|
|
[Taskcluster]: https://docs.taskcluster.net/
|
|
[Treeherder]: https://treeherder.mozilla.org
|
|
|
|
## Setting up the Marionette environment
|
|
|
|
Best here is to use a virtual environment, which has all the
|
|
necessary packages installed. If no modifications to any Python
|
|
package will be done, the already created environment by the
|
|
wizard can be used:
|
|
|
|
```shell
|
|
% cd /builds/worker/workspace/build
|
|
% source venv/bin/activate
|
|
```
|
|
|
|
Otherwise a new virtual environment needs to be created and
|
|
populated with the mozbase and marionette packages installed:
|
|
|
|
```shell
|
|
% cd /builds/worker/workspace/build && rm -r venv
|
|
% virtualenv venv && source venv/bin/activate
|
|
% cd tests/mozbase && ./setup_development.py
|
|
% cd ../marionette/client && python setup.py develop
|
|
% cd ../harness && python setup.py develop
|
|
% cd ../../../
|
|
```
|
|
|
|
## Running Marionette tests
|
|
|
|
### Firefox
|
|
|
|
To run the Marionette tests execute the `runtests.py` script. For all
|
|
the required options as best search in the log file of the failing job
|
|
the interactive task has been created from. Then copy the complete
|
|
command and run it inside the already sourced virtual environment:
|
|
|
|
```shell
|
|
% /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --gecko-log=- -vv --binary=/builds/worker/workspace/build/application/firefox/firefox --address=127.0.0.1:2828 --symbols-path=https://queue.taskcluster.net/v1/task/GSuwee61Qyibujtxq4UV3A/artifacts/public/build/target.crashreporter-symbols.zip /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini
|
|
```
|
|
|
|
### Fennec
|
|
|
|
The Marionette tests for Fennec are executed by using an Android
|
|
emulator which runs on the host platform. As such some extra setup
|
|
steps compared to Firefox on desktop are required.
|
|
|
|
The following lines set necessary environment variables before
|
|
starting the emulator in the background, and to let Marionette
|
|
know of various Android SDK tools.
|
|
|
|
```shell
|
|
% export ADB_PATH=/builds/worker/workspace/build/android-sdk-linux/platform-tools/adb
|
|
% export ANDROID_AVD_HOME=/builds/worker/workspace/build/.android/avd/
|
|
% /builds/worker/workspace/build/android-sdk-linux/tools/emulator -avd test-1 -show-kernel -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config,socket &
|
|
```
|
|
|
|
The actual call to `runtests.py` is different per test job because
|
|
those are using chunks on Android. As best search for the command
|
|
and its options in the log file of the failing job the interactive
|
|
task has been created from. Then copy the complete command and run
|
|
it inside the already sourced virtual environment.
|
|
|
|
Here an example for chunk 1 which runs all the tests in the current
|
|
chunk with some options for logs removed:
|
|
|
|
```shell
|
|
% /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --emulator --app=fennec --package=org.mozilla.fennec_aurora --address=127.0.0.1:2828 /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini --gecko-log=- --symbols-path=/builds/worker/workspace/build/symbols --startup-timeout=300 --this-chunk 1 --total-chunks 10
|
|
```
|
|
|
|
To execute a specific test only simply replace `unit-tests.ini`
|
|
with its name.
|