diff options
Diffstat (limited to 'remote/doc/marionette/Taskcluster.md')
| -rw-r--r-- | remote/doc/marionette/Taskcluster.md | 98 |
1 files changed, 98 insertions, 0 deletions
diff --git a/remote/doc/marionette/Taskcluster.md b/remote/doc/marionette/Taskcluster.md new file mode 100644 index 0000000000..11381745aa --- /dev/null +++ b/remote/doc/marionette/Taskcluster.md @@ -0,0 +1,98 @@ +# Testing with one-click loaners + +[Taskcluster] is the task execution framework that supports Mozilla's +continuous integration and release processes. + +Build and test jobs (like Marionette) are executed across all supported +platforms, and job results are pushed to [Treeherder] for observation. + +The best way to debug issues for intermittent test failures of +Marionette tests for Firefox and Fennec (Android) is to use a +one-click loaner as provided by Taskcluster. Such a loaner creates +an interactive task you can interact with via a shell and VNC. + +To create an interactive task for a Marionette job which is shown +as failed on Treeherder, select the job, click the ellipse in the lower +left pane, and choose `Create Interactive Task`. + +Please note that you need special permissions to actually request +such a loaner. + +When the task has been created you will receive an email with the connection +details. Open the referenced shell and you will be connected via a WebSocket. +Once that has been done a wizard will automatically launch and +provide some options. Best here is to choose the second option, +which will run all the setup steps, installs the Firefox or Fennec +binary, and then exits. + +[Taskcluster]: https://docs.taskcluster.net/ +[Treeherder]: https://treeherder.mozilla.org + +## Setting up the Marionette environment + +Best here is to use a virtual environment, which has all the +necessary packages installed. If no modifications to any Python +package will be done, the already created environment by the +wizard can be used: + +```shell +% cd /builds/worker/workspace/build +% source venv/bin/activate +``` + +Otherwise a new virtual environment needs to be created and +populated with the mozbase and marionette packages installed: + +```shell +% cd /builds/worker/workspace/build && rm -r venv +% virtualenv venv && source venv/bin/activate +% cd tests/mozbase && ./setup_development.py +% cd ../marionette/client && python setup.py develop +% cd ../harness && python setup.py develop +% cd ../../../ +``` + +## Running Marionette tests + +### Firefox + +To run the Marionette tests execute the `runtests.py` script. For all +the required options as best search in the log file of the failing job +the interactive task has been created from. Then copy the complete +command and run it inside the already sourced virtual environment: + +```shell +% /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --gecko-log=- -vv --binary=/builds/worker/workspace/build/application/firefox/firefox --address=127.0.0.1:2828 --symbols-path=https://queue.taskcluster.net/v1/task/GSuwee61Qyibujtxq4UV3A/artifacts/public/build/target.crashreporter-symbols.zip /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini +``` + +### Fennec + +The Marionette tests for Fennec are executed by using an Android +emulator which runs on the host platform. As such some extra setup +steps compared to Firefox on desktop are required. + +The following lines set necessary environment variables before +starting the emulator in the background, and to let Marionette +know of various Android SDK tools. + +```shell +% export ADB_PATH=/builds/worker/workspace/build/android-sdk-linux/platform-tools/adb +% export ANDROID_AVD_HOME=/builds/worker/workspace/build/.android/avd/ +% /builds/worker/workspace/build/android-sdk-linux/tools/emulator -avd test-1 -show-kernel -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config,socket & +``` + +The actual call to `runtests.py` is different per test job because +those are using chunks on Android. As best search for the command +and its options in the log file of the failing job the interactive +task has been created from. Then copy the complete command and run +it inside the already sourced virtual environment. + +Here an example for chunk 1 which runs all the tests in the current +chunk with some options for logs removed: + +```shell +% /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --emulator --app=fennec --package=org.mozilla.fennec_aurora --address=127.0.0.1:2828 /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini --gecko-log=- --symbols-path=/builds/worker/workspace/build/symbols --startup-timeout=300 --this-chunk 1 --total-chunks 10 +``` + +To execute a specific test only simply replace `unit-tests.ini` +with its name. |