diff options
Diffstat (limited to 'remote/doc/marionette')
| -rw-r--r-- | remote/doc/marionette/Building.md | 56 | ||||
| -rw-r--r-- | remote/doc/marionette/CodeStyle.md | 229 | ||||
| -rw-r--r-- | remote/doc/marionette/Contributing.md | 69 | ||||
| -rw-r--r-- | remote/doc/marionette/Debugging.md | 86 | ||||
| -rw-r--r-- | remote/doc/marionette/Intro.md | 70 | ||||
| -rw-r--r-- | remote/doc/marionette/NewContributors.md | 84 | ||||
| -rw-r--r-- | remote/doc/marionette/Patches.md | 32 | ||||
| -rw-r--r-- | remote/doc/marionette/Prefs.md | 24 | ||||
| -rw-r--r-- | remote/doc/marionette/Protocol.md | 114 | ||||
| -rw-r--r-- | remote/doc/marionette/PythonTests.md | 69 | ||||
| -rw-r--r-- | remote/doc/marionette/SeleniumAtoms.md | 90 | ||||
| -rw-r--r-- | remote/doc/marionette/Taskcluster.md | 88 | ||||
| -rw-r--r-- | remote/doc/marionette/Testing.md | 192 | ||||
| -rw-r--r-- | remote/doc/marionette/index.rst | 64 |
14 files changed, 1267 insertions, 0 deletions
diff --git a/remote/doc/marionette/Building.md b/remote/doc/marionette/Building.md new file mode 100644 index 0000000000..1f55697d2e --- /dev/null +++ b/remote/doc/marionette/Building.md @@ -0,0 +1,56 @@ +# Building + +Marionette is built into Firefox by default and ships in the official +Firefox binary. As Marionette is written in [XPCOM] flavoured +JavaScript, you may choose to rely on so called [artifact builds], +which will download pre-compiled Firefox blobs to your computer. +This means you don’t have to compile Firefox locally, but does +come at the cost of having a good internet connection. To enable +[artifact builds] you may choose ‘Firefox for Desktop Artifact +Mode’ when bootstrapping. + +Once you have a clone of [mozilla-unified], you can set up your +development environment by running this command and following the +on-screen instructions: + + % ./mach bootstrap + +When you're getting asked to choose the version of Firefox you want to build, +you may want to consider choosing "Firefox for Desktop Artifact Mode". This +significantly reduces the time it takes to build Firefox on your machine +(from 30+ minutes to just 1-2 minutes) if you have a fast internet connection. + +To perform a regular build, simply do: + + % ./mach build + +You can clean out the objdir using this command: + + % ./mach clobber + +Occasionally a clean build will be required after you fetch the +latest changes from mozilla-central. You will find that the the +build will error when this is the case. To automatically do clean +builds when this happens you may optionally add this line to the +[mozconfig] file in your top source directory: + + mk_add_options AUTOCLOBBER=1 + +If you compile Firefox frequently you will also want to enable +[ccache] and [sccache] if you develop on a macOS or Linux system: + + mk_add_options 'export RUSTC_WRAPPER=sccache' + mk_add_options 'export CCACHE_CPP2=yes' + ac_add_options --with-ccache + +You may also opt out of building all the WebDriver specific components +(Marionette, and the [Remote Agent]) by setting the following flag: + + ac_add_options --disable-webdriver + +[mozilla-unified]: https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/unifiedrepo.html +[artifact builds]: /contributing/build/artifact_builds.rst +[mozconfig]: /build/buildsystem/mozconfigs.rst +[ccache]: https://ccache.samba.org/ +[sccache]: https://github.com/mozilla/sccache +[Remote Agent]: /remote/index.rst diff --git a/remote/doc/marionette/CodeStyle.md b/remote/doc/marionette/CodeStyle.md new file mode 100644 index 0000000000..ad6f03092f --- /dev/null +++ b/remote/doc/marionette/CodeStyle.md @@ -0,0 +1,229 @@ +# Style guide + +Like other projects, we also have some guidelines to keep to the code. +For the overall Marionette project, a few rough rules are: + +* Make your code readable and sensible, and don’t try to be + clever. Prefer simple and easy solutions over more convoluted + and foreign syntax. + +* Fixing style violations whilst working on a real change as a + preparatory clean-up step is good, but otherwise avoid useless + code churn for the sake of conforming to the style guide. + +* Code is mutable and not written in stone. Nothing that + is checked in is sacred and we encourage change to make + remote/marionette a pleasant ecosystem to work in. + +## JavaScript + +Marionette is written in JavaScript and ships +as part of Firefox. We have access to all the latest ECMAScript +features currently in development, usually before it ships in the +wild and we try to make use of new features when appropriate, +especially when they move us off legacy internal replacements +(such as Promise.jsm and Task.jsm). + +One of the peculiarities of working on JavaScript code that ships as +part of a runtime platform is, that unlike in a regular web document, +we share a single global state with the rest of Firefox. This means +we have to be responsible and not leak resources unnecessarily. + +JS code in Gecko is organised into _modules_ carrying _.js_ or _.jsm_ +file extensions. Depending on the area of Gecko you’re working on, +you may find they have different techniques for exporting symbols, +varying indentation and code style, as well as varying linting +requirements. + +To export symbols to other Marionette modules, remember to assign +your exported symbols to the shared global `this`: + + const EXPORTED_SYMBOLS = ["PollPromise", "TimedPromise"]; + +When importing symbols in Marionette code, try to be specific about +what you need: + + const { TimedPromise } = ChromeUtils.import( + "chrome://remote/content/marionette/sync.js" + ); + +We prefer object assignment shorthands when redefining names, +for example when you use functionality from the `Components` global: + + const {classes: Cc, interfaces: Ci, utils: Cu, results: Cr} = Components; + +When using symbols by their own name, the assignment name can be +omitted: + + const {TYPE_ONE_SHOT, TYPE_REPEATING_SLACK} = Ci.nsITimer; + +In addition to the default [Mozilla eslint rules], we have [our +own specialisations] that are stricter and enforce more security. +A few notable examples are that we disallow fallthrough `case` +statements unless they are explicitly grouped together: + + switch (x) { + case "foo": + doSomething(); + + case "bar": // <-- disallowed! + doSomethingElse(); + break; + + case "baz": + case "bah": // <-- allowed (-: + doCrazyThings(); + } + +We disallow the use of `var`, for which we always prefer `let` and +`const` as replacements. Do be aware that `const` does not mean +that the variable is immutable: just that it cannot be reassigned. +We require all lines to end with semicolons, disallow construction +of plain `new Object()`, require variable names to be camel-cased, +and complain about unused variables. + +For purely aesthetic reasons we indent our code with two spaces, +which includes switch-statement `case`s, and limit the maximum +line length to 78 columns. When you need to wrap a statement to +the next line, the second line is indented with four spaces, like this: + + throw new TypeError( + "Expected an element or WindowProxy, " + + pprint`got: ${el}`); + +This is not normally something you have to think to deeply about as +it is enforced by the [linter]. The linter also has an automatic +mode that fixes and formats certain classes of style violations. + +If you find yourself struggling to fit a long statement on one line, +this is usually an indication that it is too long and should be +split into multiple lines. This is also a helpful tip to make the +code easier to read. Assigning transitive values to descriptive +variable names can serve as self-documentation: + + let location = event.target.documentURI || event.target.location.href; + log.debug(`Received DOM event ${event.type} for ${location}`); + +On the topic of variable naming the opinions are as many as programmers +writing code, but it is often helpful to keep the input and output +arguments to functions descriptive (longer), and let transitive +internal values to be described more succinctly: + + /** Prettifies instance of Error and its stacktrace to a string. */ + function stringify(error) { + try { + let s = error.toString(); + if ("stack" in error) { + s += "\n" + error.stack; + } + return s; + } catch (e) { + return "<unprintable error>"; + } + } + +When we can, we try to extract the relevant object properties in +the arguments to an event handler or a function: + + const responseListener = ({name, target, json, data}) => { … }; + +Instead of: + + const responseListener = msg => { + let name = msg.name; + let target = msg.target; + let json = msg.json; + let data = msg.data; + … + }; + +All source files should have `"use strict";` as the first directive +so that the file is parsed in [strict mode]. + +Every source code file that ships as part of the Firefox bundle +must also have a [copying header], such as this: + + /* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this file, + * You can obtain one at http://mozilla.org/MPL/2.0/. */ + +New xpcshell test files _should not_ have a license header as all +new Mozilla tests should be in the [public domain] so that they can +easily be shared with other browser vendors. We want to re-license +existing tests covered by the [MPL] so that they can be shared. +We very much welcome your help in doing version control archeology +to make this happen! + +The practical details of working on the Marionette code is outlined +in [Contributing.md], but generally you do not have to re-build +Firefox when changing code. Any change to remote/marionette/*.js +will be picked up on restarting Firefox. The only notable exception +is remote/components/Marionette.jsm, which does require +a re-build. + +[strict mode]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Strict_mode +[Mozilla eslint rules]: https://searchfox.org/mozilla-central/source/.eslintrc.js +[our own specialisations]: https://searchfox.org/mozilla-central/source/remote/marionette/.eslintrc.js +[linter]: #linting +[copying header]: https://www.mozilla.org/en-US/MPL/headers/ +[public domain]: https://creativecommons.org/publicdomain/zero/1.0/ +[MPL]: https://www.mozilla.org/en-US/MPL/2.0/ +[Contributing.md]: ./Contributing.md + +## Python + +TODO + +## Documentation + +We keep our documentation in-tree under [remote/doc/marionette] +and [testing/geckodriver/doc]. Updates and minor changes to +documentation should ideally not be scrutinised to the same degree +as code changes to encourage frequent updates so that the documentation +does not go stale. To that end, documentation changes with `r=me` +from module peers are permitted. + +Use fmt(1) or an equivalent editor specific mechanism (such as Meta-Q +in Emacs) to format paragraphs at a maximum width of 75 columns +with a goal of roughly 65. This is equivalent to `fmt -w 75 -g 65`, +which happens to be the default on BSD and macOS. + +We endeavour to document all _public APIs_ of the Marionette component. +These include public functions—or command implementations—on +the `GeckoDriver` class, as well as all exported symbols from +other modules. Documentation for non-exported symbols is not required. + +[remote/doc/marionette]: https://searchfox.org/mozilla-central/source/remote/marionette/doc +[testing/geckodriver/doc]: https://searchfox.org/mozilla-central/source/testing/geckodriver/doc + +## Linting + +Marionette consists mostly of JavaScript (server) and Python (client, +harness, test runner) code. We lint our code with [mozlint], +which harmonises the output from [eslint] and [flake8]. + +To run the linter with a sensible output: + + % ./mach lint -funix remote/marionette + +For certain classes of style violations the eslint linter has +an automatic mode for fixing and formatting your code. This is +particularly useful to keep to whitespace and indentation rules: + + % ./mach eslint --fix remote/marionette + +The linter is also run as a try job (shorthand `ES`) which means +any style violations will automatically block a patch from landing +(if using Autoland) or cause your changeset to be backed out (if +landing directly on mozilla-inbound). + +If you use git(1) you can [enable automatic linting] before you push +to a remote through a pre-push (or pre-commit) hook. This will +run the linters on the changed files before a push and abort if +there are any problems. This is convenient for avoiding a try run +failing due to a stupid linting issue. + +[mozlint]: /code-quality/lint/mozlint.rst +[eslint]: /code-quality/lint/linters/eslint.rst +[flake8]: /code-quality/lint/linters/flake8.rst +[enable automatic linting]: /code-quality/lint/usage.html#using-a-vcs-hook diff --git a/remote/doc/marionette/Contributing.md b/remote/doc/marionette/Contributing.md new file mode 100644 index 0000000000..13fe137905 --- /dev/null +++ b/remote/doc/marionette/Contributing.md @@ -0,0 +1,69 @@ +# Contributing + +If you are new to open source or to Mozilla, you might like this +[tutorial for new Marionette contributors](NewContributors.md). + +We are delighted that you want to help improve Marionette! +‘Marionette’ means different a few different things, depending +on who you talk to, but the overall scope of the project involves +these components: + +* [_Marionette_] is a Firefox remote protocol to communicate with, + instrument, and control Gecko-based applications such as Firefox + and Firefox for mobile. It is built in to the application and + written in JavaScript. + + It serves as the backend for the geckodriver WebDriver implementation, + and is used in the context of Firefox UI tests, reftesting, + Web Platform Tests, test harness bootstrapping, and in many + other far-reaching places where browser instrumentation is required. + +* [_geckodriver_] provides the HTTP API described by the [WebDriver + protocol] to communicate with Gecko-based applications such as + Firefox and Firefox for mobile. It is a standalone executable + written in Rust, and can be used with compatible W3C WebDriver clients. + +* [_webdriver_] is a Rust crate providing interfaces, traits + and types, errors, type- and bounds checks, and JSON marshaling + for correctly parsing and emitting the [WebDriver protocol]. + +By participating in this project, you agree to abide by the Mozilla +[Community Participation Guidelines]. Here are some guidelines +for contributing high-quality and actionable bugs and code. + +[_Marionette_]: ./index.rst +[_geckodriver_]: /testing/geckodriver/index.rst +[_webdriver_]: https://searchfox.org/mozilla-central/source/testing/webdriver/README.md +[WebDriver protocol]: https://w3c.github.io/webdriver/webdriver-spec.html#protocol +[Community Participation Guidelines]: https://www.mozilla.org/en-US/about/governance/policies/participation/ + +## Writing code + +Because there are many moving parts involved remote controlling +a web browser, it can be challenging to a new contributor to know +where to start. Please don’t hesitate to [ask questions]! + +The canonical source code repository is [mozilla-central]. Bugs are +filed in the [Testing :: Marionette] component on Bugzilla. We also +have a curated set of [good first bugs] you may consider attempting first. + +We have collected a lot of good advice for working on Marionette +code in our [code style document], which we highly recommend you read. + +[ask questions]: ./index.rst#communication +[mozilla-central]: https://searchfox.org/mozilla-central/source/remote/marionette/ +[Testing :: Marionette]: https://bugzilla.mozilla.org/buglist.cgi?resolution=---&component=Marionette +[good first bugs]: https://codetribute.mozilla.org/projects/automation?project%3DMarionette +[code style document]: CodeStyle.md + +## Next steps + +* [Building](Building.md) +* [Debugging](Debugging.md) +* [Testing](Testing.md) +* [Patching](Patches.md) + +## Other resources + +* [Code style](CodeStyle.md) +* [New Contributor Tutorial](NewContributors.md) diff --git a/remote/doc/marionette/Debugging.md b/remote/doc/marionette/Debugging.md new file mode 100644 index 0000000000..682d7adec5 --- /dev/null +++ b/remote/doc/marionette/Debugging.md @@ -0,0 +1,86 @@ +# Debugging + +## Redirecting the Gecko output + +The most common way to debug Marionette, as well as chrome code in +general, is to use `dump()` to print a string to stdout. In Firefox, +this log output normally ends up in the gecko.log file in your current +working directory. With Fennec it can be inspected using `adb logcat`. + +`mach marionette-test` takes a `--gecko-log` option which lets +you redirect this output stream. This is convenient if you want to +“merge” the test harness output with the stdout from the browser. +Per Unix conventions you can use `-` (dash) to have Firefox write +its log to stdout instead of file: + + % ./mach marionette-test --gecko-log - + +It is common to use this in conjunction with an option to increase +the Marionette log level: + + % ./mach test --gecko-log - -vv TEST + +A single `-v` enables debug logging, and a double `-vv` enables +trace logging. + +This debugging technique can be particularly effective when combined +with using [pdb] in the Python client or the JS remote debugger +that is described below. + +[pdb]: https://docs.python.org/3/library/pdb.html + +## JavaScript debugger + +You can attach the [Browser Toolbox] JavaScript debugger to the +Marionette server using the `--jsdebugger` flag. This enables you +to introspect and set breakpoints in Gecko chrome code, which is a +more powerful debugging technique than using `dump()` or `console.log()`. + +To automatically open the JS debugger for `Mn` tests: + + % ./mach marionette-test --jsdebugger + +It will prompt you when to start to allow you time to set your +breakpoints. It will also prompt you between each test. + +You can also use the `debugger;` statement anywhere in chrome code +to add a breakpoint. In this example, a breakpoint will be added +whenever the `WebDriver:GetPageSource` command is called: + + GeckoDriver.prototype.getPageSource = async function() { + debugger; + … + } + +To be prompted at the start of the test run or between tests, +you can set the `marionette.debugging.clicktostart` preference to +`true` this way: + + % ./mach marionette-test --setpref='marionette.debugging.clicktostart=true' --jsdebugger + +For reference, below is the list of preferences that enables the +chrome debugger for Marionette. These are all set implicitly when +`--jsdebugger` is passed to mach. In non-official builds, which +are the default when built using `./mach build`, you will find that +the chrome debugger won’t prompt for connection and will allow +remote connections. + +* `devtools.browsertoolbox.panel` -> `jsdebugger` + + Selects the Debugger panel by default. + +* `devtools.chrome.enabled` → true + + Enables debugging of chrome code. + +* `devtools.debugger.prompt-connection` → false + + Controls the remote connection prompt. Note that this will + automatically expose your Firefox instance to localhost. + +* `devtools.debugger.remote-enabled` → true + + Allows a remote debugger to connect, which is necessary for + debugging chrome code. + +[Browser Toolbox]: /devtools-user/browser_toolbox/index.rst diff --git a/remote/doc/marionette/Intro.md b/remote/doc/marionette/Intro.md new file mode 100644 index 0000000000..8c34eef700 --- /dev/null +++ b/remote/doc/marionette/Intro.md @@ -0,0 +1,70 @@ +# Introduction to Marionette + +Marionette is an automation driver for Mozilla's Gecko engine. +It can remotely control either the UI or the internal JavaScript of +a Gecko platform, such as Firefox. It can control both the chrome +(i.e. menus and functions) or the content (the webpage loaded inside +the browsing context), giving a high level of control and ability +to replicate user actions. In addition to performing actions on the +browser, Marionette can also read the properties and attributes of +the DOM. + +If this sounds similar to [Selenium/WebDriver] then you're +correct! Marionette shares much of the same ethos and API as +Selenium/WebDriver, with additional commands to interact with +Gecko's chrome interface. Its goal is to replicate what Selenium +does for web content: to enable the tester to have the ability to +send commands to remotely control a user agent. + +[Selenium/WebDriver]: https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html + +## How does it work? + +Marionette consists of two parts: a server which takes requests and +executes them in Gecko, and a client. The client sends commands to +the server and the server executes the command inside the browser. + +## When would I use it? + +If you want to perform UI tests with browser chrome or content, +Marionette is the tool you're looking for! You can use it to +control either web content, or Firefox itself. + +A test engineer would typically import the Marionette client package +into their test framework, import the classes and use the class +functions and methods to control the browser. After controlling +the browser, Marionette can be used to return information about +the state of the browser which can then be used to validate that +the action was performed correctly. + +## Using Marionette + +Marionette combines a gecko component (the Marionette server) with an +outside component (the Marionette client), which drives the tests. +The Marionette server ships with Firefox, and to use it you will +need to download a Marionette client or use the in-tree client. + +* [Download and setup the Python client for Marionette][1] +* [Run Tests with Python][2] – How to run tests using the + Python client +* You might want to experiment with [using Marionette interactively + at a Python command prompt][2] +* Start [writing and running][3] tests +* Tips on [debugging][4] Marionette code +* [Download and setup the Marionette JS client][5] +* [Protocol definition][6] + +[1]: /python/marionette_driver.rst +[2]: /python/marionette_driver.rst +[3]: PythonTests.md +[4]: Debugging.md +[5]: https://github.com/mozilla-b2g/marionette_js_client +[6]: Protocol.md + +## Bugs + +Please file any bugs you may find in the `Testing :: Marionette` +component in Bugzilla. You can view a [list of current bugs] +to see if your problem is already being addressed. + +[list of current bugs]: https://bugzilla.mozilla.org/buglist.cgi?product=Testing&component=Marionette diff --git a/remote/doc/marionette/NewContributors.md b/remote/doc/marionette/NewContributors.md new file mode 100644 index 0000000000..06051090f4 --- /dev/null +++ b/remote/doc/marionette/NewContributors.md @@ -0,0 +1,84 @@ +# New contributors + +This page is aimed at people who are new to Mozilla and want to contribute +to Mozilla source code related to Marionette Python tests, WebDriver +spec tests and related test harnesses and tools. Mozilla has both +git and Mercurial repositories, but this guide only describes Mercurial. + +If you run into issues or have doubts, check out the [Resources](#resources) +section below and **don't hesitate to ask questions**. :) The goal of these +steps is to make sure you have the basics of your development environment +working. Once you do, we can get you started with working on an +actual bug, yay! + +## Accounts, communication + + 1. Set up a [Bugzilla] account (and, if you like, a [Mozillians] profile). + Please include your Element nickname in both of these accounts so we can work + with you more easily. For example, Eve Smith would set the Bugzilla name + to "Eve Smith (:esmith)", where "esmith" is the Element nick. + + 2. For a direct communication with us it will be beneficial to setup [Element]. + Make sure to also register your nickname as described in the linked document. + + 3. Join our [#webdriver:mozilla.org] channel, and introduce yourself to the + team. :whimboo, :jdescottes, and :jgraham are all familiar with Marionette. + We're nice, I promise, but we might not answer right away due to different + time zones, time off, etc. So please be patient. + + 4. When you want to ask a question on Element, just go ahead an ask it even if + no one appears to be around/responding. + Provide lots of detail so that we have a better chance of helping you. + If you don't get an answer right away, check again in a few hours -- + someone may have answered you in the mean time. + + 5. If you're having trouble reaching us over Element, you are welcome to send an + email to our [mailing list](index.html#communication) instead. It's a good + idea to include your Element nick in your email message. + +[Element]: https://chat.mozilla.org +[#webdriver:mozilla.org]: https://chat.mozilla.org/#/room/#webdriver:mozilla.org +[Bugzilla]: https://bugzilla.mozilla.org/ +[Mozillians]: https://mozillians.org/ + +## Getting the code, running tests + +Follow the documentation on [Contributing](Contributing.md) to get a sense of +our projects, and which is of most interest for you. You will also learn how to +get the Firefox source code, build your custom Firefox build, and how to run the +tests. + +## Work on bugs and get code review + +Once you are familiar with the code of the test harnesses, and the tests you might +want to start with your first contribution. The necessary steps to submit and verify +your patches are laid out in [Patches](Patches.md). + +## Resources + +* Search Mozilla's code repository with searchfox to find the [code for + Marionette] and the [Marionette client/harness]. + +* Another [guide for new contributors]. It has not been updated in a long + time but it's a good general resource if you ever get stuck on something. + The most relevant sections to you are about Bugzilla, Mercurial, Python and the + Development Process. + +* [Mercurial for Mozillians] + +* More general resources are available in this little [guide] :maja_zf wrote + in 2015 to help a student get started with open source contributions. + +* Textbook about general open source practices: [Practical Open Source Software Exploration] + +* If you'd rather use git instead of hg, see [git workflow for + Gecko development] and/or [this blog post by :ato]. + +[code for Marionette]: https://searchfox.org/mozilla-central/source/remote/marionette/ +[Marionette client/harness]: https://searchfox.org/mozilla-central/source/testing/marionette/ +[guide for new contributors]: https://ateam-bootcamp.readthedocs.org/en/latest/guide/index.html#new-contributor-guide +[Mercurial for Mozillians]: https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html +[guide]: https://gist.github.com/mjzffr/d2adef328a416081f543 +[Practical Open Source Software Exploration]: https://quaid.fedorapeople.org/TOS/Practical_Open_Source_Software_Exploration/html/index.html +[git workflow for Gecko development]: https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development +[this blog post by :ato]: https://sny.no/2016/03/geckogit diff --git a/remote/doc/marionette/Patches.md b/remote/doc/marionette/Patches.md new file mode 100644 index 0000000000..acccb7c45f --- /dev/null +++ b/remote/doc/marionette/Patches.md @@ -0,0 +1,32 @@ +# Submitting patches + +You can submit patches by using [Phabricator]. Walk through its documentation +in how to set it up, and uploading patches for review. Don't worry about which +person to select for reviewing your code. It will be done automatically. + +Please also make sure to follow the [commit creation guidelines]. + +Once you have contributed a couple of patches, we are happy to +sponsor you in [becoming a Mozilla committer]. When you have been +granted commit access level 1 you will have permission to use the +[Firefox CI] to trigger your own “try runs” to test your changes. + +You can use the `remote-protocol` [try preset]: + + mach try --preset remote-protocol + +This preset will schedule tests related to the Remote Protocol component on +various platforms. You can reduce the number of tasks by filtering on platforms +(e.g. linux) or build type (e.g. opt): + + mach try --preset remote-protocol -xq "'linux 'opt" + +But you can also schedule tests by selecting relevant jobs yourself: + + mach try fuzzy + +[Phabricator]: https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html +[commit creation guidelines]: https://mozilla-version-control-tools.readthedocs.io/en/latest/devguide/contributing.html#submitting-patches-for-review +[becoming a Mozilla committer]: https://www.mozilla.org/en-US/about/governance/policies/commit/ +[Firefox CI]: https://treeherder.mozilla.org/ +[try preset]: /tools/try/presets diff --git a/remote/doc/marionette/Prefs.md b/remote/doc/marionette/Prefs.md new file mode 100644 index 0000000000..2f464054c3 --- /dev/null +++ b/remote/doc/marionette/Prefs.md @@ -0,0 +1,24 @@ +# Preferences + +There are a couple of [Remote Agent preferences] associated with the Gecko remote +protocol. Those listed below are additional ones uniquely used for Marionette. + +[Remote Agent preferences]: /remote/Prefs.md + +## `marionette.debugging.clicktostart` + +Delay server startup until a modal dialogue has been clicked to +allow time for user to set breakpoints in the [Browser Toolbox]. + +[Browser Toolbox]: /devtools-user/browser_toolbox/index.rst + +## `marionette.port` + +Defines the port on which the Marionette server will listen. Defaults +to port 2828. + +This can be set to 0 to have the system atomically allocate a free +port, which can be useful when running multiple Marionette servers +on the same system. The effective port is written to the user +preference file when the server has started and is also logged to +stdout. diff --git a/remote/doc/marionette/Protocol.md b/remote/doc/marionette/Protocol.md new file mode 100644 index 0000000000..cd42c453b6 --- /dev/null +++ b/remote/doc/marionette/Protocol.md @@ -0,0 +1,114 @@ +# Protocol + +Marionette provides an asynchronous, parallel pipelining user-facing +interface. Message sequencing limits chances of payload race +conditions and provides a uniform way in which payloads are serialised. + +Clients that deliver a blocking WebDriver interface are still +expected to not send further command requests before the response +from the last command has come back, but if they still happen to do +so because of programming error, no harm will be done. This guards +against [mixing up responses]. + +Schematic flow of messages: + + client server + | | + msgid=1 |----------->| + | command | + | | + msgid=2 |<-----------| + | command | + | | + msgid=2 |----------->| + | response | + | | + msgid=1 |<-----------| + | response | + | | + +The protocol consists of a `command` message and the corresponding +`response` message. A `response` message must always be sent in +reply to a `command` message. + +This means that the server implementation does not need to send +the reply precisely in the order of the received commands: if it +receives multiple messages, the server may even reply in random order. +It is therefore strongly advised that clients take this into account +when imlpementing the client end of this wire protocol. + +This is required for pipelining messages. On the server side, +some functions are fast, and some less so. If the server must +reply in order, the slow functions delay the other replies even if +its execution is already completed. + +[mixing up responses]: https://bugzil.la/1207125 + +## Command + +The request, or `command` message, is a four element JSON Array as shown +below, that may originate from either the client- or server remote ends: + + [type, message ID, command, parameters] + +* _type_ must be 0 (integer). This indicates that the message + is a `command`. + +* _message ID_ is a 32-bit unsigned integer. This number is + used as a sequencing number that uniquely identifies a pair of + `command` and `response` messages. The other remote part will + reply with a corresponding `response` with the same message ID. + +* _command_ is a string identifying the RPC method or command + to execute. + +* _parameters_ is an arbitrary JSON serialisable object. + +## Response + +The response message is also a four element array as shown below, +and must always be sent after receiving a `command`: + + [type, message ID, error, result] + +* _type_ must be 1 (integer). This indicates that the message is a + `response`. + +* _message ID_ is a 32-bit unsigned integer. This corresponds + to the `command`’s message ID. + +* _error_ is null if the command executed correctly. If the + error occurred on the server-side, then this is an [error] object. + +* _result_ is the result object from executing the `command`, if + it executed correctly. If an error occurred on the server-side, + this field is null. + +The structure of the result field can vary, but is documented +individually for each command. + +## Error object + +An error object is a serialisation of JavaScript error types, +and it is structured like this: + + { + "error": "invalid session id", + "message": "No active session with ID 1234", + "stacktrace": "" + } + +All the fields of the error object are required, so the stacktrace and +message fields may be empty strings. The error field is guaranteed +to be one of the JSON error codes as laid out by the [WebDriver standard]. + +## Clients + +Clients may be implemented in any language that is capable of writing +and receiving data over TCP socket. A [reference client] is provided. +Clients may be implemented both synchronously and asynchronously, +although the latter is impossible in protocol levels 2 and earlier +due to the lack of message sequencing. + +[WebDriver standard]: https://w3c.github.io/webdriver/#dfn-error-code +[reference client]: https://searchfox.org/mozilla-central/source/testing/marionette/client/ diff --git a/remote/doc/marionette/PythonTests.md b/remote/doc/marionette/PythonTests.md new file mode 100644 index 0000000000..c3497d6272 --- /dev/null +++ b/remote/doc/marionette/PythonTests.md @@ -0,0 +1,69 @@ +# Mn Python tests + +_Marionette_ is the codename of a [remote protocol] built in to +Firefox as well as the name of a functional test framework for +automating user interface tests. + +The in-tree test framework supports tests written in Python, using +Python’s [unittest] library. Test cases are written as a subclass +of `MarionetteTestCase`, with child tests belonging to instance +methods that have a name starting with `test_`. + +You can additionally define [`setUp`] and [`tearDown`] instance +methods to execute code before and after child tests, and +[`setUpClass`]/[`tearDownClass`] for the parent test. When you use +these, it is important to remember calling the `MarionetteTestCase` +superclass’ own [`setUp`]/[`tearDown`] methods since they handle +setup/cleanup of the session. + +The test structure is illustrated here: + +```python +from marionette_harness import MarionetteTestCase + +class TestSomething(MarionetteTestCase): + def setUp(self): + # code to execute before any tests are run + MarionetteTestCase.setUp(self) + + def test_foo(self): + # run test for 'foo' + + def test_bar(self): + # run test for 'bar' + + def tearDown(self): + # code to execute after all tests are run + MarionetteTestCase.tearDown(self) +``` + +[remote protocol]: Protocol.md +[unittest]: https://docs.python.org/3/library/unittest.html +[`setUp`]: https://docs.python.org/3/library/unittest.html#unittest.TestCase.setUp +[`setUpClass`]: https://docs.python.org/3/library/unittest.html#unittest.TestCase.setUpClass +[`tearDown`]: https://docs.python.org/3/library/unittest.html#unittest.TestCase.tearDown +[`tearDownClass`]: https://docs.python.org/3/library/unittest.html#unittest.TestCase.tearDownClass + +## Test assertions + +Assertions are provided courtesy of [unittest]. For example: + +```python +from marionette_harness import MarionetteTestCase + +class TestSomething(MarionetteTestCase): + def test_foo(self): + self.assertEqual(9, 3 * 3, '3 x 3 should be 9') + self.assertTrue(type(2) == int, '2 should be an integer') +``` + +## The API + +The full API documentation is found [here], but the key objects are: + +* `MarionetteTestCase`: a subclass for `unittest.TestCase` + used as a base class for all tests to run. + +* {class}`Marionette <marionette_driver.marionette.Marionette>`: client that speaks to Firefox + +[here]: /python/marionette_driver.rst diff --git a/remote/doc/marionette/SeleniumAtoms.md b/remote/doc/marionette/SeleniumAtoms.md new file mode 100644 index 0000000000..3788de2c54 --- /dev/null +++ b/remote/doc/marionette/SeleniumAtoms.md @@ -0,0 +1,90 @@ +# Selenium atoms + +Marionette uses a small list of [Selenium atoms] to interact with +web elements. Initially those have been added to ensure a better +reliability due to a wider usage inside the Selenium project. But +by adding full support for the [WebDriver specification] they will +be removed step by step. + +Currently the following atoms are in use: + +- `getElementText` +- `isElementDisplayed` +- `isElementEnabled` + +To use one of those atoms Javascript modules will have to import +[atom.sys.mjs]. + +[Selenium atoms]: https://github.com/SeleniumHQ/selenium/tree/master/javascript/webdriver/atoms +[WebDriver specification]: https://w3c.github.io/webdriver/webdriver-spec.html +[atom.sys.mjs]: https://searchfox.org/mozilla-central/source/remote/marionette/atom.sys.mjs + +## Update required Selenium atoms + +In regular intervals the atoms, which are still in use, have to +be updated. Therefore they have to be exported from the Selenium +repository first, and then updated in [atom.sys.mjs]. + +### Export Selenium Atoms + +The canonical GitHub repository for Selenium is + + <https://github.com/SeleniumHQ/selenium.git> + +so make sure to have an up-to-date local copy of it. If you have to clone +it first, it is recommended to specify the `--depth=1` argument, so only the +last changeset is getting downloaded (which itself might already be +more than 100 MB). + +```bash +git clone --depth=1 https://github.com/SeleniumHQ/selenium.git +``` + +To export the correct version of the atoms identify the changeset id (SHA1) of +the Selenium repository in the [index section] of the WebDriver specification. + +Fetch that changeset and check it out: + +```bash +git fetch --depth=1 origin SHA1 +git checkout SHA1 +``` + +Now you can export all the required atoms by running the following +commands. Make sure to [install bazelisk] first. + +```bash +bazel build //javascript/atoms/fragments:get-text +bazel build //javascript/atoms/fragments:is-displayed +bazel build //javascript/atoms/fragments:is-enabled +``` + +For each of the exported atoms a file can now be found in the folder +`bazel-bin/javascript/atoms/fragments/`. They contain all the +code including dependencies for the atom wrapped into a single function. + +[index section]: <https://w3c.github.io/webdriver/#index> +[install bazelisk]: <https://github.com/bazelbuild/bazelisk#installation> + +### Update atom.sys.mjs + +To update the atoms for Marionette the `atoms.js` file has to be edited. For +each atom to be updated the steps as laid out below have to be performed: + +1. Open the Javascript file of the exported atom. See above for + its location. + +2. Remove the contained license header, which can be found somewhere + in the middle of the file. + +3. Update the parameters of the wrapper function (at the very top) + so that those are equal with the used parameters in `atom.sys.mjs`. + +4. Copy the whole content of the file, and replace the existing + code for the atom in `atom.sys.mjs`. + +### Test the changes + +To ensure that the update of the atoms doesn't cause a regression +a try build should be run including Marionette unit tests, Firefox +ui tests, and all the web-platform-tests. diff --git a/remote/doc/marionette/Taskcluster.md b/remote/doc/marionette/Taskcluster.md new file mode 100644 index 0000000000..c5c9832523 --- /dev/null +++ b/remote/doc/marionette/Taskcluster.md @@ -0,0 +1,88 @@ +# Testing with one-click loaners + +[Taskcluster] is the task execution framework that supports Mozilla's +continuous integration and release processes. + +Build and test jobs (like Marionette) are executed across all supported +platforms, and job results are pushed to [Treeherder] for observation. + +The best way to debug issues for intermittent test failures of +Marionette tests for Firefox and Fennec (Android) is to use a +one-click loaner as provided by Taskcluster. Such a loaner creates +an interactive task you can interact with via a shell and VNC. + +To create an interactive task for a Marionette job which is shown +as failed on Treeherder, select the job, click the ellipse in the lower +left pane, and choose `Create Interactive Task`. + +Please note that you need special permissions to actually request +such a loaner. + +When the task has been created you will receive an email with the connection +details. Open the referenced shell and you will be connected via a WebSocket. +Once that has been done a wizard will automatically launch and +provide some options. Best here is to choose the second option, +which will run all the setup steps, installs the Firefox or Fennec +binary, and then exits. + +[Taskcluster]: https://docs.taskcluster.net/ +[Treeherder]: https://treeherder.mozilla.org + +## Setting up the Marionette environment + +Best here is to use a virtual environment, which has all the +necessary packages installed. If no modifications to any Python +package will be done, the already created environment by the +wizard can be used: + + % cd /builds/worker/workspace/build + % source venv/bin/activate + +Otherwise a new virtual environment needs to be created and +populated with the mozbase and marionette packages installed: + + % cd /builds/worker/workspace/build && rm -r venv + % virtualenv venv && source venv/bin/activate + % cd tests/mozbase && ./setup_development.py + % cd ../marionette/client && python setup.py develop + % cd ../harness && python setup.py develop + % cd ../../../ + +## Running Marionette tests + +### Firefox + +To run the Marionette tests execute the `runtests.py` script. For all +the required options as best search in the log file of the failing job +the interactive task has been created from. Then copy the complete +command and run it inside the already sourced virtual environment: + + % /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --gecko-log=- -vv --binary=/builds/worker/workspace/build/application/firefox/firefox --address=127.0.0.1:2828 --symbols-path=https://queue.taskcluster.net/v1/task/GSuwee61Qyibujtxq4UV3A/artifacts/public/build/target.crashreporter-symbols.zip /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini + +### Fennec + +The Marionette tests for Fennec are executed by using an Android +emulator which runs on the host platform. As such some extra setup +steps compared to Firefox on desktop are required. + +The following lines set necessary environment variables before +starting the emulator in the background, and to let Marionette +know of various Android SDK tools. + + % export ADB_PATH=/builds/worker/workspace/build/android-sdk-linux/platform-tools/adb + % export ANDROID_AVD_HOME=/builds/worker/workspace/build/.android/avd/ + % /builds/worker/workspace/build/android-sdk-linux/tools/emulator -avd test-1 -show-kernel -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config,socket & + +The actual call to `runtests.py` is different per test job because +those are using chunks on Android. As best search for the command +and its options in the log file of the failing job the interactive +task has been created from. Then copy the complete command and run +it inside the already sourced virtual environment. + +Here an example for chunk 1 which runs all the tests in the current +chunk with some options for logs removed: + + % /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --emulator --app=fennec --package=org.mozilla.fennec_aurora --address=127.0.0.1:2828 /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini --gecko-log=- --symbols-path=/builds/worker/workspace/build/symbols --startup-timeout=300 --this-chunk 1 --total-chunks 10 + +To execute a specific test only simply replace `unit-tests.ini` +with its name. diff --git a/remote/doc/marionette/Testing.md b/remote/doc/marionette/Testing.md new file mode 100644 index 0000000000..a40443dccb --- /dev/null +++ b/remote/doc/marionette/Testing.md @@ -0,0 +1,192 @@ +# Testing + +We verify and test Marionette in a couple of different ways, using +a combination of unit tests and functional tests. There are three +distinct components that we test: + +- the Marionette **server**, using a combination of xpcshell +unit tests and functional tests written in Python spread across +Marionette- and WPT tests; + +- the Python **client** is tested with the same body of functional +Marionette tests; + +- and the **harness** that backs the Marionette, or `Mn` job on +try, tests is verified using separate mock-styled unit tests. + +All these tests can be run by using [mach]. + +[mach]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach + +## xpcshell unit tests + +Marionette has a set of [xpcshell] unit tests located in +_remote/marionette/test/xpcshell. These can be run this way: + + % ./mach test remote/marionette/test/unit + +Because tests are run in parallel and xpcshell itself is quite +chatty, it can sometimes be useful to run the tests sequentially: + + % ./mach test --sequential remote/marionette/test/xpcshell/test_error.js + +These unit tests run as part of the `X` jobs on Treeherder. + +[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests + +## Marionette functional tests + +We also have a set of [functional tests] that make use of the Marionette +Python client. These start a Firefox process and tests the Marionette +protocol input and output, and will appear as `Mn` on Treeherder. +The following command will run all tests locally: + + % ./mach marionette-test + +But you can also run individual tests: + + % ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py + +In case you want to run the tests with another binary like [Firefox Nightly]: + + % ./mach marionette-test --binary /path/to/nightly/firefox TEST + +When working on Marionette it is often useful to surface the stdout +from Gecko, which can be achieved using the `--gecko-log` option. +See [Debugging](Debugging.md) for usage instructions, but the gist is that +you can redirect all Gecko output to stdout: + + % ./mach marionette-test --gecko-log - TEST + +Our functional integration tests pop up Firefox windows sporadically, +and a helpful tip is to suppress the window can be to use Firefox’ +headless mode: + + % ./mach marionette-test -z TEST + +`-z` is an alias for the `--headless` flag and equivalent to setting +the `MOZ_HEADLESS` output variable. In addition to `MOZ_HEADLESS` +there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for +controlling the dimensions of the no-op virtual display. This is +similar to using Xvfb(1) which you may know from the X windowing system, +but has the additional benefit of also working on macOS and Windows. + +[functional tests]: PythonTests.md +[Firefox Nightly]: https://nightly.mozilla.org/ + +### Android + +Prerequisites: + +- You have [built Fennec](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_for_Android_build). +- You can run an Android [emulator](https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#Running_tests_on_the_Android_emulator), + which means you have the AVD you need. + +When running tests on Fennec, you can have Marionette runner take care of +starting Fennec and an emulator, as shown below. + + % ./mach marionette-test --emulator --app fennec + --avd-home /path/to/.mozbuild/android-device/avd + --emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator + --avd=mozemulator-x86 + +For Fennec tests, if the appropriate `emulator` command is in your `PATH`, you may omit the `--emulator-binary` argument. See `./mach marionette-test -h` +for additional options. + +Alternately, you can start an emulator yourself and have the Marionette runner +start Fennec for you: + + % ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 + +To connect to an already-running Fennec in an emulator or on a device, +you will need to have it started with the `-marionette` command line argument, +or by setting the environment variable `MOZ_MARIONETTE=1` for the process. + +Make sure port 2828 is forwarded: + + % adb forward tcp:2828 tcp:2828 + +If Fennec is already started: + + % ./mach marionette-test --app='fennec' --address=127.0.0.1:2828 + +If Fennec is not already started on the emulator/device, add the `--emulator` +option. Marionette Test Runner will take care of forwarding the port and +starting Fennec with the correct prefs. (You may need to run +`adb forward --remove-all` to allow the runner to start.) + + % ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --startup-timeout=300 + +If you need to troubleshoot the Marionette connection, the most basic check is +to start Fennec with `-marionette` or the environment variable `MOZ_MARIONETTE=1`, +make sure that the port 2828 is forwarded, and then see if you get any response from +Marionette when you connect manually: + + % telnet 127.0.0.1:2828 + +You should see output like `{"applicationType":"gecko","marionetteProtocol":3}` + +[geckodriver]: /testing/geckodriver/index.rst + +## WPT functional tests + +Marionette is also indirectly tested through [geckodriver] with WPT +(`Wd` on Treeherder). To run them: + + % ./mach wpt testing/web-platform/tests/webdriver + +WPT tests conformance to the [WebDriver] standard and uses +[geckodriver]. Together with the Marionette remote protocol in +Gecko, they make up Mozilla’s WebDriver implementation. + +This command supports a `--webdriver-arg='-vv'` argument that +enables more detailed logging, as well as `--jsdebugger` for opening +the Browser Toolbox. + +A particularly useful trick is to combine this with the headless +mode for Firefox: + + % ./mach wpt --webdriver-arg='-vv' --headless testing/web-platform/tests/webdriver + +[WebDriver]: https://w3c.github.io/webdriver/ + +## Harness tests + +The Marionette harness Python package has a set of mock-styled unit +tests that uses the [pytest] framework. The following command will +run all tests: + + % ./mach python-test testing/marionette + +To run a specific test specify the full path to the module: + + % ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py + +[pytest]: https://docs.pytest.org/en/latest/ + +## One-click loaners + +Additionally, for debugging hard-to-reproduce test failures in CI, +one-click loaners from [Taskcluster](Taskcluster.md) can be particularly useful. + +## Out-of-tree testing + +All the above examples show tests running _in-tree_, with a local +checkout of _central_ and a local build of Firefox. It is also +possibly to run the Marionette tests _without_ a local build and +with a downloaded test archive from [Taskcluster](Taskcluster.md) + +If you want to run tests from a downloaded test archive, you will +need to download the `target.common.tests.tar.gz` artifact attached to +Treeherder [build jobs] `B` for your system. Extract the archive +and set up the Python Marionette client and harness by executing +the following command in a virtual environment: + + % pip install -r config/marionette_requirements.txt + +The tests can then be found under +_marionette/tests/testing/marionette/harness/marionette_harness/tests_ +and can be executed with the command `marionette`. It supports +the same options as described above for `mach`. + +[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build diff --git a/remote/doc/marionette/index.rst b/remote/doc/marionette/index.rst new file mode 100644 index 0000000000..7272aff162 --- /dev/null +++ b/remote/doc/marionette/index.rst @@ -0,0 +1,64 @@ +========== +Marionette +========== + +Marionette is a remote `protocol`_ that lets out-of-process programs +communicate with, instrument, and control Gecko-based browsers. + +It provides interfaces for interacting with both the internal JavaScript +runtime and UI elements of Gecko-based browsers, such as Firefox +and Fennec. It can control both the chrome- and content documents, +giving a high level of control and ability to emulate user interaction. + +Within the central tree, Marionette is used in most TaskCluster +test jobs to instrument Gecko. It can additionally be used to +write different kinds of functional tests: + + * The `Marionette Python client`_ is used in the `Mn` job, which + is generally what you want to use for interacting with web documents + +Outside the tree, Marionette is used by `geckodriver`_ to implement +`WebDriver`_. + +Marionette supports to various degrees all the Gecko based applications, +including Firefox, Thunderbird, Fennec, and Fenix. + +.. _protocol: Protocol.html +.. _Marionette Python client: /python/marionette_driver.html +.. _geckodriver: /testing/geckodriver/ +.. _WebDriver: https://w3c.github.io/webdriver/ + +Some further documentation can be found here: + +.. toctree:: + :maxdepth: 1 + + Intro.md + Building.md + PythonTests.md + Protocol.md + Contributing.md + NewContributors.md + Patches.md + Debugging.md + Testing.md + Taskcluster.md + CodeStyle.md + SeleniumAtoms.md + Prefs.md + + +Bugs +==== + +Bugs are tracked in the `Testing :: Marionette` component. + + +Communication +============= + +The mailing list for Marionette discussion is +https://groups.google.com/a/mozilla.org/g/dev-webdriver. + +If you prefer real-time chat, ask your questions +on `#webdriver:mozilla.org <https://chat.mozilla.org/#/room/#webdriver:mozilla.org>`__. |