diff options
Diffstat (limited to '')
| -rw-r--r-- | remote/doc/cdp/Architecture.md | 163 | ||||
| -rw-r--r-- | remote/doc/cdp/PuppeteerVendor.md | 103 | ||||
| -rw-r--r-- | remote/doc/cdp/RequiredPreferences.md | 14 | ||||
| -rw-r--r-- | remote/doc/cdp/Usage.md | 62 | ||||
| -rw-r--r-- | remote/doc/cdp/index.rst | 27 |
5 files changed, 369 insertions, 0 deletions
diff --git a/remote/doc/cdp/Architecture.md b/remote/doc/cdp/Architecture.md new file mode 100644 index 0000000000..f9465e3c71 --- /dev/null +++ b/remote/doc/cdp/Architecture.md @@ -0,0 +1,163 @@ +Remote Agent overall architecture +================================= + +This document will cover the Remote Agent architecture by following the sequence of steps needed to start the agent, connect a client and debug a target. + +Remote Agent startup +-------------------- + +Everything starts with the `RemoteAgent` implementation, which handles command line +arguments (--remote-debugging-port) to eventually +start a server listening on the TCP port 9222 (or the one specified by the command line). +The browser target websocket URL will be printed to stderr. +To do that this component glue together three main high level components: + + * `server/HTTPD` + This is a copy of httpd.js, from /netwerk/ folder. This is a JS implementation of an http server. + This will be used to implement the various http endpoints of CDP. + There is a few static URL implemented by `JSONHandler` and one dynamic URL per target. + + * `cdp/JSONHandler` + This implements the following three static http endpoints: + * /json/version: + Returns information about the runtime as well as the url of the browser target websocket url. + * /json/list: + Returns a list of all debuggable targets with, for each, their dynamic websocket URL. + For now it only reports tabs, but will report workers and addons as soon as we support them. + The main browser target is the only one target not listed here. + * /json/protocol: + Returns a big dictionary describing the supported protocol. + This is currently hard coded and returns the full CDP protocol schema, including APIs we don’t support. + We have a future intention to fix this and report only what Firefox implements. + You can connect to these websocket URL in order to debug things. + + * `cdp/targets/TargetList` + This component is responsible of maintaining the list of all debuggable targets. + For now it can be either: + * The main browser target + A special target which allows to inspect the browser, but not any particular tab. + This is implemented by `cdp/targets/MainProcessTarget` and is instantiated on startup. + * Tab targets + Each opened tab will have a related `cdp/targets/TabTarget` instantiated on their opening, + or on server startup for already opened ones. + Each target aims at focusing on one particular context. This context is typically running in one + particular environment. This can be a particular process or thread. + In the future, we will most likely support targets for workers and add-ons. + All targets inherit from `cdp/targets/Target`. + +Connecting to Websocket endpoints +--------------------------------- + +Each target's websocket URL will be registered as a HTTP endpoint via `server/HTTPD:registerPathHandler` (This registration is done from `RemoteAgentParentProcess:#listen`). +Once a HTTP request happens, `server/HTTPD` will call the `handle` method on the object passed to `registerPathHandler`. +For static endpoints registered by `JSONHandler`, this will call `JSONHandler:handle` and return a JSON string as http body. +For target's endpoint, it is slightly more complicated as it requires a special handshake to morph the HTTP connection into a WebSocket one. +The WebSocket is then going to be long lived and be used to inspect the target over time. +When a request is made to a target URL, `cdp/targets/Target:handle` is called and: + + * delegate the complex HTTP to WebSocket handshake operation to `server/WebSocketHandshake:upgrade` + In return we retrieve a WebSocket object. + + * hand over this WebSocket to `server/WebSocketTransport` + and get a transport object in return. The transport implements a basic JSON stream over WebSocket. With that, you can send and receive JSON objects over a WebSocket connection. + + * hand over the transport to a freshly instantiated `Connection` + The Connection has two goals: + * Interpret incoming CDP packets by reading the JSON object attribute (`id`, `method`, `params` and `sessionId`) + This is done in `Connection:onPacket`. + * Format outgoing CDP packets by writing the right JSON object for command response (`id`, `result` and `sessionId`) and events (`method`, `params` and `sessionId`) + * Redirect CDP packet from/to the right session. + A connection may have more than one session attached to it. + + * instantiate the default session + The session is specific to each target kind and all of them inherit from `cdp/session/Session`. + For example, tabs targets uses `cdp/session/TabSession` and the main browser target uses `cdp/session/MainProcessSession`. + Which session class is used is defined by the Target subclass’ constructor, which pass a session class reference to `cdp/targets/Target:constructor`. + A session is mostly responsible of accommodating the eventual cross process/cross thread aspects of the target. + The code we are currently describing (`cdp/targets/Target:handle`) is running in the parent process. + The session class receive CDP commands from the connection and first try to execute the Domain commands in the parent process. + Then, if the target actually runs in some other context, the session tries to forward this command to this other context, which can be a thread or a process. + Typically, the `cdp/sessions/TabSession` forward the CDP command to the content process where the tab is running. + It also redirects back the command response as well as Domain events from that process back to the parent process in order to + forward them to the connection. + Sessions will be using the `DomainCache` class as a helper to manage a list of Domain implementations in a given context. + +Debugging additional Targets +---------------------------- + +From a given connection you can know about the other potential targets. +You typically do that via `Target.setDiscoverTargets()`, which will emit `Target.targetCreated` events providing a target ID. +You may create a new session for the new target by handing the ID to `Target.attachToTarget()`, which will return a session ID. +"Target" here is a reference to the CDP Domain implemented in `cdp/domains/parent/Target.jsm`. That is different from `cdp/targets/Target` +class which is an implementation detail of the Remote Agent. + +Then, there is two ways to communicate with the other targets: + + * Use `Target.sendMessageToTarget()` and `Target.receivedMessageFromTarget` + You will manually send commands via the `Target.sendMessageToTarget()` command and receive command's response as well as events via `Target.receivedMessageFromTarget`. + In both cases, a session ID attribute is passed in the command or event arguments in order to select which additional target you are communicating with. + + * Use `Target.attachToTarget({ flatten: true })` and include `sessionId` in CDP packets + This requires a special client, which will use the `sessionId` returned by `Target.attachToTarget()` in order to spawn a distinct client instance. + This client will re-use the same WebSocket connection, but every single CDP packet will contain an additional `sessionId` attribute. + This helps distinguish packets which relate to the original target as well as the multiple additional targets you may attach to. + +In both cases, `Target.attachToTarget()` is special as it will spawn `cdp/session/TabSession` for the tab you are attaching to. +This is the codepath creating non-default session. The default session is related to the target you originally connected to, +so that you don't need any ID for this one. When you want to debug more than one target over a single connection +you need additional sessions, which will have a unique ID. +`Target.attachToTarget` will compute this ID and instantiate a new session bound to the given target. +This additional session will be managed by the `Connection` class, which will then redirect CDP packets to the +right session when you are using flatten session. + +Cross Process / Layers +---------------------- + +Because targets may runs in different contexts, the Remote Agent code runs in different processes. +The main and startup code of the Remote Agent code runs in the parent process. +The handling of the command line as well as all the HTTP and WebSocket work is all done in the parent process. +The browser target is also all implemented in the parent process. +But when it comes to a tab target, as the tab runs in the content process, we have to run code there as well. +Let's start from the `cdp/sessions/TabSession` class, which has already been described. +We receive here JSON packets from the WebSocket connection and we are in the parent process. +In this class, we route the messages to the parent process domains first. +If there is no implementation of the domain or the particular method, +we forward the command to a `cdp/session/ContentProcessSession` which runs in the tab's content process. +These two Session classes will interact with each other in order to forward back the returned value +of the method we just called, as well as piping back any event being sent by a Domain implemented in any +of the two processes. + +Organizational chart of all the classes +---------------------------------------- +``` + ┌─────────────────────────────────────────────────┐ + │ │ + 1 ▼ │ + ┌───────────────┐ 1 ┌───────────────┐ 1..n┌───────────────┐ + │ RemoteAgent │──────▶│ HttpServer │◀───────▶│ JsonHandler │ + └───────────────┘ └───────────────┘ 1 └───────────────┘ + │ + │ + │ 1 ┌────────────────┐ 1 + └───────────────▶│ TargetList │◀─┐ + └────────────────┘ │ + │ │ + ▼ 1..n │ + ┌────────────┐ │ + ┌─────────────────│ Target [1]│ │ + │ └────────────┘ │ + │ ▲ 1 │ + ▼ 1..n │ │ + ┌────────────┐ 1..n┌────────────┐ │ + │ Connection │◀─────────▶│ Session [2]│──────┘ + └────────────┘ 1 └────────────┘ + │ 1 ▲ + │ │ + ▼ 1 ▼ 1 +┌────────────────────┐ ┌──────────────┐ 1..n┌────────────┐ +│ WebSocketTransport │ │ DomainCache | │──────────▶│ Domain [3]│ +└────────────────────┘ └──────────────┘ └────────────┘ +``` + [1] Target is inherited by TabTarget and MainProcessTarget. + [2] Session is inherited by TabSession and MainProcessSession. + [3] Domain is inherited by Log, Page, Browser, Target.... i.e. all domain implementations. From both cdp/domains/parent and cdp/domains/content folders. diff --git a/remote/doc/cdp/PuppeteerVendor.md b/remote/doc/cdp/PuppeteerVendor.md new file mode 100644 index 0000000000..248f5b7c18 --- /dev/null +++ b/remote/doc/cdp/PuppeteerVendor.md @@ -0,0 +1,103 @@ +# Vendoring Puppeteer + +As mentioned in the chapter on [Testing], we run the full [Puppeteer +test suite] on try. These tests are vendored in central under +_remote/test/puppeteer/_ and we have a script to pull in upstream changes. + +We periodically perform a manual two-way sync. Below is an outline of the +process interspersed with some tips. + +## Check for not-yet upstreamed changes + +Before vendoring a new Puppeteer release make sure that there are no Puppeteer +specific changes in mozilla-central repository that haven't been upstreamed yet +since the last vendor happened. Run one of the following commands and check the +listed bugs or related upstream code to verify: + +```shell +% hg log remote/test/puppeteer +% git log remote/test/puppeteer +``` + +If an upstream pull request is needed please see their [contributing.md]. +Typically, the changes we push to Puppeteer include unskipping newly passing +unit tests for Firefox along with minor fixes to the tests or +to Firefox-specific browser-fetching and launch code. + +Be sure to [run tests against both Chromium and Firefox] in the Puppeteer +repo. You can specify your local Firefox build when you do so: + +```shell +% BINARY=<path-to-objdir-binary> npm run funit +``` + +## Prepare the Puppeteer Repository + +Clone the Puppeteer git repository and checkout the release tag you want +to vendor into mozilla-central. + +```shell +% git checkout tags/puppeteer-%version% +``` + +You might want to [install the project] at this point and make sure unit tests pass. +Check the project's `package.json` for relevant testing commands. + +## Update the Puppeteer code in mozilla-central + +You can run the following mach command to copy over the Puppeteer branch you +just prepared. The mach command has flags to specify a local or remote +repository as well as a commit: + +```shell +% ./mach remote vendor-puppeteer --commitish puppeteer-%version% +``` + +By default, this command also installs the newly-pulled Puppeteer package in +order to generate a new `package-lock.json` file for the purpose of pinning +Puppeteer dependencies for our CI. There is a `--no-install` option if you want +to skip this step; for example, if you want to run installation separately at +a later point. + +### Validate that the new code works + +Use `./mach puppeteer-test` (see [Testing]) to run Puppeteer tests against both +Chromium and Firefox in headless mode. Again, only running a subset of tests +against Firefox is fine -- at this point you just want to check that the +typescript compiles and the browser binaries are launched successfully. + +If something at this stage fails, you might want to check changes in +`remote/test/puppeteer/package.json` and update `remote/mach_commands.py` +with new npm scripts. + +### Verify the expectation meta data + +Next, you want to make sure that the expectation meta data is correct. Check +changes in [TestExpectations.json]. If there are +newly skipped tests for Firefox, you might need to update these expectations. +To do this, run the Puppeteer test job on try (see [Testing]). If these tests +are specific for Chrome or time out, we want to keep them skipped, if they fail +we want to have `FAIL` status for all platforms in the expectation meta data. +You can see, if the meta data needs to be updated, at the end of the log file. + +Examine the job logs and make sure the run didn't get interrupted early by a +crash or a hang, especially if you see a lot of `TEST-UNEXPECTED-MISSING` in +the Treeherder Failure Summary. You might have to fix some new bug in the unit +tests. This is the fun part. + +Some tests can also unexpectedly pass. Make sure it's correct, and if needed +update the expectation data by following the instructions at the end of the +log file. + +### Submit the code changes + +Once you are happy with the metadata and are ready to submit the sync patch +up for review, run the Puppeteer test job on try again with `--rebuild 10` +to check for stability. + +[Testing]: ../Testing.md +[Puppeteer test suite]: https://github.com/GoogleChrome/puppeteer/tree/master/test +[install the project]: https://github.com/puppeteer/puppeteer/blob/main/docs/contributing.md#getting-started +[run tests against both Chromium and Firefox]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md#running-tests +[TestExpectations.json]: https://searchfox.org/mozilla-central/source/remote/test/puppeteer/test/TestExpectations.json +[contributing.md]: https://github.com/puppeteer/puppeteer/blob/main/docs/contributing.md diff --git a/remote/doc/cdp/RequiredPreferences.md b/remote/doc/cdp/RequiredPreferences.md new file mode 100644 index 0000000000..26b4b54d90 --- /dev/null +++ b/remote/doc/cdp/RequiredPreferences.md @@ -0,0 +1,14 @@ +Required Preferences for Fission +================================ + +Fission (site isolation for Firefox) introduced some architectural changes that are incompatible with our CDP implementation. To keep using CDP for Firefox, make sure the following preferences are set in the profile before starting Firefox with `--remote-debugging-port`: + + * `fission.bfcacheInParent` should be set to `false`. + + * `fission.webContentIsolationStrategy` should be set to `0`. + +Without those preferences, expect issues related to navigation in several domains (Page, Runtime, ...). + +Third party tools relying on CDP such as Puppeteer ensure that those preferences are correctly set before starting Firefox. + +The work to lift those restrictions is tracked in [Bug 1732263](https://bugzilla.mozilla.org/show_bug.cgi?id=1732263) and [Bug 1706353](https://bugzilla.mozilla.org/show_bug.cgi?id=1706353). diff --git a/remote/doc/cdp/Usage.md b/remote/doc/cdp/Usage.md new file mode 100644 index 0000000000..747bcb8fe2 --- /dev/null +++ b/remote/doc/cdp/Usage.md @@ -0,0 +1,62 @@ +Usage +===== + +When using the CDP-based Remote Agent in Firefox, there are +three different programs/components running simultaneously: + + * the __client__, being the out-of-process script or library + (such as Puppeteer) or web inspector frontend you use to control + and retrieve information out of Firefox; + + * the __agent__ that the client connects to which is an HTTPD living + inside Firefox, facilitating communication between clients + and targets; + + * and the __target__, which is the web document being debugging. + +Since Firefox 86 the Remote Agent ships in all Firefox releases by default. + +To check if your Firefox binary has the Remote Agent enabled, you +can look in its help message for this: + + % ./firefox -h + … + --remote-debugging-port [<port>] Start the Firefox Remote Agent, which is + a low-level debugging interface based on the CDP protocol. + Defaults to listen on localhost:9222. + … + +When used, the Remote Agent will start an HTTP server and print a +message on stderr with the location of the main target’s WebSocket +listener: + + % firefox --remote-debugging-port + DevTools listening on ws://localhost:9222/devtools/browser/7b4e84a4-597f-4839-ac6d-c9e86d16fb83 + +`--remote-debugging-port` takes an optional port as input: + + [<port>] + +You can use this to instruct the Remote Agent to bind to a particular +port on your system. port is optional, +which means `firefox --remote-debugging-port` will bind the HTTPD to +the default `localhost:9222`. + +If port has been specified the default port will be overridden: + + % firefox --remote-debugging-port 9989 + DevTools listening on ws://localhost:9989/devtools/browser/b49481af-8ad3-9b4d-b1bf-bb0cdb9a0620 + +When you ask the Remote Agent to listen on port 0, +the system will atomically allocate an arbitrary free port: + + % firefox --remote-debugging-port 0 + DevTools listening on ws://localhost:59982/devtools/browser/a12b22a9-1b8b-954a-b81f-bd31552d3f1c + +Allocating an atomic port can be useful if you want to avoid race +conditions. The atomically allocated port will be somewhere in the +ephemeral port range, which varies depending on your system and +system configuration, but is always guaranteed to be free thus +eliminating the risk of binding to a port that is already in use. + +[Firefox Nightly]: https://www.mozilla.org/firefox/channel/desktop/#nightly diff --git a/remote/doc/cdp/index.rst b/remote/doc/cdp/index.rst new file mode 100644 index 0000000000..4892c5c457 --- /dev/null +++ b/remote/doc/cdp/index.rst @@ -0,0 +1,27 @@ +===================== +Remote Protocol (CDP) +===================== + +The Firefox **remote protocol (CDP)** is a low-level debugging interface +you can use to inspect the state and control execution of documents +running in web content, instrument the browser in interesting ways, +simulate user interaction for automation purposes, and for subscribing +to updates in the browser such as network- or console logs. + +It complements the existing Firefox Developer Tools :ref:`Remote Debugging +Protocol <Remote Debugging Protocol>` (RDP) by implementing a subset of the +`Chrome DevTools Protocol`_ (CDP). + +To use Firefox remote protocol with Fission, CDP client authors should read the +`Required Preferences`_ page. + +.. _Chrome DevTools Protocol: https://chromedevtools.github.io/devtools-protocol/ +.. _Required Preferences: /remote/cdp/RequiredPreferences.html + +.. toctree:: + :maxdepth: 1 + + Usage.md + Architecture.md + PuppeteerVendor.md + RequiredPreferences.md |