diff options
Diffstat (limited to 'gfx/docs/AsyncPanZoom.rst')
| -rw-r--r-- | gfx/docs/AsyncPanZoom.rst | 929 |
1 files changed, 929 insertions, 0 deletions
diff --git a/gfx/docs/AsyncPanZoom.rst b/gfx/docs/AsyncPanZoom.rst new file mode 100644 index 0000000000..39bdc48c7d --- /dev/null +++ b/gfx/docs/AsyncPanZoom.rst @@ -0,0 +1,929 @@ +.. _apz: + +Asynchronous Panning and Zooming +================================ + +**This document is a work in progress. Some information may be missing +or incomplete.** + +.. image:: AsyncPanZoomArchitecture.png + +Goals +----- + +We need to be able to provide a visual response to user input with +minimal latency. In particular, on devices with touch input, content +must track the finger exactly while panning, or the user experience is +very poor. According to the UX team, 120ms is an acceptable latency +between user input and response. + +Context and surrounding architecture +------------------------------------ + +The fundamental problem we are trying to solve with the Asynchronous +Panning and Zooming (APZ) code is that of responsiveness. By default, +web browsers operate in a “game loop” that looks like this: + +:: + + while true: + process input + do computations + repaint content + display repainted content + +In browsers the “do computation” step can be arbitrarily expensive +because it can involve running event handlers in web content. Therefore, +there can be an arbitrary delay between the input being received and the +on-screen display getting updated. + +Responsiveness is always good, and with touch-based interaction it is +even more important than with mouse or keyboard input. In order to +ensure responsiveness, we split the “game loop” model of the browser +into a multithreaded variant which looks something like this: + +:: + + Thread 1 (compositor thread) + while true: + receive input + send a copy of input to thread 2 + adjust rendered content based on input + display adjusted rendered content + + Thread 2 (main thread) + while true: + receive input from thread 1 + do computations + rerender content + update the copy of rendered content in thread 1 + +This multithreaded model is called off-main-thread compositing (OMTC), +because the compositing (where the content is displayed on-screen) +happens on a separate thread from the main thread. Note that this is a +very very simplified model, but in this model the “adjust rendered +content based on input” is the primary function of the APZ code. + +A couple of notes on APZ's relationship to other browser architecture +improvements: + +1. Due to Electrolysis (e10s), Site Isolation (Fission), and GPU Process + isolation, the above two threads often actually run in different + processes. APZ is largely agnostic to this, as all communication + between the two threads for APZ purposes happens using an IPC layer + that abstracts over communication between threads vs. processes. +2. With the WebRender graphics backend, part of the rendering pipeline is + also offloaded from the main thread. In this architecture, the + information sent from the main thread consists of a display list, and + scrolling-related metadata referencing content in that display list. + The metadata is kept in a queue until the display list undergoes an + additional rendering step in the compositor (scene building). At this + point, we are ready to tell APZ about the new content and have it + start applying adjustments to it, as further rendering steps beyond + scene building are done synchronously on each composite. + +The compositor in theory can continuously composite previously rendered +content (adjusted on each composite by APZ) to the screen while the +main thread is busy doing other things and rendering new content. + +The APZ code takes the input events that are coming in from the hardware +and uses them to figure out what the user is trying to do (e.g. pan the +page, zoom in). It then expresses this user intention in the form of +translation and/or scale transformation matrices. These transformation +matrices are applied to the rendered content at composite time, so that +what the user sees on-screen reflects what they are trying to do as +closely as possible. + +Technical overview +------------------ + +As per the heavily simplified model described above, the fundamental +purpose of the APZ code is to take input events and produce +transformation matrices. This section attempts to break that down and +identify the different problems that make this task non-trivial. + +Checkerboarding +~~~~~~~~~~~~~~~ + +The area of page content for which a display list is built and sent to +the compositor is called the “displayport”. The APZ code is responsible +for determining how large the displayport should be. On the one hand, we +want the displayport to be as large as possible. At the very least it +needs to be larger than what is visible on-screen, because otherwise, as +soon as the user pans, there will be some unpainted area of the page +exposed. However, we cannot always set the displayport to be the entire +page, because the page can be arbitrarily long and this would require an +unbounded amount of memory to store. Therefore, a good displayport size +is one that is larger than the visible area but not so large that it is a +huge drain on memory. Because the displayport is usually smaller than the +whole page, it is always possible for the user to scroll so fast that +they end up in an area of the page outside the displayport. When this +happens, they see unpainted content; this is referred to as +“checkerboarding”, and we try to avoid it where possible. + +There are many possible ways to determine what the displayport should be +in order to balance the tradeoffs involved (i.e. having one that is too +big is bad for memory usage, and having one that is too small results in +excessive checkerboarding). Ideally, the displayport should cover +exactly the area that we know the user will make visible. Although we +cannot know this for sure, we can use heuristics based on current +panning velocity and direction to ensure a reasonably-chosen displayport +area. This calculation is done in the APZ code, and a new desired +displayport is frequently sent to the main thread as the user is panning +around. + +Multiple scrollable elements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Consider, for example, a scrollable page that contains an iframe which +itself is scrollable. The iframe can be scrolled independently of the +top-level page, and we would like both the page and the iframe to scroll +responsively. This means that we want independent asynchronous panning +for both the top-level page and the iframe. In addition to iframes, +elements that have the overflow:scroll CSS property set are also +scrollable. In the display list, scrollable elements are arranged in a +tree structure, and in the APZ code we have a matching tree of +AsyncPanZoomController (APZC) objects, one for each scrollable element. +To manage this tree of APZC instances, we have a single APZCTreeManager +object. Each APZC is relatively independent and handles the scrolling for +its associated scrollable element, but there are some cases in which they +need to interact; these cases are described in the sections below. + +Hit detection +~~~~~~~~~~~~~ + +Consider again the case where we have a scrollable page that contains an +iframe which itself is scrollable. As described above, we will have two +APZC instances - one for the page and one for the iframe. When the user +puts their finger down on the screen and moves it, we need to do some +sort of hit detection in order to determine whether their finger is on +the iframe or on the top-level page. Based on where their finger lands, +the appropriate APZC instance needs to handle the input. + +This hit detection is done by APZCTreeManager in collaboration with +WebRender, which has more detailed information about the structure of +the page content than is stored in APZ directly. See +:ref:`this section <wr-hit-test-details>` for more details. + +Also note that for some types of input (e.g. when the user puts two +fingers down to do a pinch) we do not want the input to be “split” +across two different APZC instances. In the case of a pinch, for +example, we find a “common ancestor” APZC instance - one that is +zoomable and contains all of the touch input points, and direct the +input to that APZC instance. + +Scroll Handoff +~~~~~~~~~~~~~~ + +Consider yet again the case where we have a scrollable page that +contains an iframe which itself is scrollable. Say the user scrolls the +iframe so that it reaches the bottom. If the user continues panning on +the iframe, the expectation is that the top-level page will start +scrolling. However, as discussed in the section on hit detection, the +APZC instance for the iframe is separate from the APZC instance for the +top-level page. Thus, we need the two APZC instances to communicate in +some way such that input events on the iframe result in scrolling on the +top-level page. This behaviour is referred to as “scroll handoff” (or +“fling handoff” in the case where analogous behaviour results from the +scrolling momentum of the page after the user has lifted their finger). + +.. _input-event-untransformation: + +Input event untransformation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The APZC architecture by definition results in two copies of a “scroll +position” for each scrollable element. There is the original copy on the +main thread that is accessible to web content and the layout and +painting code. And there is a second copy on the compositor side, which +is updated asynchronously based on user input, and corresponds to what +the user visually sees on the screen. Although these two copies may +diverge temporarily, they are reconciled periodically. In particular, +they diverge while the APZ code is performing an async pan or zoom +action on behalf of the user, and are reconciled when the APZ code +requests a repaint from the main thread. + +Because of the way input events are represented, this has some +unfortunate consequences. Input event coordinates are represented +relative to the device screen - so if the user touches at the same +physical spot on the device, the same input events will be delivered +regardless of the content scroll position. When the main thread receives +a touch event, it combines that with the content scroll position in order +to figure out what DOM element the user touched. However, because we now +have two different scroll positions, this process may not work perfectly. +A concrete example follows: + +Consider a device with screen size 600 pixels tall. On this device, a +user is viewing a document that is 1000 pixels tall, and that is +scrolled down by 200 pixels. That is, the vertical section of the +document from 200px to 800px is visible. Now, if the user touches a +point 100px from the top of the physical display, the hardware will +generate a touch event with y=100. This will get sent to the main +thread, which will add the scroll position (200) and get a +document-relative touch event with y=300. This new y-value will be used +in hit detection to figure out what the user touched. If the document +had a absolute-positioned div at y=300, then that would receive the +touch event. + +Now let us add some async scrolling to this example. Say that the user +additionally scrolls the document by another 10 pixels asynchronously +(i.e. only on the compositor thread), and then does the same touch +event. The same input event is generated by the hardware, and as before, +the document will deliver the touch event to the div at y=300. However, +visually, the document is scrolled by an additional 10 pixels so this +outcome is wrong. What needs to happen is that the APZ code needs to +intercept the touch event and account for the 10 pixels of asynchronous +scroll. Therefore, the input event with y=100 gets converted to y=110 in +the APZ code before being passed on to the main thread. The main thread +then adds the scroll position it knows about and determines that the +user touched at a document-relative position of y=310. + +Analogous input event transformations need to be done for horizontal +scrolling and zooming. + +Content independently adjusting scrolling +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As described above, there are two copies of the scroll position in the +APZ architecture - one on the main thread and one on the compositor +thread. Usually for architectures like this, there is a single “source +of truth” value and the other value is simply a copy. However, in this +case that is not easily possible to do. The reason is that both of these +values can be legitimately modified. On the compositor side, the input +events the user is triggering modify the scroll position, which is then +propagated to the main thread. However, on the main thread, web content +might be running Javascript code that programmatically sets the scroll +position (via window.scrollTo, for example). Scroll changes driven from +the main thread are just as legitimate and need to be propagated to the +compositor thread, so that the visual display updates in response. + +Because the cross-thread messaging is asynchronous, reconciling the two +types of scroll changes is a tricky problem. Our design solves this +using various flags and generation counters. The general heuristic we +have is that content-driven scroll position changes (e.g. scrollTo from +JS) are never lost. For instance, if the user is doing an async scroll +with their finger and content does a scrollTo in the middle, then some +of the async scroll would occur before the “jump” and the rest after the +“jump”. + +Content preventing default behaviour of input events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Another problem that we need to deal with is that web content is allowed +to intercept touch events and prevent the “default behaviour” of +scrolling. This ability is defined in web standards and is +non-negotiable. Touch event listeners in web content are allowed call +preventDefault() on the touchstart or first touchmove event for a touch +point; doing this is supposed to “consume” the event and prevent +touch-based panning. As we saw in a previous section, the input event +needs to be untransformed by the APZ code before it can be delivered to +content. But, because of the preventDefault problem, we cannot fully +process the touch event in the APZ code until content has had a chance +to handle it. + +To balance the needs of correctness (which calls for allowing web content +to successfully prevent default handling of events if it wishes to) and +responsiveness (which calls for avoiding blocking on web content +Javascript for a potentially-unbounded amount of time before reacting to +an event), APZ gives web content a "deadline" to process the event and +tell APZ whether preventDefault() was called on the event. The deadline +is 400ms from the time APZ receives the event on desktop, and 600ms on +mobile. If web content is able to process the event before this deadline, +the decision to preventDefault() the event or not will be respected. If +web content fails to process the event before the deadline, APZ assumes +preventDefault() will not be called and goes ahead and processes the +event. + +To implement this, upon receiving a touch event, APZ immediately returns +an untransformed version that can be dispatched to content. It also +schedules the 400ms or 600ms timeout. There is an API that allows the +main-thread event dispatching code to notify APZ as to whether or not the +default action should be prevented. If the APZ content response timeout +expires, or if the main-thread event dispatching code notifies the APZ of +the preventDefault status, then the APZ continues with the processing of +the events (which may involve discarding the events). + +To limit the responsiveness impact of this round-trip to content, APZ +tries to identify cases where it can rule out preventDefault() as a +possible outcome. To this end, the hit-testing information sent to the +compositor includes information about which regions of the page are +occupied by elements that have a touch event listener. If an event +targets an area outside of these regions, preventDefault() can be ruled +out, and the round-trip skipped. + +Additionally, recent enhancements to web standards have given page +authors new tools that can further limit the responsiveness impact of +preventDefault(): + +1. Event listeners can be registered as "passive", which means they + are not allowed to call preventDefault(). Authors can use this flag + when writing listeners that only need to observe the events, not alter + their behaviour via preventDefault(). The presence of passive event + listeners does not cause APZ to perform the content round-trip. +2. If page authors wish to disable certain types of touch interactions + completely, they can use the ``touch-action`` CSS property from the + pointer-events spec to do so declaratively, instead of registering + event listeners that call preventDefault(). Touch-action flags are + also included in the hit-test information sent to the compositor, and + APZ uses this information to respect ``touch-action``. (Note that the + touch-action information sent to the compositor is not always 100% + accurate, and sometimes APZ needs to fall back on asking the main + thread for touch-action information, which again involves a + round-trip.) + +Other event types +~~~~~~~~~~~~~~~~~ + +The above sections talk mostly about touch events, but over time APZ has +been extended to handle a variety of other event types, such as trackpad +and mousewheel scrolling, scrollbar thumb dragging, and keyboard +scrolling in some cases. Much of the above applies to these other event +types too (for example, wheel events can be prevent-defaulted as well). + +Importantly, the "untransformation" described above needs to happen even +for event types which are not handled in APZ, such as mouse click events, +since async scrolling can still affect the correct targeting of such +events. + + +Technical details +----------------- + +This section describes various pieces of the APZ code, and goes into +more specific detail on APIs and code than the previous sections. The +primary purpose of this section is to help people who plan on making +changes to the code, while also not going into so much detail that it +needs to be updated with every patch. + +Overall flow of input events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section describes how input events flow through the APZ code. + +Disclaimer: some details in this section are out of date (for example, +it assumes the case where the main thread and compositor thread are +in the same process, which is rarely the case these days, so in practice +e.g. steps 6 and 8 involve IPC, not just "stack unwinding"). + +1. Input events arrive from the hardware/widget code into the APZ via + APZCTreeManager::ReceiveInputEvent. The thread that invokes this is + called the "controller thread", and may or may not be the same as the + Gecko main thread. +2. Conceptually the first thing that the APZCTreeManager does is to + associate these events with “input blocks”. An input block is a set + of events that share certain properties, and generally are intended + to represent a single gesture. For example with touch events, all + events following a touchstart up to but not including the next + touchstart are in the same block. All of the events in a given block + will go to the same APZC instance and will either all be processed + or all be dropped. +3. Using the first event in the input block, the APZCTreeManager does a + hit-test to see which APZC it hits. If no APZC is hit, the events are + discarded and we jump to step 6. Otherwise, the input block is tagged + with the hit APZC as a tentative target and put into a global APZ + input queue. In addition the target APZC, the result of the hit test + also includes whether the input event landed on a "dispatch-to-content" + region. These are regions of the page where there is something going + on that requires dispatching the event to content and waiting for + a response _before_ processing the event in APZ; an example of this + is a region containing an element with a non-passive event listener, + as described above. (TODO: Add a section that talks about the other + uses of the dispatch-to-content mechanism.) +4. + + i. If the input events landed outside a dispatch-to-content region, + any available events in the input block are processed. These may + trigger behaviours like scrolling or tap gestures. + ii. If the input events landed inside a dispatch-to-content region, + the events are left in the queue and a timeout is initiated. If + the timeout expires before step 9 is completed, the APZ assumes + the input block was not cancelled and the tentative target is + correct, and processes them as part of step 10. + +5. The call stack unwinds back to APZCTreeManager::ReceiveInputEvent, + which does an in-place modification of the input event so that any + async transforms are removed. +6. The call stack unwinds back to the widget code that called + ReceiveInputEvent. This code now has the event in the coordinate + space Gecko is expecting, and so can dispatch it to the Gecko main + thread. +7. Gecko performs its own usual hit-testing and event dispatching for + the event. As part of this, it records whether any touch listeners + cancelled the input block by calling preventDefault(). It also + activates inactive scrollframes that were hit by the input events. +8. The call stack unwinds back to the widget code, which sends two + notifications to the APZ code on the controller thread. The first + notification is via APZCTreeManager::ContentReceivedInputBlock, and + informs the APZ whether the input block was cancelled. The second + notification is via APZCTreeManager::SetTargetAPZC, and informs the + APZ of the results of the Gecko hit-test during event dispatch. Note + that Gecko may report that the input event did not hit any + scrollable frame at all. The SetTargetAPZC notification happens only + once per input block, while the ContentReceivedInputBlock + notification may happen once per block, or multiple times per block, + depending on the input type. +9. + + i. If the events were processed as part of step 4(i), the + notifications from step 8 are ignored and step 10 is skipped. + ii. If events were queued as part of step 4(ii), and steps 5-8 + complete before the timeout, the arrival of both notifications + from step 8 will mark the input block ready for processing. + iii. If events were queued as part of step 4(ii), but steps 5-8 take + longer than the timeout, the notifications from step 8 will be + ignored and step 10 will already have happened. + +10. If events were queued as part of step 4(ii) they are now either + processed (if the input block was not cancelled and Gecko detected a + scrollframe under the input event, or if the timeout expired) or + dropped (all other cases). Note that the APZC that processes the + events may be different at this step than the tentative target from + step 3, depending on the SetTargetAPZC notification. Processing the + events may trigger behaviours like scrolling or tap gestures. + +If the CSS touch-action property is enabled, the above steps are +modified as follows: + +* In step 4, the APZC also requires the allowed touch-action behaviours + for the input event. This might have been determined as part of the + hit-test in APZCTreeManager; if not, the events are queued. +* In step 6, the widget code determines the content element at the point + under the input element, and notifies the APZ code of the allowed + touch-action behaviours. This notification is sent via a call to + APZCTreeManager::SetAllowedTouchBehavior on the input thread. +* In step 9(ii), the input block will only be marked ready for processing + once all three notifications arrive. + +Threading considerations +^^^^^^^^^^^^^^^^^^^^^^^^ + +The bulk of the input processing in the APZ code happens on what we call +“the controller thread”. In practice the controller thread could be the +Gecko main thread, the compositor thread, or some other thread. There are +obvious downsides to using the Gecko main thread - that is,“asynchronous” +panning and zooming is not really asynchronous as input events can only +be processed while Gecko is idle. In an e10s environment, using the Gecko +main thread of the chrome process is acceptable, because the code running +in that process is more controllable and well-behaved than arbitrary web +content. Using the compositor thread as the controller thread could work +on some platforms, but may be inefficient on others. For example, on +Android (Fennec) we receive input events from the system on a dedicated +UI thread. We would have to redispatch the input events to the compositor +thread if we wanted to the input thread to be the same as the compositor +thread. This introduces a potential for higher latency, particularly if +the compositor does any blocking operations - blocking SwapBuffers +operations, for example. As a result, the APZ code itself does not assume +that the controller thread will be the same as the Gecko main thread or +the compositor thread. + +Active vs. inactive scrollframes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The number of scrollframes on a page is potentially unbounded. However, +we do not want to create a separate displayport for each scrollframe +right away, as this would require large amounts of memory. Therefore, +scrollframes as designated as either “active” or “inactive”. Active +scrollframes get a displayport, and an APZC on the compositor side. +Inactive scrollframes do not get a displayport (a display list is only +built for their viewport, i.e. what is currently visible) and do not get +an APZC. + +Consider a page with a scrollframe that is initially inactive. This +scroll frame does not get an APZC, and therefore events targeting it will +target the APZC for the nearest active scrollable ancestor (let's call it +P; note, the rootmost scroll frame in a given process is always active). +However, the presence of the inactive scroll frame is reflected by a +dispatch-to-content region that prevents events over the frame from +erroneously scrolling P. + +When the user starts interacting with that content, the hit-test in the +APZ code hits the dispatch-to-content region of P. The input block +therefore has a tentative target of P when it goes into step 4(ii) in the +flow above. When gecko processes the input event, it must detect the +inactive scrollframe and activate it, as part of step 7. Finally, the +widget code sends the SetTargetAPZC notification in step 8 to notify the +APZ that the input block should really apply to this new APZC. An issue +here is that the transaction containing metadata for the newly active +scroll frame must reach the compositor and APZ before the SetTargetAPZC +notification. If this does not occur within the 400ms timeout, the APZ +code will be unable to update the tentative target, and will continue to +use P for that input block. Input blocks that start after the transaction +will get correctly routed to the new scroll frame as there will now be an +APZC instance for the active scrollframe. + +This model implies that when the user initially attempts to scroll an +inactive scrollframe, it may end up scrolling an ancestor scrollframe. +Only after the round-trip to the gecko thread is complete is there an +APZC for async scrolling to actually occur on the scrollframe itself. At +that point the scrollframe will start receiving new input blocks and will +scroll normally. + +Note: with Fission (where inactive scroll frames would make it impossible +to target the correct process in all situations; see +:ref:`this section <fission-hit-testing>` for more details) and WebRender +(which makes displayports more lightweight as the actual rendering is +offloaded to the compositor and can be done on demand), inactive scroll +frames are being phased out, and we are moving towards a model where all +scroll frames with nonempty scroll ranges are active and get a +displayport and an APZC. To conserve memory, displayports for scroll +frames which have not been recently scrolled are kept to a "minimal" size +equal to the viewport size. + +WebRender Integration +~~~~~~~~~~~~~~~~~~~~~ + +This section describes how APZ interacts with the WebRender graphics +backend. + +Note that APZ predates WebRender, having initially been written to work +with the earlier Layers graphics backend. The design of Layers has +influenced APZ significantly, and this still shows in some places in the +code. Now that the Layers backend has been removed, there may be +opportunities to streamline the interaction between APZ and WebRender. + + +HitTestingTree +^^^^^^^^^^^^^^ + +The APZCTreeManager keeps as part of its internal state a tree of +HitTestingTreeNode instances. This is referred to as the HitTestingTree. + +The main purpose of the HitTestingTree is to model the spatial +relationships between content that's affected by async scrolling. Tree +nodes fall roughly into the following categories: + +* Nodes representing scrollable content in an active scroll frame. These + nodes are associated with the scroll frame's APZC. +* Nodes representing other content that may move in special ways in + response to async scrolling, such as fixed content, sticky content, and + scrollbars. +* (Non-leaf) nodes which do not represent any content, just metadata + (e.g. a transform) that applies to its descendant nodes. + +An APZC may be associated with multiple nodes, if e.g. a scroll frame +scrolls two pieces of content that are interleaved with non-scrolling +content. + +Arranging these nodes in a tree allows modelling relationships such as +what content is scrolled by a given scroll frame, what the scroll handoff +relationships are between APZCs, and what content is subject to what +transforms. + +An additional use of the HitTestingTree is to allow APZ to keep content +processes up to date about enclosing transforms that they are subject to. +See :ref:`this section <sending-transforms-to-content-processes>` for +more details. + +(In the past, with the Layers backend, the HitTestingTree was also used +for compositor hit testing, hence the name. This is no longer the case, +and there may be opportunities to simplify the tree as a result.) + +The HitTestingTree is created from another tree data structure called +WebRenderScrollData. The relevant types here are: + +* WebRenderScrollData which stores the entire tree. +* WebRenderLayerScrollData, which represents a single "layer" of content, + i.e. a group of display items that move together when scrolling (or + metadata applying to a subtree of such layers). In the Layers backend, + such content would be rendered into a single texture which could then + be moved asynchronously at composite time. Since a layer of content can + be scrolled by multiple (nested) scroll frames, a + WebRenderLayerScrollData may contain scroll metadata for more than one + scroll frame. +* WebRenderScrollDataWrapper, which wraps WebRenderLayerScrollData + but "expanded" in a way that each node only stores metadata for + a single scroll frame. WebRenderScrollDataWrapper nodes have a + 1:1 correspondence with HitTestingTreeNodes. + +It's not clear whether the distinction between WebRenderLayerScrollData +and WebRenderScrollDataWrapper is still useful in a WebRender-only world. +The code could potentially be revised such that we directly build and +store nodes of a single type with the behaviour of +WebRenderScrollDataWrapper. + +The WebRenderScrollData structure is built on the main thread, and +then shipped over IPC to the compositor where it's used to construct +the HitTestingTree. + +WebRenderScrollData is built in WebRenderCommandBuilder, during the +same traversal of the Gecko display list that is used to build the +WebRender display list. As of this writing, the architecture for this is +that, as we walk the Gecko display list, we query it to see if it +contains any information that APZ might need to know (e.g. CSS +transforms) via a call to ``nsDisplayItem::UpdateScrollData(nullptr, +nullptr)``. If this call returns true, we create a +WebRenderLayerScrollData instance for the item, and populate it with the +necessary information in ``WebRenderLayerScrollData::Initialize``. We also +create WebRenderLayerScrollData instances if we detect (via ASR changes) +that we are now processing a Gecko display item that is in a different +scrollframe than the previous item. + +The main sources of complexity in this code come from: + +1. Ensuring the ScrollMetadata instances end on the proper + WebRenderLayerScrollData instances (such that every path from a leaf + WebRenderLayerScrollData node to the root has a consistent ordering of + scrollframes without duplications). +2. The deferred-transform optimization that is described in more detail + at the declaration of ``StackingContextHelper::mDeferredTransformItem``. + +.. _wr-hit-test-details: + +Hit-testing +^^^^^^^^^^^ + +Since the HitTestingTree is not used for actual hit-testing purposes +with the WebRender backend (see previous section), this section describes +how hit-testing actually works with WebRender. + +The Gecko display list contains display items +(``nsDisplayCompositorHitTestInfo``) that store hit-testing state. These +items implement the ``CreateWebRenderCommands`` method and generate a "hit-test +item" into the WebRender display list. This is basically just a rectangle +item in the WebRender display list that is no-op for painting purposes, +but contains information that should be returned by the hit-test (specifically +the hit info flags and the scrollId of the enclosing scrollframe). The +hit-test item gets clipped and transformed in the same way that all the other +items in the WebRender display list do, via clip chains and enclosing +reference frame/stacking context items. + +When WebRender needs to do a hit-test, it goes through its display list, +taking into account the current clips and transforms, adjusted for the +most recent async scroll/zoom, and determines which hit-test item(s) are under +the target point, and returns those items. APZ can then take the frontmost +item from that list (or skip over it if it happens to be inside a OOP +subdocument that's ``pointer-events:none``) and use that as the hit target. +Note that the hit-test uses the last transform provided by the +``SampleForWebRender`` API (see next section) which generally reflects the +last composite, and doesn't take into account further changes to the +transforms that have occurred since then. In practice, we should be +compositing frequently enough that this doesn't matter much. + +When debugging hit-test issues, it is often useful to apply the patches +on bug 1656260, which introduce a guid on Gecko display items and propagate +it all the way through to where APZ gets the hit-test result. This allows +answering the question "which nsDisplayCompositorHitTestInfo was responsible +for this hit-test result?" which is often a very good first step in +solving the bug. From there, one can determine if there was some other +display item in front that should have generated a +nsDisplayCompositorHitTestInfo but didn't, or if display item itself had +incorrect information. The second patch on that bug further allows exposing +hand-written debug info to the APZ code, so that the WR hit-testing +mechanism itself can be more effectively debugged, in case there is a problem +with the WR display items getting improperly transformed or clipped. + +The information returned by WebRender to APZ in response to the hit test +is enough for APZ to identify a HitTestingTreeNode as the target of the +event. APZ can then take actions such as scrolling the target node's +associated APZC, or other appropriate actions (e.g. initiating a scrollbar +drag if a scrollbar thumb node was targeted by a mouse-down event). + +Sampling +^^^^^^^^ + +The compositing step needs to read the latest async transforms from APZ +in order to ensure scrollframes are rendered at the right position. The API for this is +exposed via the ``APZSampler`` class. When WebRender is ready to do a composite, +it invokes ``APZSampler::SampleForWebRender``. In here, APZ gathers all async +transforms that WebRender needs to know about, including transforms to apply +to scrolled content, fixed and sticky content, and scrollbar thumbs. + +Along with sampling the APZ transforms, the compositor also triggers APZ +animations to advance to the next timestep (usually the next vsync). This +happens just before reading the APZ transforms. + +Fission Integration +~~~~~~~~~~~~~~~~~~~ + +This section describes how APZ interacts with the Fission (Site Isolation) +project. + +Introduction +^^^^^^^^^^^^ + +Fission is an architectural change motivated by security considerations, +where web content from each origin is isolated in its own process. Since +a page can contain a mixture of content from different origins (for +example, the top level page can be content from origin A, and it can +contain an iframe with content from origin B), that means that rendering +and interacting with a page can now involve coordination between APZ and +multiple content processes. + +.. _fission-hit-testing: + +Content Process Selection for Input Events +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Input events are initially received in the browser's parent process. +With Fission, the browser needs to decide which of possibly several +content processes an event is targeting. + +Since process boundaries correspond to iframe (subdocument) boundaries, +and every (html) document has a root scroll frame, process boundaries are +therefore also scroll frame boundaries. Since APZ already needs a hit +test mechanism to be able to determine which scroll frame an event +targets, this hit test mechanism was a good fit to also use to determine +which content process an event targets. + +APZ's hit test was therefore expanded to serve this purpose as well. This +mostly required only minor modifications, such as making sure that APZ +knows about the root scroll frames of iframes even if they're not +scrollable. Since APZ already needs to process all input events to +potentially apply :ref:`untransformations <input-event-untransformation>` +related to async scrolling, as part of this process it now also labels +input events with information identifying which content process they +target. + +Hit Testing Accuracy +^^^^^^^^^^^^^^^^^^^^ + +Prior to Fission, APZ's hit test could afford to be somewhat inaccurate, +as it could fall back on the dispatch-to-content mechanism to wait for +a more accurate answer from the main thread if necessary, suffering a +performance cost only (not a correctness cost). + +With Fission, an inaccurate compositor hit test now implies a correctness +cost, as there is no cross-process main-thread fallback mechanism. +(Such a mechanism was considered, but judged to require too much +complexity and IPC traffic to be worth it.) + +Luckily, with WebRender the compositor has much more detailed information +available to use for hit testing than it did with Layers. For example, +the compositor can perform accurate hit testing even in the presence of +irregular shapes such as rounded corners. + +APZ leverages WebRender's more accurate hit testing ability to aim to +accurately select the target process (and target scroll frame) for an +event in general. + +One consequence of this is that the dispatch-to-content mechanism is now +used less often than before (its primary remaining use is handling +`preventDefault()`). + +.. _sending-transforms-to-content-processes: + +Sending Transforms To Content Processes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Content processes sometimes need to be able to convert between screen +coordinates and their local coordinates. To do this, they need to know +about any transforms that their containing iframe and its ancestors are +subject to, including async transforms (particularly in cases where the +async transforms persist for more than just a few frames). + +APZ has information about these transforms in its HitTestingTree. With +Fission, APZ periodically sends content processes information about these +transforms so that they are kept relatively up to date. + +Testing +------- + +APZ makes use of several test frameworks to verify the expected behavior +is seen. + +Mochitest +~~~~~~~~~ + +The APZ specific mochitests are useful when specific gestures or events need to be tested +with specific content. The APZ mochitests are located in `gfx/layers/apz/test/mochitest`_. +To run all of the APZ mochitests, run something like the following: + +:: + + ./mach mochitest ./gfx/layers/apz/test/mochitest + +The APZ mochitests are often organized as subtests that run in a group. For example, +the `test_group_hittest-2.html`_ contains >20 subtests like +`helper_hittest_overscroll.html`_. When working on a specific subtest, it is often +helpful to use the `apz.subtest` preference to filter the subtests run to just the +tests you are working on. For example, the following would only run the +`helper_hittest_overscroll.html`_ subtest of the `test_group_hittest-2.html`_ group. + +:: + + ./mach mochitest --setpref apz.subtest=helper_hittest_overscroll.html \ + ./gfx/layers/apz/test/mochitest/test_group_hittest-2.html + +For more information on mochitest, see the `Mochitest Documentation`_. + +.. _gfx/layers/apz/test/mochitest: https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/mochitest +.. _test_group_hittest-2.html: https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/mochitest/test_group_hittest-2.html +.. _helper_hittest_overscroll.html: https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/mochitest/helper_hittest_overscroll.html +.. _Mochitest Documentation: /testing/mochitest-plain/index.html + +GTest +~~~~~ + +The APZ specific GTests can be found in `gfx/layers/apz/test/gtest/`_. To run +these tests, run something like the following: + +:: + + ./mach gtest "APZ*" + +For more information, see the `GTest Documentation`_. + +.. _GTest Documentation: /gtest/index.html +.. _gfx/layers/apz/test/gtest/: https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/gtest/ + +Reftests +~~~~~~~~ + +The APZ reftests can be found in `layout/reftests/async-scrolling/`_ and +`gfx/layers/apz/test/reftest`_. To run the relevant reftests for APZ, run +a large portion of the APZ reftests, run something like the following: + +:: + + ./mach reftest ./layout/reftests/async-scrolling/ + +Useful information about the reftests can be found in the `Reftest Documentation`_. + +There is no defined process for choosing which directory the APZ reftests +should be placed in, but in general reftests should exist where other +similar tests do. + +.. _layout/reftests/async-scrolling/: https://searchfox.org/mozilla-central/source/layout/reftests/async-scrolling/ +.. _gfx/layers/apz/test/reftest: https://searchfox.org/mozilla-central/source/gfx/layers/apz/test/reftest/ +.. _Reftest Documentation: /layout/Reftest.html + +Threading / Locking Overview +---------------------------- + +Threads +~~~~~~~ + +There are three threads relevant to APZ: the **controller thread**, +the **updater thread**, and the **sampler thread**. This table lists +which threads play these roles on each platform / configuration: + +===================== ============= ============== ============= +APZ Thread Name Desktop Desktop+GPU Android +===================== ============= ============== ============= +**controller thread** UI main GPU main Java UI +**updater thread** SceneBuilder SceneBuilder SceneBuilder +**sampler thread** RenderBackend RenderBackend RenderBackend +===================== ============= ============== ============= + +Locks +~~~~~ + +There are also a number of locks used in APZ code: + +======================= ============================== +Lock type How many instances +======================= ============================== +APZ tree lock one per APZCTreeManager +APZC map lock one per APZCTreeManager +APZC instance lock one per AsyncPanZoomController +APZ test lock one per APZCTreeManager +Checkerboard event lock one per AsyncPanZoomController +======================= ============================== + +Thread / Lock Ordering +~~~~~~~~~~~~~~~~~~~~~~ + +To avoid deadlocks, the threads and locks have a global **ordering** +which must be respected. + +Respecting the ordering means the following: + +- Let "A < B" denote that A occurs earlier than B in the ordering +- Thread T may only acquire lock L, if T < L +- A thread may only acquire lock L2 while holding lock L1, if L1 < L2 +- A thread may only block on a response from another thread T while holding a lock L, if L < T + +**The lock ordering is as follows**: + +1. UI main +2. GPU main (only if GPU process enabled) +3. Compositor thread +4. SceneBuilder thread +5. **APZ tree lock** +6. RenderBackend thread +7. **APZC map lock** +8. **APZC instance lock** +9. **APZ test lock** +10. **Checkerboard event lock** + +Example workflows +^^^^^^^^^^^^^^^^^ + +Here are some example APZ workflows. Observe how they all obey +the global thread/lock ordering. Feel free to add others: + +- **Input handling** (with GPU process): UI main -> GPU main -> APZ tree lock -> RenderBackend thread +- **Sync messages** in ``PCompositorBridge.ipdl``: UI main thread -> Compositor thread +- **GetAPZTestData**: Compositor thread -> SceneBuilder thread -> test lock +- **Scene swap**: SceneBuilder thread -> APZ tree lock -> RenderBackend thread +- **Updating hit-testing tree**: SceneBuilder thread -> APZ tree lock -> APZC instance lock +- **Updating APZC map**: SceneBuilder thread -> APZ tree lock -> APZC map lock +- **Sampling and animation deferred tasks** [1]_: RenderBackend thread -> APZC map lock -> APZC instance lock +- **Advancing animations**: RenderBackend thread -> APZC instance lock + +.. [1] It looks like there are two deferred tasks that actually need the tree lock, + ``AsyncPanZoomController::HandleSmoothScrollOverscroll`` and + ``AsyncPanZoomController::HandleFlingOverscroll``. We should be able to rewrite + these to use the map lock instead of the tree lock. + This will allow us to continue running the deferred tasks on the sampler + thread rather than having to bounce them to another thread. |