375 lines
19 KiB
ReStructuredText
375 lines
19 KiB
ReStructuredText
.. _components/pictureinpicture:
|
||
|
||
==================
|
||
Picture-in-Picture
|
||
==================
|
||
|
||
This component makes it possible for a ``<video>`` element on a web page to be played within
|
||
an always-on-top video player.
|
||
|
||
This documentation covers the architecture and inner workings of both the mechanism that
|
||
displays the ``<video>`` in the always-on-top video player, as well as the mechanism that
|
||
displays the Picture-in-Picture toggle that overlays ``<video>`` elements, which is the primary
|
||
method for launching the feature.
|
||
|
||
|
||
High-level overview
|
||
===================
|
||
|
||
The following diagram tries to illustrate the subcomponents, and how they interact with one another.
|
||
|
||
.. image:: PiP-diagram.svg
|
||
|
||
Let's suppose that the user has loaded a document with a ``<video>`` in it, and they decide to open
|
||
it in a Picture-in-Picture window. What happens?
|
||
|
||
First the ``PictureInPictureToggleChild`` component notices when ``<video>`` elements are added to the
|
||
DOM, and monitors the mouse as it moves around the document. Once the mouse intersects a ``<video>``,
|
||
``PictureInPictureToggleChild`` causes the Picture-in-Picture toggle to appear on that element.
|
||
|
||
If the user clicks on that toggle, then the ``PictureInPictureToggleChild`` dispatches a chrome-only
|
||
``MozTogglePictureInPicture`` event on the video, which is handled by the ``PictureInPictureLauncherChild`` actor
|
||
for that document. The reason for the indirection via the event is that the media context menu can also
|
||
trigger Picture-in-Picture by dispatching the same event on the video. Upon handling the event, the
|
||
``PictureInPictureLauncherChild`` actor then sends a ``PictureInPicture:Request`` message to the parent process.
|
||
The parent process opens up the always-on-top player window, with a remote ``<xul:browser>`` that runs in
|
||
the same content process as the original ``<video>``. The parent then sends a message to the player
|
||
window's remote ``<xul:browser>`` loaded in the player window. A ``PictureInPictureChild`` actor
|
||
is instantiated for the empty document loaded inside of the player window browser. This
|
||
``PictureInPictureChild`` actor constructs its own ``<video>`` element, and then tells Gecko to clone the
|
||
frames from the original ``<video>`` to the newly created ``<video>``.
|
||
|
||
At this point, the video is displaying in the Picture-in-Picture player window.
|
||
|
||
Next, we'll discuss the individual subcomponents, and how they operate at a more detailed level.
|
||
|
||
|
||
The Picture-in-Picture toggle
|
||
=============================
|
||
|
||
One of the primary challenges faced when developing this feature was the fact that, in practice, mouse
|
||
events tend not to reach ``<video>`` elements. This is usually because the ``<video>`` element is
|
||
contained within a hierarchy of other DOM elements that are capturing and handling any events that
|
||
come down. This often occurs on sites that construct their own video controls. This is why we cannot
|
||
simply use a ``mouseover`` event handler on the ``<video>`` UAWidget - on sites that do the event
|
||
capturing, we'll never receive those events and the toggle will not be accessible.
|
||
|
||
Other times, the problem is that the video is overlaid with a semi or fully transparent element
|
||
which captures any mouse events that would normally be dispatched to the underlying ``<video>``.
|
||
This can occur, for example, on sites that want to display an overlay when the video is paused.
|
||
|
||
To work around this problem, the `PictureInPictureToggleChild` actor class samples the latest
|
||
``mousemove`` event every ``MOUSEMOVE_PROCESSING_DELAY_MS`` milliseconds, and then calls
|
||
``nsIDOMWindowUtils.nodesFromRect`` with the ``aOnlyVisible`` argument to get the full
|
||
list of visible nodes that exist underneath a 1x1 rect positioned at the mouse cursor.
|
||
|
||
If a ``<video>`` is in that list, then we reach into its shadow root, and update some
|
||
attributes to tell it to maybe show the toggle.
|
||
|
||
The underlying ``UAWidget`` for the video is defined in ``videocontrols.js``, and ultimately
|
||
chooses whether or not to display the toggle based on the following heuristics:
|
||
|
||
1. Is the video less than 45 seconds?
|
||
2. Is either the width or the height of the video less than 160px?
|
||
3. Is the video silent?
|
||
|
||
If any of the above is true, the underlying ``UAWidget`` will hide the toggle, since it's
|
||
unlikely that the user will want to pop the video out into an always-on-top player window.
|
||
|
||
|
||
Video registration
|
||
==================
|
||
|
||
Sampling the latest ``mousemove`` event every ``MOUSEMOVE_PROCESSING_DELAY_MS`` is not free,
|
||
computationally speaking, so we only do this if there are one or more ``<video>`` elements
|
||
visible on the page. We use an ``IntersectionObserver`` to notice when there is a ``<video>``
|
||
within the viewport, and if there are 1 or more ``<video>`` elements visible, then we start
|
||
sampling the ``mousemove`` event.
|
||
|
||
Videos are added to the ``IntersectionObserver`` when they are added to the DOM by listening
|
||
for the ``UAWidgetSetupOrChange`` event. This is considered being "registered".
|
||
|
||
|
||
``docState``
|
||
============
|
||
|
||
``PictureInPictureChild.sys.mjs`` contains a ``WeakMap`` mapping ``document``'s to various information
|
||
that ``PictureInPictureToggleChild`` wants to retain for the lifetime of that ``document``. For
|
||
example, whether or not we're in the midst of handling the user clicking down on their pointer
|
||
device. Any state that needs to be remembered should be added to the ``docState`` ``WeakMap``.
|
||
|
||
|
||
Clicking on the toggle
|
||
======================
|
||
|
||
If the user clicks on the Picture-in-Picture toggle, we don't want the underlying webpage to
|
||
know that this happened, since this could result in unexpected behaviour, like a page
|
||
navigation (for example, if the ``<video>`` is a long-running advertisement that navigates
|
||
upon click).
|
||
|
||
To accomplish this, we listen for all events fired on a mouse click on the root window during
|
||
the capturing phase. This allows us to handle the events before they are dispatched to content.
|
||
|
||
The first event that is fired, ``pointerdown``, is captured, and we check the ``docState`` to see
|
||
whether or not we're showing a toggle on any videos. If so, we check the coordinates of that
|
||
toggle against the coordinates of the ``pointerdown`` event to determine if the user is clicking
|
||
on the toggle. If so, we set a flag in the ``docState`` so that any subsequent events from the
|
||
click (like ``mousedown``, ``mouseup``, ``pointerup``, ``click``) are captured and suppressed.
|
||
If the ``pointerdown`` event didn't occur within a toggle, we let the events pass through as
|
||
normal.
|
||
|
||
If we determine that the click has occurred on the toggle, a ``MozTogglePictureInPicture`` event
|
||
is dispatched on the underlying ``<video>``. This event is handled by the separate
|
||
``PictureInPictureLauncherChild`` class.
|
||
|
||
PictureInPictureLauncherChild
|
||
=============================
|
||
|
||
A small actor class whose only responsibility is to tell the parent process to open an always-on-top-window by sending a ``PictureInPicture:Request`` message to its parent actor.
|
||
|
||
Currently, this only occurs when a chrome-only ``MozTogglePictureInPicture`` event is dispatched by the ``PictureInPictureToggleChild`` when the user clicks the Picture-in-Picture toggle button
|
||
or uses the context-menu.
|
||
|
||
PictureInPictureChild
|
||
=====================
|
||
|
||
The ``PictureInPictureChild`` actor class will run in a content process containing a video, and is instantiated when the player window's `player.js` script runs its initialization. A ``PictureInPictureChild`` maps an individual ``<video>``
|
||
to a player window instance. It creates an always-on-top window, and sets up a new ``<video>`` inside of this window to clone frames from another ``<video>``
|
||
(which will be in the same process, and have its own ``PictureInPictureChild``). Creating this window also causes the new ``PictureInPictureChild`` to be created.
|
||
This instance will monitor the originating ``<video>`` for changes, and to receive commands from the player window if the user wants to control the ``<video>``.
|
||
|
||
PictureInPicture.sys.mjs
|
||
========================
|
||
|
||
This module runs in the parent process, and is also the scope where all ``PictureInPictureParent`` instances reside. ``PictureInPicture.sys.mjs``'s job is to send and receive messages from ``PictureInPictureChild`` instances, and to react appropriately.
|
||
|
||
Critically, ``PictureInPicture.sys.mjs`` is responsible for opening up the always-on-top player window, and passing the relevant information about the ``<video>`` to be displayed to it.
|
||
|
||
|
||
The Picture-in-Picture player window
|
||
====================================
|
||
|
||
The Picture-in-Picture player window is a chrome-privileged window that loads an XHTML document. That document contains a remote ``<browser>`` element which is repurposed during window initialization to load in the same content process as the originating ``<video>``.
|
||
|
||
The player window is where the player controls are defined, like "Play" and "Pause". When the user interacts with the player controls, a message is sent down to the appropriate ``PictureInPictureChild`` to call the appropriate method on the underlying ``<video>`` element in the originating tab.
|
||
|
||
|
||
Cloning the video frames
|
||
========================
|
||
|
||
While it appears as if the video is moving from the original ``<video>`` element to the player window, what's actually occurring is that the video frames are being *cloned* to the player window ``<video>`` element. This cloning is done at the platform level using a privileged method on the ``<video>`` element: ``cloneElementVisually``.
|
||
|
||
|
||
``cloneElementVisually``
|
||
------------------------
|
||
|
||
.. code-block:: js
|
||
|
||
Promise<void> video.cloneElementVisually(otherVideo);
|
||
|
||
This will clone the frames being decoded for ``video`` and display them on the ``otherVideo`` element as well. The returned Promise resolves once the cloning has successfully started.
|
||
|
||
|
||
``stopCloningElementVisually``
|
||
------------------------------
|
||
|
||
.. code-block:: js
|
||
|
||
void video.stopCloningElementVisually();
|
||
|
||
If ``video`` is being cloned visually to another element, calling this method will stop the cloning.
|
||
|
||
|
||
``isCloningElementVisually``
|
||
----------------------------
|
||
|
||
.. code-block:: js
|
||
|
||
boolean video.isCloningElementVisually;
|
||
|
||
A read-only value that returns ``true`` if ``video`` is being cloned visually.
|
||
|
||
Site-specific video wrappers
|
||
============================
|
||
|
||
A site-specific video wrapper allows for the creation of custom scripts that the Picture-in-Picture component can utilize when videos are loaded in specific domains. Currently, some uses of video wrappers include:
|
||
|
||
* Integration of captions and subtitles support on certain video streaming sites
|
||
* Fixing inconsistent video behaviour when using Picture-in-Picture controls
|
||
* Hiding the Picture-in-Picture toggle for videos on particular areas of a page, given a URL (rather than hiding the toggle for all videos on a page)
|
||
|
||
``PictureInPictureChildVideoWrapper`` and ``videoWrapperScriptPath``
|
||
--------------------------------------------------------------------
|
||
``PictureInPictureChildVideoWrapper`` is a special class that represents a video wrapper. It is defined in ``PictureInPictureChild.sys.mjs`` and maps to a ``videoWrapperScriptPath``, which is the path of the custom wrapper script to use.
|
||
``videoWrapperScriptPath`` is defined in `browser/extensions/pictureinpicture/data/picture_in_picture_overrides.js <https://searchfox.org/mozilla-central/source/browser/extensions/pictureinpicture/data/picture_in_picture_overrides.js>`_ for a domain,
|
||
and custom wrapper scripts are defined in `browser/extensions/pictureinpicture/video-wrappers <https://searchfox.org/mozilla-central/source/browser/extensions/pictureinpicture/video-wrappers>`_.
|
||
|
||
If a ``videoWrapperScriptPath`` is detected while initializing the Picture-in-Picture toggle or window, we immediately create a new instance of ``PictureInPictureChildVideoWrapper`` based on the given path, allowing us to run our custom scripts.
|
||
|
||
API
|
||
^^^
|
||
See the full list of methods at `API References <#toolkit-actors-pictureinpicturechild-jsm>`_.
|
||
|
||
Sandbox
|
||
^^^^^^^
|
||
Performing video control operations on the originating video requires executing code in the browser content. For security reasons, we utilize a *sandbox* to isolate these operations and prevent direct access to ``PictureInPictureChild``. In other words, we run content code within the sandbox itself.
|
||
However, it is necessary to waive :ref:`xray vision <Waiving_Xray_vision>` so that we can execute the video control operations. This is done by reading the wrapper’s ``.wrappedJSObject`` property.
|
||
|
||
Adding a new site-specific video wrapper
|
||
----------------------------------------
|
||
Creating a new wrapper script file
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
Add a new JS file for the new video wrapper in `browser/extensions/pictureinpicture/video-wrappers <https://searchfox.org/mozilla-central/source/browser/extensions/pictureinpicture/video-wrappers>`_.
|
||
The file must meet several requirements to get the wrapper working.
|
||
|
||
**Script file requirements**:
|
||
|
||
* Defined class ``PictureInPictureVideoWrapper``
|
||
* Assigned ``this.PictureInPictureVideoWrapper = PictureInPictureVideoWrapper``
|
||
|
||
**PictureInPictureVideoWrapper class requirements**:
|
||
|
||
* Implementation of at least one overridable method (see :ref:`picture_in_picture_child_video_wrapper_api`)
|
||
|
||
**Overriden method requirements**:
|
||
|
||
* Return value with a type that corresponds to ``validateRetVal`` in ``PictureInPictureChildVideoWrapper.#callWrapperMethod()``
|
||
|
||
Below is an example of a script file ``mock-wrapper.js`` that overrides an existing method ``setMuted()`` in ``PictureInPictureChildVideoWrapper``:
|
||
|
||
.. code-block:: js
|
||
|
||
// sample file `mock-wrapper.js`
|
||
class PictureInPictureVideoWrapper {
|
||
setMuted(video, shouldMute) {
|
||
if (video.muted !== shouldMute) {
|
||
let muteButton = document.querySelector("#player .mute-button");
|
||
if (muteButton) {
|
||
muteButton.click();
|
||
} else {
|
||
video.muted = shouldMute;
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
this.PictureInPictureVideoWrapper = PictureInPictureVideoWrapper
|
||
|
||
.. note::
|
||
If a new ``PictureInPictureChildVideoWrapper`` video control method is needed, see `Adding a new video control method`_.
|
||
|
||
Declaring ``videoWrapperScriptPath``
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
Declare a property ``videoWrapperScriptPath`` for the site at `browser/extensions/pictureinpicture/data/picture_in_picture_overrides.js <https://searchfox.org/mozilla-central/source/browser/extensions/pictureinpicture/data/picture_in_picture_overrides.js>`_:
|
||
|
||
.. code-block:: js
|
||
|
||
someWebsite: {
|
||
"https://*.somewebsite.com/*": {
|
||
videoWrapperScriptPath: "video-wrappers/mock-wrapper.js",
|
||
},
|
||
}
|
||
|
||
In this example, the URL pattern ``https://*.somewebsite.com/*`` is provided for a site named ``someWebsite``.
|
||
Picture-in-Picture checks for any overrides upon initialization, and it will load scripts specified by ``videoWrapperScriptPath``.
|
||
The scripts located at ``video-wrappers/mock-wrapper.js`` will therefore run whenever we view a video from a URL matching ``somewebsite.com``.
|
||
|
||
Registering the new wrapper in ``moz.build``
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
Any wrapper in the `video-wrappers` folder will be automatically included.
|
||
|
||
Adding a new video control method
|
||
---------------------------------
|
||
If none of the existing overridable methods in ``PictureInPictureChildVideoWrapper`` are applicable for a bug fix or feature enhancement,
|
||
we can create a new one by calling ``#callWrapperMethod()``. Below is an example of how we would define a new overridable method ``setMuted()``:
|
||
|
||
.. code-block:: js
|
||
|
||
// class PictureInPictureChildVideoWrapper in PictureInPictureChild.sys.mjs
|
||
setMuted(video, shouldMute) {
|
||
return this.#callWrapperMethod({
|
||
name: "setMuted",
|
||
args: [video, shouldMute],
|
||
fallback: () => {
|
||
video.muted = shouldMute;
|
||
},
|
||
validateRetVal: retVal => retVal == null,
|
||
});
|
||
}
|
||
|
||
The new method passes to ``#callWrapperMethod()``:
|
||
|
||
#. The method name
|
||
#. The expected arguments that a wrapper script may use
|
||
#. A fallback function
|
||
#. A conditional expression that validates the return value
|
||
|
||
The fallback function only executes if a wrapper script fails or if the method is not overriden.
|
||
``validateRetVal`` checks the type of the return value and ensures it matches the expected type. If there is no return value, simply validate if type is ``null``.
|
||
|
||
.. note::
|
||
Generic method names are preferred so that they can be used for any video wrapper.
|
||
For example: instead of naming a method ``updateCaptionsContainerForSiteA()``, use ``updateCaptionsContainer()``.
|
||
|
||
Using the new video control method
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
Once the new method is defined, it can be used throughout ``PictureInPictureChild.sys.mjs``. In the current example, we call
|
||
``PictureInPictureChildVideoWrapper.setMuted()`` to mute or unmute a video. ``this.videoWrapper`` is an instance of
|
||
``PictureInPictureChildVideoWrapper``:
|
||
|
||
.. code-block:: js
|
||
|
||
// class PictureInPictureChild in PictureInPictureChild.sys.mjs
|
||
mute() {
|
||
let video = this.getWeakVideo();
|
||
if (video && this.videoWrapper) {
|
||
this.videoWrapper.setMuted(video, true);
|
||
}
|
||
}
|
||
|
||
unmute() {
|
||
let video = this.getWeakVideo();
|
||
if (video && this.videoWrapper) {
|
||
this.videoWrapper.setMuted(video, false);
|
||
}
|
||
}
|
||
|
||
Testing site-specific video wrappers
|
||
------------------------------------
|
||
Automated Tests
|
||
^^^^^^^^^^^^^^^
|
||
Automated tests for site specific wrappers are currently limited. New tests can be made in `browser/extensions/pictureinpicture/tests/browser <https://searchfox.org/mozilla-central/source/browser/extensions/pictureinpicture/tests/browser>`_ to ensure
|
||
general functionality, but these are restricted to Firefox Nightly and do not test functionality on specific sites.
|
||
|
||
Some challenges with writing tests include:
|
||
|
||
* Accessing DRM content
|
||
* Log-in credentials if a site requires a user account
|
||
* Detecting modifications to a web page or video player that render a wrapper script obsolete
|
||
|
||
Manual Tests
|
||
^^^^^^^^^^^^
|
||
The go-to approach right now is to test video wrappers manually, in tandem with reviews provided by the phabricator group `#pip-reviewers <https://phabricator.services.mozilla.com/project/profile/163/>`_. Below are some questions that reviewers will consider:
|
||
|
||
* Does Picture-in-Picture crash or freeze?
|
||
* Does the wrapper work on Windows, MacOS, and Linux?
|
||
* Do Picture-in-Picture features work as expected? (Picture-in-Picture toggle, text tracks, video controls, etc.)
|
||
* Do existing automated tests work as they should?
|
||
|
||
.. warning::
|
||
DRM content may not load for all local Firefox builds. One possible solution is to test the video wrapper in a try build (ex. Linux).
|
||
Depending on the changes made, we may also require the script to run under a temporary pref such as ``media.videocontrols.picture-in-picture.WIP.someWebsiteWrapper`` for the purpose of testing changes in Firefox Nightly.
|
||
|
||
API References
|
||
==============
|
||
``toolkit/components/pictureinpicture``
|
||
---------------------------------------
|
||
.. toctree::
|
||
:maxdepth: 1
|
||
|
||
picture-in-picture-api
|
||
player-api
|
||
|
||
``toolkit/actors/PictureInPictureChild.sys.mjs``
|
||
------------------------------------------------
|
||
* :ref:`picture_in_picture_child_video_wrapper_api`
|