diff options
Diffstat (limited to 'toolkit/components/extensions/docs/webext-storage.rst')
| -rw-r--r-- | toolkit/components/extensions/docs/webext-storage.rst | 227 |
1 files changed, 227 insertions, 0 deletions
diff --git a/toolkit/components/extensions/docs/webext-storage.rst b/toolkit/components/extensions/docs/webext-storage.rst new file mode 100644 index 0000000000..23005bb679 --- /dev/null +++ b/toolkit/components/extensions/docs/webext-storage.rst @@ -0,0 +1,227 @@ +======================== +How webext storage works +======================== + +This document describes the implementation of the the `storage.sync` part of the +`WebExtensions Storage APIs +<https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage>`_. +The implementation lives in the `toolkit/components/extensions/storage folder <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage>`_ + +Ideally you would already know about Rust and XPCOM - `see this doc for more details <../../../../writing-rust-code/index.html>`_ + +At a very high-level, the system looks like: + +.. mermaid:: + + graph LR + A[Extensions API] + A --> B[Storage JS API] + B --> C{magic} + C --> D[app-services component] + +Where "magic" is actually the most interesting part and the primary focus of this document. + + Note: The general mechanism described below is also used for other Rust components from the + app-services team - for example, "dogear" uses a similar mechanism, and the sync engines + too (but with even more complexity) to manage the threads. Unfortunately, at time of writing, + no code is shared and it's not clear how we would, but this might change as more Rust lands. + +The app-services component `lives on github <https://github.com/mozilla/application-services/blob/main/components/webext-storage>`_. +There are docs that describe `how to update/vendor this (and all) external rust code <../../../../build/buildsystem/rust.html>`_ you might be interested in. + +To set the scene, let's look at the parts exposed to WebExtensions first; there are lots of +moving part there too. + +WebExtension API +################ + +The WebExtension API is owned by the addons team. The implementation of this API is quite complex +as it involves multiple processes, but for the sake of this document, we can consider the entry-point +into the WebExtension Storage API as being `parent/ext-storage.js <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/parent/ext-storage.js>`_ + +This entry-point ends up using the implementation in the +`ExtensionStorageSync JS class <https://searchfox.org/mozilla-central/rev/9028b0458cc1f432870d2996b186b0938dda734a/toolkit/components/extensions/ExtensionStorageSync.jsm#84>`_. +This class/module has complexity for things like migration from the earlier Kinto-based backend, +but importantly, code to adapt a callback API into a promise based one. + +Overview of the API +################### + +At a high level, this API is quite simple - there are methods to "get/set/remove" extension +storage data. Note that the "external" API exposed to the addon has subtly changed the parameters +for this "internal" API, so there's an extension ID parameter and the JSON data has already been +converted to a string. +The semantics of the API are beyond this doc but are +`documented on MDN <https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync>`_. + +As you will see in those docs, the API is promise-based, but the rust implementation is fully +synchronous and Rust knows nothing about Javascript promises - so this system converts +the callback-based API to a promise-based one. + +xpcom as the interface to Rust +############################## + +xpcom is old Mozilla technology that uses C++ "vtables" to implement "interfaces", which are +described in IDL files. While this traditionally was used to interface +C++ and Javascript, we are leveraging existing support for Rust. The interface we are +exposing is described in `mozIExtensionStorageArea.idl <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/mozIExtensionStorageArea.idl>`_ + +The main interface of interest in this IDL file is `mozIExtensionStorageArea`. +This interface defines the functionality - and is the first layer in the sync to async model. +For example, this interface defines the following method: + +.. code-block:: rust + + interface mozIExtensionStorageArea : nsISupports { + ... + // Sets one or more key-value pairs specified in `json` for the + // `extensionId`... + void set(in AUTF8String extensionId, + in AUTF8String json, + in mozIExtensionStorageCallback callback); + +As you will notice, the 3rd arg is another interface, `mozIExtensionStorageCallback`, also +defined in that IDL file. This is a small, generic interface defined as: + +.. code-block:: + + interface mozIExtensionStorageCallback : nsISupports { + // Called when the operation completes. Operations that return a result, + // like `get`, will pass a `UTF8String` variant. Those that don't return + // anything, like `set` or `remove`, will pass a `null` variant. + void handleSuccess(in nsIVariant result); + + // Called when the operation fails. + void handleError(in nsresult code, in AUTF8String message); + }; + +Note that this delivers all results and errors, so must be capable of handling +every result type, which for some APIs may be problematic - but we are very lucky with this API +that this simple XPCOM callback interface is capable of reasonably representing the return types +from every function in the `mozIExtensionStorageArea` interface. + +(There's another interface, `mozIExtensionStorageListener` which is typically +also implemented by the actual callback to notify the extension about changes, +but that's beyond the scope of this doc.) + +*Note the thread model here is async* - the `set` call will return immediately, and later, on +the main thread, we will call the callback param with the result of the operation. + +So under the hood, what happens is something like: + +.. mermaid:: + + sequenceDiagram + Extension->>ExtensionStorageSync: call `set` and give me a promise + ExtensionStorageSync->>xpcom: call `set`, supplying new data and a callback + ExtensionStorageSync-->>Extension: your promise + xpcom->>xpcom: thread magic in the "bridge" + xpcom-->>ExtensionStorageSync: callback! + ExtensionStorageSync-->>Extension: promise resolved + +So onto the thread magic in the bridge! + +webext_storage_bridge +##################### + +The `webext_storage_bridge <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge>`_ +is a Rust crate which, as implied by the name, is a "bridge" between this Javascript/XPCOM world to +the actual `webext-storage <https://github.com/mozilla/application-services/tree/main/components/webext-storage>`_ crate. + +lib.rs +------ + +Is the entry-point - it defines the xpcom "factory function" - +an `extern "C"` function which is called by xpcom to create the Rust object +implementing `mozIExtensionStorageArea` using existing gecko support. + +area.rs +------- + +This module defines the interface itself. For example, inside that file you will find: + +.. code-block:: rust + + impl StorageSyncArea { + ... + + xpcom_method!( + set => Set( + ext_id: *const ::nsstring::nsACString, + json: *const ::nsstring::nsACString, + callback: *const mozIExtensionStorageCallback + ) + ); + /// Sets one or more key-value pairs. + fn set( + &self, + ext_id: &nsACString, + json: &nsACString, + callback: &mozIExtensionStorageCallback, + ) -> Result<()> { + self.dispatch( + Punt::Set { + ext_id: str::from_utf8(&*ext_id)?.into(), + value: serde_json::from_str(str::from_utf8(&*json)?)?, + }, + callback, + )?; + Ok(()) + } + + +Of interest here: + +* `xpcom_method` is a Rust macro, and part of the existing xpcom integration which already exists + in gecko. It declares the xpcom vtable method described in the IDL. + +* The `set` function is the implementation - it does string conversions and the JSON parsing + on the main thread, then does the work via the supplied callback param, `self.dispatch` and a `Punt`. + +* The `dispatch` method dispatches to another thread, leveraging existing in-tree `moz_task <https://searchfox.org/mozilla-central/source/xpcom/rust/moz_task>`_ support, shifting the `Punt` to another thread and making the callback when done. + +Punt +---- + +`Punt` is a whimsical name somewhat related to a "bridge" - it carries things across and back. + +It is a fairly simple enum in `punt.rs <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_. +It's really just a restatement of the API we expose suitable for moving across threads. In short, the `Punt` is created on the main thread, +then sent to the background thread where the actual operation runs via a `PuntTask` and returns a `PuntResult`. + +There's a few dances that go on, but the end result is that `inner_run() <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_ +gets executed on the background thread - so for `Set`: + +.. code-block:: rust + + Punt::Set { ext_id, value } => { + PuntResult::with_change(&ext_id, self.store()?.get()?.set(&ext_id, value)?)? + } + +Here, `self.store()` is a wrapper around the actual Rust implementation from app-services with +various initialization and mutex dances involved - see `store.rs`. +ie, this function is calling our Rust implementation and stashing the result in a `PuntResult` + +The `PuntResult` is private to that file but is a simple struct that encapsulates both +the actual result of the function (also a set of changes to send to observers, but that's +beyond this doc). + +Ultimately, the `PuntResult` ends up back on the main thread once the call is complete +and arranges to callback the JS implementation, which in turn resolves the promise created in `ExtensionStorageSync.jsm` + +End result: +----------- + +.. mermaid:: + + sequenceDiagram + Extension->>ExtensionStorageSync: call `set` and give me a promise + ExtensionStorageSync->>xpcom - bridge main thread: call `set`, supplying new data and a callback + ExtensionStorageSync-->>Extension: your promise + xpcom - bridge main thread->>moz_task worker thread: Punt this + moz_task worker thread->>webext-storage: write this data to the database + webext-storage->>webext-storage: done: result/error and observers + webext-storage-->>moz_task worker thread: ... + moz_task worker thread-->>xpcom - bridge main thread: PuntResult + xpcom - bridge main thread-->>ExtensionStorageSync: callback! + ExtensionStorageSync-->>Extension: promise resolved |