1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
|
========================
How webext storage works
========================
This document describes the implementation of the the `storage.sync` part of the
`WebExtensions Storage APIs
<https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage>`_.
The implementation lives in the `toolkit/components/extensions/storage folder <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage>`_
Ideally you would already know about Rust and XPCOM - `see this doc for more details <../../../../writing-rust-code/index.html>`_
At a very high-level, the system looks like:
.. mermaid::
graph LR
A[Extensions API]
A --> B[Storage JS API]
B --> C{magic}
C --> D[app-services component]
Where "magic" is actually the most interesting part and the primary focus of this document.
Note: The general mechanism described below is also used for other Rust components from the
app-services team - for example, "dogear" uses a similar mechanism, and the sync engines
too (but with even more complexity) to manage the threads. Unfortunately, at time of writing,
no code is shared and it's not clear how we would, but this might change as more Rust lands.
The app-services component `lives on github <https://github.com/mozilla/application-services/blob/main/components/webext-storage>`_.
There are docs that describe `how to update/vendor this (and all) external rust code <../../../../build/buildsystem/rust.html>`_ you might be interested in.
To set the scene, let's look at the parts exposed to WebExtensions first; there are lots of
moving part there too.
WebExtension API
################
The WebExtension API is owned by the addons team. The implementation of this API is quite complex
as it involves multiple processes, but for the sake of this document, we can consider the entry-point
into the WebExtension Storage API as being `parent/ext-storage.js <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/parent/ext-storage.js>`_
This entry-point ends up using the implementation in the
`ExtensionStorageSync JS class <https://searchfox.org/mozilla-central/rev/9028b0458cc1f432870d2996b186b0938dda734a/toolkit/components/extensions/ExtensionStorageSync.jsm#84>`_.
This class/module has complexity for things like migration from the earlier Kinto-based backend,
but importantly, code to adapt a callback API into a promise based one.
Overview of the API
###################
At a high level, this API is quite simple - there are methods to "get/set/remove" extension
storage data. Note that the "external" API exposed to the addon has subtly changed the parameters
for this "internal" API, so there's an extension ID parameter and the JSON data has already been
converted to a string.
The semantics of the API are beyond this doc but are
`documented on MDN <https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync>`_.
As you will see in those docs, the API is promise-based, but the rust implementation is fully
synchronous and Rust knows nothing about Javascript promises - so this system converts
the callback-based API to a promise-based one.
xpcom as the interface to Rust
##############################
xpcom is old Mozilla technology that uses C++ "vtables" to implement "interfaces", which are
described in IDL files. While this traditionally was used to interface
C++ and Javascript, we are leveraging existing support for Rust. The interface we are
exposing is described in `mozIExtensionStorageArea.idl <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/mozIExtensionStorageArea.idl>`_
The main interface of interest in this IDL file is `mozIExtensionStorageArea`.
This interface defines the functionality - and is the first layer in the sync to async model.
For example, this interface defines the following method:
.. code-block:: rust
interface mozIExtensionStorageArea : nsISupports {
...
// Sets one or more key-value pairs specified in `json` for the
// `extensionId`...
void set(in AUTF8String extensionId,
in AUTF8String json,
in mozIExtensionStorageCallback callback);
As you will notice, the 3rd arg is another interface, `mozIExtensionStorageCallback`, also
defined in that IDL file. This is a small, generic interface defined as:
.. code-block:: cpp
interface mozIExtensionStorageCallback : nsISupports {
// Called when the operation completes. Operations that return a result,
// like `get`, will pass a `UTF8String` variant. Those that don't return
// anything, like `set` or `remove`, will pass a `null` variant.
void handleSuccess(in nsIVariant result);
// Called when the operation fails.
void handleError(in nsresult code, in AUTF8String message);
};
Note that this delivers all results and errors, so must be capable of handling
every result type, which for some APIs may be problematic - but we are very lucky with this API
that this simple XPCOM callback interface is capable of reasonably representing the return types
from every function in the `mozIExtensionStorageArea` interface.
(There's another interface, `mozIExtensionStorageListener` which is typically
also implemented by the actual callback to notify the extension about changes,
but that's beyond the scope of this doc.)
*Note the thread model here is async* - the `set` call will return immediately, and later, on
the main thread, we will call the callback param with the result of the operation.
So under the hood, what happens is something like:
.. mermaid::
sequenceDiagram
Extension->>ExtensionStorageSync: call `set` and give me a promise
ExtensionStorageSync->>xpcom: call `set`, supplying new data and a callback
ExtensionStorageSync-->>Extension: your promise
xpcom->>xpcom: thread magic in the "bridge"
xpcom-->>ExtensionStorageSync: callback!
ExtensionStorageSync-->>Extension: promise resolved
So onto the thread magic in the bridge!
webext_storage_bridge
#####################
The `webext_storage_bridge <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge>`_
is a Rust crate which, as implied by the name, is a "bridge" between this Javascript/XPCOM world to
the actual `webext-storage <https://github.com/mozilla/application-services/tree/main/components/webext-storage>`_ crate.
lib.rs
------
Is the entry-point - it defines the xpcom "factory function" -
an `extern "C"` function which is called by xpcom to create the Rust object
implementing `mozIExtensionStorageArea` using existing gecko support.
area.rs
-------
This module defines the interface itself. For example, inside that file you will find:
.. code-block:: rust
impl StorageSyncArea {
...
xpcom_method!(
set => Set(
ext_id: *const ::nsstring::nsACString,
json: *const ::nsstring::nsACString,
callback: *const mozIExtensionStorageCallback
)
);
/// Sets one or more key-value pairs.
fn set(
&self,
ext_id: &nsACString,
json: &nsACString,
callback: &mozIExtensionStorageCallback,
) -> Result<()> {
self.dispatch(
Punt::Set {
ext_id: str::from_utf8(&*ext_id)?.into(),
value: serde_json::from_str(str::from_utf8(&*json)?)?,
},
callback,
)?;
Ok(())
}
Of interest here:
* `xpcom_method` is a Rust macro, and part of the existing xpcom integration which already exists
in gecko. It declares the xpcom vtable method described in the IDL.
* The `set` function is the implementation - it does string conversions and the JSON parsing
on the main thread, then does the work via the supplied callback param, `self.dispatch` and a `Punt`.
* The `dispatch` method dispatches to another thread, leveraging existing in-tree `moz_task <https://searchfox.org/mozilla-central/source/xpcom/rust/moz_task>`_ support, shifting the `Punt` to another thread and making the callback when done.
Punt
----
`Punt` is a whimsical name somewhat related to a "bridge" - it carries things across and back.
It is a fairly simple enum in `punt.rs <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_.
It's really just a restatement of the API we expose suitable for moving across threads. In short, the `Punt` is created on the main thread,
then sent to the background thread where the actual operation runs via a `PuntTask` and returns a `PuntResult`.
There's a few dances that go on, but the end result is that `inner_run() <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_
gets executed on the background thread - so for `Set`:
.. code-block:: rust
Punt::Set { ext_id, value } => {
PuntResult::with_change(&ext_id, self.store()?.get()?.set(&ext_id, value)?)?
}
Here, `self.store()` is a wrapper around the actual Rust implementation from app-services with
various initialization and mutex dances involved - see `store.rs`.
ie, this function is calling our Rust implementation and stashing the result in a `PuntResult`
The `PuntResult` is private to that file but is a simple struct that encapsulates both
the actual result of the function (also a set of changes to send to observers, but that's
beyond this doc).
Ultimately, the `PuntResult` ends up back on the main thread once the call is complete
and arranges to callback the JS implementation, which in turn resolves the promise created in `ExtensionStorageSync.sys.mjs`
End result:
-----------
.. mermaid::
sequenceDiagram
Extension->>ExtensionStorageSync: call `set` and give me a promise
ExtensionStorageSync->>xpcom - bridge main thread: call `set`, supplying new data and a callback
ExtensionStorageSync-->>Extension: your promise
xpcom - bridge main thread->>moz_task worker thread: Punt this
moz_task worker thread->>webext-storage: write this data to the database
webext-storage->>webext-storage: done: result/error and observers
webext-storage-->>moz_task worker thread: ...
moz_task worker thread-->>xpcom - bridge main thread: PuntResult
xpcom - bridge main thread-->>ExtensionStorageSync: callback!
ExtensionStorageSync-->>Extension: promise resolved
|
