diff options
Diffstat (limited to '')
| -rw-r--r-- | modules/libpref/docs/index.md | 466 |
1 files changed, 466 insertions, 0 deletions
diff --git a/modules/libpref/docs/index.md b/modules/libpref/docs/index.md new file mode 100644 index 0000000000..30bf66d278 --- /dev/null +++ b/modules/libpref/docs/index.md @@ -0,0 +1,466 @@ +# libpref +libpref is a generic key/value store that is used to implement *prefs*, a term +that encompasses a variety of things. + +- Feature enable/disable flags (e.g. `xpinstall.signatures.required`). +- User preferences (e.g. things set from `about:preferences`) +- Internal application parameters (e.g. + `javascript.options.mem.nursery.max_kb`). +- Testing and debugging flags (e.g. `network.dns.native-is-localhost`). +- Things that might need locking in an enterprise installation. +- Application data (e.g. + `browser.onboarding.tour.onboarding-tour-addons.completed`, + `services.sync.clients.lastSync`). +- A cheap and dirty form of IPC(!) (some devtools prefs). + +Some of these (particularly the last two) are not an ideal use of libpref. + +The C++ API is in the `Preferences` class. The XPIDL API is in the +`nsIPrefService` and `nsIPrefBranch` interfaces. + +## High-level design + +### Keys +Keys (a.k.a. *pref names*) are 8-bit strings, and ASCII in practice. The +convention is to use a dotted segmented form, e.g. `foo.bar.baz`, but the +segments have no built-in meaning. + +Naming is inconsistent, e.g. segments have various forms: `foo_bar`, +`foo-bar`, `fooBar`, etc. Pref names for feature flags are likewise +inconsistent: `foo.enabled`, `foo.enable`, `foo.disable`, `fooEnabled`, +`enable-foo`, `foo.enabled.bar`, etc. + +The grouping of prefs into families, via pref name segments, is ad hoc. Some of +these families are closely related, e.g. there are many font prefs that are +present for every language script. + +Some prefs only make sense when considered in combination with other prefs. + +Many pref names are known at compile time, but some are computed at runtime. + +### Basic values +The basic types of pref values are bools, 32-bit ints, and 8-bit C strings. + +Strings are used to encode many types of data: identifiers, alphanumeric IDs, +UUIDs, SHA1 hashes, CSS color hex values, large integers that don't fit into +32-bit ints (e.g. timestamps), directory names, URLs, comma-separated lists, +space-separated lists, JSON blobs, etc. There is a 1 MiB length limit on string +values; longer strings are rejected outright. + +**Problem:** The C string encoding is unclear; some API functions deal with +unrestricted 8-bit strings (i.e. Latin1), but some require UTF-8. + +There is some API support for faking floats, by converting them from/to strings when getting/setting. + +**Problem:** confusion between ints and floats can lead to bugs. + +Each pref consists of a default value and/or a user value. Default values can +be initialized from file at startup, and can be added and modified at runtime +via the API. User values can be initialized from file at startup, and can be +added, modified and removed at runtime via the API and `about:config`. + +If both values are present the user value takes precedence for most operations, +though there are operations that specifically work on the default value. + +If a user value is set to the same value as the default value, the user value +is removed, unless the pref is marked as *sticky* at startup. + +**Problem:** it would be better to have a clear notion of "reset to default", +at least for prefs that have a default value. + +Prefs can be locked. This prevents them from being given a user value, or +hides the existing user value if there is one. + +### Complex values +There is API support for some complex values. + +`nsIFile` objects are handled by storing the filename as a string, similar to +how floats are faked by storing them as strings. + +`nsIPrefLocalizedString` objects are ones for which the default value +specifies a properties file that contains an entry whose name matches the +prefname. When gotten, the value from that entry is put into the user value. +When set, the given value just overwrites the user value, like a string pref. + +**Problem:** this is weird and unlike all the other pref types. + +`nsIRelativeFilePref` objects are only used in comm-central. + +### Pref Branches +XPIDL-based access to prefs is via `nsIPrefBranch`/`nsPrefBranch`, which +lets you specify a branch of the pref tree (e.g. `font.`) and pref names work +relative to that point. + +This API can be used from C++, but for C++ code there is also direct access +through the `Preferences` class, which uses absolute pref names. + +### Threads +For the most part, all the basic API functions only work on the main thread. +However, there are two exceptions to this. + +The narrow exception is that the Servo traversal thread is allowed to get pref +values. This only occurs when the main thread is paused, which makes it safe. +(Note: [bug 1474789](https://bugzilla.mozilla.org/show_bug.cgi?id=1474789) +indicates that this may not be true.) + +The broad exception is that static prefs can have a cached copy of a pref value +that can be accessed from other threads. See below. + +### Notifications +There is a notification API for being told when a pref's value changes. C++ +code can register a callback function and JS code can register an observer (via +`nsIObserver`, which requires XPCOM). In both cases, the registered entity +will be notified when the value of the named pref value changes, or when the +value of any pref matching a given prefix changes. E.g. all font pref changes +can be observed by adding a `font.` prefix-matching observer. + +See also the section on static prefs below. + +### Static prefs +There is a special kind of pref called a static pref. Static prefs are defined +in `StaticPrefList.yaml`. See that file for more documentation. + +If a static pref is defined in both `StaticPrefList.yaml` and a pref data +file, the latter definition will take precedence. A pref shouldn't appear in +both `StaticPrefList.yaml` and `all.js`, but it may make sense for a pref +to appear in both `StaticPrefList.yaml` and an app-specific pref data file +such as `firefox.js`. + +Each static pref has a *mirror* kind. + +* `always`: A C++ *mirror variable* is associated with the pref. The variable + is always kept in sync with the pref value. This kind is common. +* `once`: A C++ mirror variable is associated with the pref. The variable is + synced once with the pref's value at startup, and then does not change. This + kind is less common, and mostly used for graphics prefs. +* `never`: No C++ mirror variable is associated with the pref. This is much + like a normal pref. + +An `always` or `once` static pref can only be used for prefs with +bool/int/float values, not strings or complex values. + +Each mirror variable is read-only, accessible via a getter function. The base +name of the getter function is the same as the pref's name, but with '.' or '-' +converted to '_'. Sometimes a suffix is added, e.g. _AtStartup for the mirror +once kind. + +Mirror variables have two benefits. First, they allow C++ and Rust code to get +the pref value directly from the variable instead of requiring a slow hash +table lookup, which is important for prefs that are consulted frequently. +Second, they allow C++ and Rust code to get the pref value off the main thread. +The mirror variable must have an atomic type if it is read off the main thread, +and assertions ensure this. + +Note that mirror variables could be implemented via vanilla callbacks without +API support, except for one detail: libpref gives their callbacks higher +priority than normal callbacks, ensuring that any static pref will be +up-to-date if read by a normal callback. + +**Problem:** It is not clear what should happen to a static pref's mirror +variable if the pref is deleted? Currently there is a missing +`NotifyCallbacks()` call so the mirror variable keeps its value from before +the deletion. The cleanest solution is probably to disallow static prefs from +being deleted. + +### Sanitized Prefs +We restrict certain prefs from entering web content subprocesses. In these +processes, a preference may be marked as 'Sanitized' to indicate that it may +or may not have a user value, but that value is not present in this process. +In the parent process no pref is marked as Sanitized. + +Pref Sanitization is used for two purposes: + 1. To protect private user data that may be stored in preferences from a + Spectre adversary. + 2. To reduce IPC use and thread wake-ups for commonly modified preferences. + +A pref is sanitized from entering the web content process if it matches a +denylist _or_ it is a dynamically-named string preference (that is not +exempted via an allowlist), See `ShouldSanitizePreference` in +`Preferences.cpp`. + +### Loading and Saving +Default pref values are initialized from various pref data files. Notable ones +include: + +- `modules/libpref/init/all.js`, used by all products; +- `browser/app/profile/firefox.js`, used by Firefox desktop; +- `mobile/android/app/geckoview-prefs.js`, used by GeckoView; +- `mail/app/profile/all-thunderbird.js`, used by Thunderbird (in comm-central); +- `suite/browser/browser-prefs.js`, used by SeaMonkey (in comm-central). + +In release builds these are all put into `omni.ja`. + +User pref values are initialized from `prefs.js` and (if present) +`user.js`, in the user's profile. This only happens once, in the parent +process. Note that `prefs.js` is managed by Firefox, and regularly +overwritten. `user.js` is created and managed by the user, and Firefox only +reads it. + +These files are not JavaScript; the `.js` suffix is present for historical +reasons. They are read by a custom parser within libpref. + +User pref file syntax is slightly more restrictive than default pref file +syntax. In user pref files `user_pref` definitions are allowed but `pref` and +`sticky_pref` definitions are not, and attributes (such as `locked`) are not +allowed. + +**Problem:** geckodriver has a separate prefs parser in the mozprofile crate. + +**Problem:** there is no versioning of these files, for either the syntax or +the data. This makes changing the file format difficult. + +There are API functions to save modified prefs, either synchronously or +asynchronously (via an off-main-thread runnable), either to the default file +(`prefs.js`) or to a named file. When saving to the default file, no action +will take place if no prefs have been modified. + +Also, whenever a pref is modified, we wait 500ms and then automatically do an +off-main-thread save to `prefs.js`. This provides an approximation of +[durability](https://en.wikipedia.org/wiki/ACID#Durability), but it is still +possible for something to go wrong (e.g. a parent process crash) and end up +with recently changed prefs not being saved. (If such a thing happens, it +compromises [atomicity](https://en.wikipedia.org/wiki/ACID#Atomicity), i.e. a +sequence of multiple related pref changes might only get partially written.) + +Only prefs whose values have changed from the default are saved to `prefs.js.` + +**Problem:** Each time prefs are saved, the entire file is overwritten -- 10s +or even 100s of KiBs -- even if only a single value has changed. This happens +at least every 5 minutes, due to sync. Furthermore, various prefs are changed +during and shortly after startup, which can result in 10s of MiBs of disk +activity. + +### about:support +about:support contains an "Important Modified Preferences" table. It contains +all prefs that (a) have had their value changed from the default, and (b) whose +prefix match a allowlist in `Troubleshoot.sys.mjs`. The allowlist matching is to +avoid exposing pref values that might be privacy-sensitive. + +**Problem:** The allowlist of prefixes is specified separately from the prefs +themselves. Having an attribute on a pref definition would be better. + +### Sync +On desktop, a pref is synced onto a device via Sync if there is an +accompanying `services.sync.prefs.sync.`-prefixed pref. I.e. the pref +`foo.bar` is synced if the pref `services.sync.prefs.sync.foo.bar` exists +and is true. + +Previously, one could push prefs onto a device even if a local +`services.sync.prefs.sync.`-prefixed pref was not present; however this +behavior changed in [bug 1538015](https://bugzilla.mozilla.org/show_bug.cgi?id=1538015) +to require the local prefixed pref to be present. The old (insecure) behavior +can be re-enabled by setting a single pref `services.sync.prefs.dangerously_allow_arbitrary` +to true on the target browser - subsequently any pref can be pushed there by +creating a *remote* `services.sync.prefs.sync.`-prefixed pref. + +In practice, only a small subset of prefs (about 70) have a `services.sync.prefs.sync.`-prefixed +pref by default. + +**Problem:** This is gross. An attribute on the pref definition would be +better, but it might be hard to change that at this point. + +The number of synced prefs is small because prefs are synced across versions; +any pref whose meaning might change shouldn't be synced. Also, we don't sync +prefs that may differ across different devices (such as a desktop machine +vs. a notebook). + +Prefs are not synced on mobile. + +### Rust +Static prefs mirror variables can be accessed from Rust code via the +`static_prefs::pref!` macro, for prefs which opt into this using +`rust: true`. Other prefs currently cannot be accessed. Parts +of libpref's C++ API could be made accessible to Rust code fairly +straightforwardly via C bindings, either hand-made or generated. + +### Cost of a pref +The cost of a single pref is low, but the cost of several thousand prefs is +reasonably high, and includes the following. + +- Parsing and initializing at startup. +- IPC costs at startup and on pref value changes. +- Disk writing costs of pref value changes, especially during startup. +- Memory usage for storing the prefs, callbacks and observers, and C++ mirror + variables. +- Complexity: most pref combinations are untested. Some can be set to a bogus + value by a curious user, which can have [serious effects](https://rejzor.wordpress.com/2015/06/14/improve-firefox-html5-video-playback-performance/) + (read the comments). Prefs can also have bugs. Real-life examples include + mistyped prefnames, `all.js` entries with incorrect types (e.g. confusing + int vs. float), both of which mean changing the pref value via about:config + or the API would have no effect (see [bug 1414150](https://bugzilla.mozilla.org/show_bug.cgi?id=1414150) for examples of + both). +- Sync cost, for synced prefs. + +### Guidelines +We have far too many prefs. This is at least partly because we have had, for a +long time, a culture of "when in doubt, add a pref". Also, we don't have any +system — either technical or cultural — for removing unnecessary prefs. See +[bug 90440] (https://bugzilla.mozilla.org/show_bug.cgi?id=90440) for a pref +that was unused for 17 years. + +In short, prefs are Firefox's equivalent of the Windows Registry: a dumping +ground for anything and everything. We should have guidelines for when to add a +pref. + +Here are some good reasons to add a pref. + +- *A user may genuinely want to change it.* E.g. it controls a feature that is + adjustable in about:preferences. +- *To enable/disable new features.* Once a feature is mature, consider removing + the pref. A pref expiry mechanism would help with this. +- *For certain testing/debugging flags.* Ideally, these would not be visible in + about:config. + +Here are some less good reasons to add a pref. + +- *I'm not confident about this numeric parameter (cache size, timeout, etc.)* + Get confident! In practice, few if any users will change it. Adding a pref + doesn't absolve you of the responsibility of finding a good default. Then + make it a code constant. +- *I need to experiment with different parameters during development.* This is + reasonable, but consider removing the pref before landing or once the feature + has matured. An expiry mechanism would help with this. +- *I sometimes fiddle with this value for debugging or testing.* + Is it worth exposing it to the whole world to save yourself a recompile every + once in a while? Consider making it a code constant. +- *Different values are needed on different platforms.* This can be done in + other ways, e.g. `#ifdef` in C++ code. + +These guidelines do not consider application data prefs (i.e. ones that +typically don't have a default value). They are quite different from the other +kinds. They arguably shouldn't prefs at all, and should be stored via some +other mechanism. + +## Low-level details +The key idea is that the prefs database consists of two pieces. The first is an +initial snapshot of pref values that is created when the first child process is +created. This snapshot is stored in immutable, shared memory, and shared by all +processes. + +Pref value changes that occur after this point are stored in a second hash +table. Each process has its own copy of this hash table. When pref values +change in the parent process, it performs IPC to inform child processes about +the changes, so they can update their copy. + +The motivation for this design is memory usage. It's not tenable for every +child process to have a full copy of the prefs database. + +Not all child processes need access to prefs. Those that do include web content +processes, the GPU process, and the RDD process. + +### Parent process startup +The parent process initially has only a hash table. + +Early in startup, the parent process loads all of the static prefs and default +prefs (mainly from `omni.ja`) into that hash table. The parent process also +registers C++ mirror variables for static prefs, initializes them, and +registers callbacks so they will be updated appropriately for all subsequent +updates. + +Slightly later in startup, the parent process loads all user prefs files, +mainly from the profile directory. + +When the first getter for a `once` static pref is called, all the `once` +static prefs have their mirror variables set and special frozen prefs are put +into the hash table. These frozen prefs are copies of the `once` prefs that +are given `$$$` prefixes and suffixes on their names. They are also marked +specially so they are ignored for all cases except when starting a new child +process. They exist so that all child processes can be given the same `once` +values as the parent process. + +### Child process startup (parent side) +When the first child process is created, the parent process serializes most of +its hash table into a shared, immutable snapshot. This snapshot is stored in a +shared memory region managed by a `SharedPrefMap` instance. + +Sanitized preferences (matching _either_ the denylist of the dynamically named +heuristic) are not included in the shared memory region. After building the +shared memory region, the parent process clears the hash table and then +re-enters sanitized prefs into it. Besides the sanitized prefs, the hash table +is subsequently used only to store changed pref values. + +When any child process is created, the parent process serializes all pref +values present in the hash table (i.e. those that have changed since the +snapshot was made) _except sanitized prefs__ and stores them in a second, +short-lived shared memory region. This represents the set of changes the child +process needs to apply on top of the snapshot, and allows it to build a hash +table which should exactly match the parent's, modulo the sanitized prefs. + +The parent process passes two file descriptors to the child process, one for +each region of memory. The snapshot is the same for all child processes. + +### Child process startup (child side) +Early in child process startup, the prefs service maps in and deserializes both +shared memory regions sent from the parent process, but defers further +initialization until requested by XPCOM initialization. Once that happens, +mirror variables are initialized for static prefs, but no default values are +set in the hash table, and no prefs files are loaded. + +Once the mirror variables have been initialized, we dispatch pref change +callbacks for any prefs in the shared snapshot which have user values or are +locked. This causes the mirror variables to be updated. + +After that, the changed pref values received from the parent process (via +`changedPrefsFd`) are added to the prefs database. Their values override the +values in the snapshot, and pref change callbacks are dispatched for them as +appropriate. `once` mirror variable are initialized from the special frozen +pref values. + +### Pref lookups +Each prefs database has both a hash table and a shared memory snapshot. A given +pref may have an entry in either or both of these. If a pref exists in both, +the hash table entry takes precedence. + +For pref lookups, the hash table is checked first, followed by the shared +snapshot. The entry in the hash table may have the type `None`, in which case +the pref is treated as if it did not exist. The entry in the static snapshot +never has the type `None`. + +For pref enumeration, both maps are enumerated, starting with the hash table. +While iterating over the hash table, any entry with the type `None` is +skipped. While iterating over the shared snapshot, any entry which also exists +in the hash table is skipped. The combined result of the two iterations +represents the full contents of the prefs database. + +### Pref changes +Pref changes can only be initiated in the parent process. All API methods that +modify prefs fail noisily (with `NS_ERROR`) if run outside the parent +process. + +Pref changes that happen before the initial snapshot have been made are simple, +and take place in the hash table. There is no shared snapshot to update, and no +child processes to synchronize with. + +Once a snapshot has been created, any changes need to happen in the hash table. + +If an entry for a changed pref already exists in the hash table, that entry can +be updated directly. Likewise for prefs that do not exist in either the hash +table or the shared snapshot: a new hash table entry can be created. + +More care is needed when a changed pref exists in the snapshot but not in the +hash table. In that case, we create a hash table entry with the same values as +the snapshot entry, and then update it... but *only* if the changes will have +an effect. If a caller attempts to set a pref to its existing value, we do not +want to waste memory creating an unnecessary hash table entry. + +Content processes must be told about any visible pref value changes. (A change +to a default value that is hidden by a user value is unimportant.) When this +happens, `ContentParent` detects the change (via an observer). Sanitized prefs +do not produce an update; and for string prefs it also checks the value(s) +don't exceed 4 KiB. If the checks pass, it sends an IPC message +(`PreferenceUpdate`) to the child process, and the child process updates the +pref (default and user value) accordingly. + +**Problem:** The denylist of prefixes is specified separately from the prefs +themselves. Having an attribute on a pref definition would be better. + +**Problem:** The 4 KiB limit can lead to inconsistencies between the parent +process and child processes. E.g. see +[bug 1303051](https://bugzilla.mozilla.org/show_bug.cgi?id=1303051#c28). + +### Pref deletions +Pref deletion is more complicated. If a pref to be deleted exists only in the +hash table of the parent process, its entry can simply be removed. If it exists +in the shared snapshot, however, its hash table entry needs to be kept (or +created), and its type changed to `None`. The presence of this entry masks +the snapshot entry, causing it to be ignored by pref enumerators. |