diff options
Diffstat (limited to 'browser/components/urlbar/docs/experiments.rst')
| -rw-r--r-- | browser/components/urlbar/docs/experiments.rst | 726 |
1 files changed, 726 insertions, 0 deletions
diff --git a/browser/components/urlbar/docs/experiments.rst b/browser/components/urlbar/docs/experiments.rst new file mode 100644 index 0000000000..b4fabc8f7a --- /dev/null +++ b/browser/components/urlbar/docs/experiments.rst @@ -0,0 +1,726 @@ +Extensions & Experiments +======================== + +This document describes address bar extensions and experiments: what they are, +how to run them, how to write them, and the processes involved in each. + +The primary purpose right now for writing address bar extensions is to run +address bar experiments. But extensions are useful outside of experiments, and +not all experiments use extensions. + +Like all Firefox extensions, address bar extensions use the WebExtensions_ +framework. + +.. _WebExtensions: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions + +.. contents:: + :depth: 2 + + +WebExtensions +------------- + +**WebExtensions** is the name of Firefox's extension architecture. The "web" +part of the name hints at the fact that Firefox extensions are built using Web +technologies: JavaScript, HTML, CSS, and to a certain extent the DOM. + +Individual extensions themselves often are referred to as *WebExtensions*. For +clarity and conciseness, this document will refer to WebExtensions as +*extensions*. + +Why are we interested in extensions? Mainly because they're a powerful way to +run experiments in Firefox. See Experiments_ for more on that. In addition, we'd +also like to build up a robust set of APIs useful to extension authors, although +right now the API can only be used by Mozilla extensions. + +WebExtensions are introduced and discussed in detail on `MDN +<WebExtensions_>`__. You'll need a lot of that knowledge in order to build +address bar extensions. + +Developing Address Bar Extensions +--------------------------------- + +Overview +~~~~~~~~ + +The address bar WebExtensions API currently lives in two API namespaces, +``browser.urlbar`` and ``browser.experiments.urlbar``. The reason for this is +historical and is discussed in the `Developing Address Bar Extension APIs`_ +section. As a consumer of the API, there are only two important things you need +to know: + +* There's no meaningful difference between the APIs of the two namespaces. + Their kinds of functions, events, and properties are similar. You should + think of the address bar API as one single API that happens to be split into + two namespaces. + +* However, there is a big difference between the two when it comes to setting up + your extension to use them. This is discussed next. + +The ``browser.urlbar`` API namespace is built into Firefox. It's a +**privileged API**, which means that only Mozilla-signed and temporarily +installed extensions can use it. The only thing your Mozilla extension needs to +do in order to use it is to request the ``urlbar`` permission in its +manifest.json, as illustrated `here <urlbarPermissionExample_>`__. + +In contrast, the ``browser.experiments.urlbar`` API namespace is bundled inside +your extension. APIs that are bundled inside extensions are called +**experimental APIs**, and the extensions in which they're bundled are called +**WebExtension experiments**. As with privileged APIs, experimental APIs are +available only to Mozilla-signed and temporarily installed extensions. +("WebExtension experiments" is a term of art and shouldn't be confused with the +general notion of experiments that happen to use extensions.) For more on +experimental APIs and WebExtension experiments, see the `WebExtensions API +implementation documentation <webextAPIImplBasicsDoc_>`__. + +Since ``browser.experiments.urlbar`` is bundled inside your extension, you'll +need to include it in your extension's repo by doing the following: + +1. The implementation consists of two files, api.js_ and schema.json_. In your + extension repo, create a *experiments/urlbar* subdirectory and copy the + files there. See `this repo`__ for an example. + +2. Add the following ``experiment_apis`` key to your manifest.json (see here__ + for an example in context):: + + "experiment_apis": { + "experiments_urlbar": { + "schema": "experiments/urlbar/schema.json", + "parent": { + "scopes": ["addon_parent"], + "paths": [["experiments", "urlbar"]], + "script": "experiments/urlbar/api.js" + } + } + } + +As mentioned, only Mozilla-signed and temporarily installed extensions can use +these two API namespaces. For information on running the extensions you develop +that use these namespaces, see `Running Address Bar Extensions`_. + +.. _urlbarPermissionExample: https://github.com/0c0w3/urlbar-top-sites-experiment/blob/ac1517118bb7ee165fb9989834514b1082575c10/src/manifest.json#L24 +.. _webextAPIImplBasicsDoc: https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/basics.html +.. _api.js: https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/api.js +.. _schema.json: https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/schema.json +__ https://github.com/0c0w3/dynamic-result-type-extension/tree/master/src/experiments/urlbar +__ https://github.com/0c0w3/dynamic-result-type-extension/blob/0987da4b259b9fcb139b31d771883a2f822712b5/src/manifest.json#L28 + +browser.urlbar +~~~~~~~~~~~~~~ + +Currently the only documentation for ``browser.urlbar`` is its `schema +<urlbar.json_>`__. Fortunately WebExtension schemas are JSON and aren't too hard +to read. If you need help understanding it, see the `WebExtensions API +implementation documentation <webextAPIImplDoc_>`__. + +For examples on using the API, see the Cookbook_ section. + +.. _urlbar.json: https://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/urlbar.json + +browser.experiments.urlbar +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As with ``browser.urlbar``, currently the only documentation for +``browser.experiments.urlbar`` is its schema__. For examples on using the API, +see the Cookbook_ section. + +__ https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/schema.json + +Workflow +~~~~~~~~ + +The web-ext_ command-line tool makes the extension-development workflow very +simple. Simply start it with the *run* command, passing it the location of the +Firefox binary you want to use. web-ext will launch your Firefox and remain +running until you stop it, watching for changes you make to your extension's +files. When it sees a change, it automatically reloads your extension — in +Firefox, in the background — without your having to do anything. It's really +nice. + +The `web-ext documentation <web-ext commands_>`__ lists all its options, but +here are some worth calling out for the *run* command: + +``--browser-console`` + Automatically open the browser console when Firefox starts. Very useful for + watching your extension's console logging. (Make sure "Show Content Messages" + is checked in the console.) + +``-p`` + This option lets you specify a path to a profile directory. + +``--keep-profile-changes`` + Normally web-ext doesn't save any changes you make to the profile. Use this + option along with ``-p`` to reuse the same profile again and again. + +``--verbose`` + web-ext suppresses Firefox messages in the terminal unless you pass this + option. If you've added some ``dump`` calls in Firefox because you're working + on a new ``browser.urlbar`` API, for example, you won't see them without this. + +web-ext also has a *build* command that packages your extension's files into a +zip file. The following *build* options are useful: + +``--overwrite-dest`` + Without this option, web-ext won't overwrite a zip file it previously created. + +web-ext can load its configuration from your extension's package.json. That's +the recommended way to configure it. Here's an example__. + +Finally, web-ext can also sign extensions, but if you're developing your +extension for an experiment, you'll use a different process for signing. See +`The Experiment Development Process`_. + +.. _web-ext: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Getting_started_with_web-ext +.. _web-ext commands: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/web-ext_command_reference +__ https://github.com/0c0w3/urlbar-top-sites-experiment/blob/6681a7126986bc2565d036b888cb5b8807397ce5/package.json#L7 + +Automated Tests +~~~~~~~~~~~~~~~ + +It's possible to write `browser chrome mochitests`_ for your extension the same +way we write tests for Firefox. One of the example extensions linked throughout +this document includes a test_, for instance. + +See the readme in the example-addon-experiment_ repo for a workflow. + +.. _browser chrome mochitests: https://developer.mozilla.org/en-US/docs/Mozilla/Browser_chrome_tests +.. _test: https://github.com/0c0w3/urlbar-top-sites-experiment/blob/master/tests/tests/browser/browser_urlbarTopSitesExtension.js + +Cookbook +~~~~~~~~ + +*To be written.* For now, you can find example uses of ``browser.experiments.urlbar`` and ``browser.urlbar`` in the following repos: + +* https://github.com/mozilla-extensions/firefox-quick-suggest-weather +* https://github.com/0c0w3/urlbar-tips-experiment +* https://github.com/0c0w3/urlbar-top-sites-experiment +* https://github.com/0c0w3/urlbar-search-interventions-experiment + +Further Reading +~~~~~~~~~~~~~~~ + +`WebExtensions on MDN <WebExtensions_>`__ + The place to learn about developing WebExtensions in general. + +`Getting started with web-ext <web-ext_>`__ + MDN's tutorial on using web-ext. + +`web-ext command reference <web-ext commands_>`__ + MDN's documentation on web-ext's commands and their options. + +Developing Address Bar Extension APIs +------------------------------------- + +Built-In APIs vs. Experimental APIs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Originally we developed the address bar extension API in the ``browser.urlbar`` +namespace, which is built into Firefox as discussed above. By "built into +Firefox," we mean that the API is developed in `mozilla-central +<urlbar.json_>`__ and shipped inside Firefox just like any other Firefox +feature. At the time, that seemed like the right thing to do because we wanted +to build an API that ultimately could be used by all extension authors, not only +Mozilla. + +However, there were a number of disadvantages to this development model. The +biggest was that it tightly coupled our experiments to specific versions of +Firefox. For example, if we were working on an experiment that targeted Firefox +72, then any APIs used by that experiment needed to land and ship in 72. If we +weren't able to finish an API by the time 72 shipped, then the experiment would +have to be postponed until 73. Our experiment development timeframes were always +very short because we always wanted to ship our experiments ASAP. Often we +targeted the Firefox version that was then in Nightly; sometimes we even +targeted the version in Beta. Either way, it meant that we were always uplifting +patch after patch to Beta. This tight coupling between Firefox versions and +experiments erased what should have been a big advantage of implementing +experiments as extensions in the first place: the ability to ship experiments +outside the usual cyclical release process. + +Another notable disadvantage of this model was just the cognitive weight of the +idea that we were developing APIs not only for ourselves and our experiments but +potentially for all extensions. This meant that not only did we have to design +APIs to meet our immediate needs, we also had to imagine use cases that could +potentially arise and then design for them as well. + +For these reasons, we stopped developing ``browser.urlbar`` and created the +``browser.experiments.urlbar`` experimental API. As discussed earlier, +experimental APIs are APIs that are bundled inside extensions. Experimental APIs +can do anything that built-in APIs can do with the added flexibility of not +being tied to specific versions of Firefox. + +Adding New APIs +~~~~~~~~~~~~~~~ + +All new address bar APIs should be added to ``browser.experiments.urlbar``. +Although this API does not ship in Firefox, it's currently developed in +mozilla-central, in `browser/components/urlbar/tests/ext/ <extDirectory_>`__ -- +note the "tests" subdirectory. Developing it in mozilla-central lets us take +advantage of our usual build and testing infrastructure. This way we have API +tests running against each mozilla-central checkin, against all versions of +Firefox that are tested on Mozilla's infrastructure, and we're alerted to any +breaking changes we accidentally make. When we start a new extension repo, we +copy schema.json and api.js to it as described earlier (or clone an example repo +with up-to-date copies of these files). + +Generally changes to the API should be reviewed by someone on the address bar +team and someone on the WebExtensions team. Shane (mixedpuppy) is a good +contact. + +.. _extDirectory: https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/ + +Anatomy of an API +~~~~~~~~~~~~~~~~~ + +Roughly speaking, a WebExtensions API implementation comprises three different +pieces: + +Schema + The schema declares the functions, properties, events, and types that the API + makes available to extensions. Schemas are written in JSON. + + The ``browser.experiments.urlbar`` schema is schema.json_, and the + ``browser.urlbar`` schema is urlbar.json_. + + For reference, the schemas of built-in APIs are in + `browser/components/extensions/schemas`_ and + `toolkit/components/extensions/schemas`_. + + .. _browser/components/extensions/schemas: https://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/ + .. _toolkit/components/extensions/schemas: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/schemas/ + +Internals + Every API hooks into some internal part of Firefox. For the address bar API, + that's the Urlbar implementation in `browser/components/urlbar`_. + + .. _browser/components/urlbar: https://searchfox.org/mozilla-central/source/browser/components/urlbar/ + +Glue + Finally, there's some glue code that implements everything declared in the + schema. Essentially, this code mediates between the previous two pieces. It + translates the function calls, property accesses, and event listener + registrations made by extensions using the public-facing API into terms that + the Firefox internals understand, and vice versa. + + For ``browser.experiments.urlbar``, this is api.js_, and for + ``browser.urlbar``, it's ext-urlbar.js_. + + For reference, the implementations of built-in APIs are in + `browser/components/extensions`_ and `toolkit/components/extensions`_, in the + *parent* and *child* subdirecties. As you might guess, code in *parent* runs + in the main process, and code in *child* runs in the extensions process. + Address bar APIs deal with browser chrome and their implementations therefore + run in the parent process. + + .. _ext-urlbar.js: https://searchfox.org/mozilla-central/source/browser/components/extensions/parent/ext-urlbar.js + .. _browser/components/extensions: https://searchfox.org/mozilla-central/source/browser/components/extensions/ + .. _toolkit/components/extensions: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/ + +Keep in mind that extensions run in a different process from the main process. +That has implications for your APIs. They'll generally need to be async, for +example. + +Further Reading +~~~~~~~~~~~~~~~ + +`WebExtensions API implementation documentation <webextAPIImplDoc_>`__ + Detailed info on implementing a WebExtensions API. + +.. _webextAPIImplDoc: https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/ + +Running Address Bar Extensions +------------------------------ + +As discussed above, ``browser.experiments.urlbar`` and ``browser.urlbar`` are +privileged APIs. There are two different points to consider when it comes to +running an extension that uses privileged APIs: loading the extension in the +first place, and granting it access to privileged APIs. There's a certain bar +for loading any extension regardless of its API usage that depends on its signed +state and the Firefox build you want to run it in. There's yet a higher bar for +granting it access to privileged APIs. This section discusses how to load +extensions so that they can access privileged APIs. + +Since we're interested in extensions primarily for running experiments, there +are three particular signed states relevant to us: + +Unsigned + There are two ways to run unsigned extensions that use privileged APIs. + + They can be loaded temporarily using a Firefox Nightly build or + Developer Edition but not Beta or Release [source__], and the + ``extensions.experiments.enabled`` preference must be set to true [source__]. + You can load extensions temporarily by visiting + about:debugging#/runtime/this-firefox and clicking "Load Temporary Add-on." + `web-ext <Workflow_>`__ also loads extensions temporarily. + + __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/components/extensions/Extension.jsm#1884 + __ https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/toolkit/mozapps/extensions/internal/AddonSettings.jsm#93 + + They can be also be loaded normally (not temporarily) in a custom build where + the build-time setting ``AppConstants.MOZ_REQUIRE_SIGNING`` [source__, source__] + and ``xpinstall.signatures.required`` pref are both false. As in the previous + paragraph, such builds include Nightly and Developer Edition but not Beta or + Release [source__]. In addition, your custom build must modify the + ``Extension.isPrivileged`` getter__ to return true. This getter determines + whether an extension can access privileged APIs. + + __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/mozapps/extensions/internal/XPIProvider.jsm#2382 + __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/mozapps/extensions/internal/AddonSettings.jsm#36 + __ https://searchfox.org/mozilla-central/search?q=MOZ_REQUIRE_SIGNING&case=false®exp=false&path= + __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/components/extensions/Extension.jsm#1874 + + Extensions remain unsigned as you develop them. See the Workflow_ section for + more. + +Signed for testing (Signed for QA) + Signed-for-testing extensions that use privileged APIs can be run using the + same techniques for running unsigned extensions. + + They can also be loaded normally (not temporarily) if you use a Firefox build + where the build-time setting ``AppConstants.MOZ_REQUIRE_SIGNING`` is false and + you set the ``xpinstall.signatures.dev-root`` pref to true + [source__]. ``xpinstall.signatures.dev-root`` does not exist by default and + must be created. + + __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/mozapps/extensions/internal/XPIInstall.jsm#262 + + You encounter extensions that are signed for testing when you are writing + extensions for experiments. See the Experiments_ section for details. + + "Signed for QA" is another way of referring to this signed state. + +Signed for release + Signed-for-release extensions that use privileged APIs can be run in any + Firefox build with no special requirements. + + You encounter extensions that are signed for release when you are writing + extensions for experiments. See the Experiments_ section for details. + +.. important:: + To see console logs from extensions in the browser console, select the "Show + Content Messages" option in the console's settings. This is necessary because + extensions run outside the main process. + +Experiments +----------- + +**Experiments** let us try out ideas in Firefox outside the usual release cycle +and on particular populations of users. + +For example, say we have a hunch that the top sites shown on the new-tab page +aren't very discoverable, so we want to make them more visible. We have one idea +that might work — show them every time the user begins an interaction with the +address bar — but we aren't sure how good an idea it is. So we test it. We write +an extension that does just that, make sure it collects telemetry that will help +us answer our question, ship it outside the usual release cycle to a small +percentage of Beta users, collect and analyze the telemetry, and determine +whether the experiment was successful. If it was, then we might want to ship the +feature to all Firefox users. + +Experiments sometimes are also called **studies** (not to be confused with *user +studies*, which are face-to-face interviews with users conducted by user +researchers). + +There are two types of experiments: + +Pref-flip experiments + Pref-flip experiments are simple. If we have a fully baked feature in the + browser that's preffed off, a pref-flip experiment just flips the pref on, + enabling the feature for users running the experiment. No code is required. + We tell the experiments team the name of the pref we want to flip, and they + handle it. + + One important caveat to pref-flip studies is that they're currently capable of + flipping only a single pref. There's an extension called Multipreffer_ that + can flip multiple prefs, though. + + .. _Multipreffer: https://github.com/mozilla/multipreffer + +Add-on experiments + Add-on experiments are much more complex but much more powerful. (Here + *add-on* is a synonym for extension.) They're the type of experiments that + this document has been discussing all along. + + An add-on experiment is shipped as an extension that we write and that + implements the experimental feature we want to test. To reiterate, the + extension is a WebExtension and uses WebExtensions APIs. If the current + WebExtensions APIs do not meet the needs of your experiment, then you must + create either experimental or built-in APIs so that your extension can use + them. If necessary, you can make any new built-in APIs privileged so that they + are available only to Mozilla extensions. + + An add-on experiment can collect additional telemetry that's not collected in + the product by using the privileged ``browser.telemetry`` WebExtensions API, + and of course the product will continue to collect all the telemetry it + usually does. The telemetry pings from users running the experiment will be + correlated with the experiment with no extra work on our part. + +A single experiment can deliver different UXes to different groups of users +running the experiment. Each group or UX within an experiment is called a +**branch**. Experiments often have two branches, control and treatment. The +**control branch** actually makes no UX changes. It may capture additional +telemetry, though. Think of it as the control in a science experiment. It's +there so we can compare it to data from the **treatment branch**, which does +make UX changes. Some experiments may require multiple treatment branches, in +which case the different branches will have different names. Add-on experiments +can implement all branches in the same extension or each branch in its own +extension. + +Experiments are delivered to users by a system called **Normandy**. Normandy +comprises a client side that lives in Firefox and a server side. In Normandy, +experiments are defined server-side in files called **recipes**. Recipes include +information about the experiment like the Firefox release channel and version +that the experiment targets, the number of users to be included in the +experiment, the branches in the experiment, the percentage of users on each +branch, and so on. + +Experiments are tracked by Mozilla project management using a system called +Experimenter_. + +Finally, there was an older version of the experiments program called +**Shield**. Experiments under this system were called **Shield studies** and +could be be shipped as extensions too. + +.. _Experimenter: https://experimenter.services.mozilla.com/ + +Further Reading +~~~~~~~~~~~~~~~ + +`Pref-Flip and Add-On Experiments <https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=FIREFOX&title=Pref-Flip+and+Add-On+Experiments>`__ + A comprehensive document on experiments from the Experimenter team. See the + child pages in the sidebar, too. + +`Client Implementation Guidelines for Experiments <https://docs.telemetry.mozilla.org/cookbooks/client_guidelines.html>`_ + Relevant documentation from the telemetry team. + +#ask-experimenter Slack channel + A friendly place to get answers to your experiment questions. + +The Experiment Development Process +---------------------------------- + +This section describes an experiment's life cycle. + +1. Experiments usually originate with product management and UX. They're + responsible for identifying a problem, deciding how an experiment should + approach it, the questions we want to answer, the data we need to answer + those questions, the user population that should be enrolled in the + experiment, the definition of success, and so on. + +2. UX makes a spec that describes what the extension looks like and how it + behaves. + +3. There's a kickoff meeting among the team to introduce the experiment and UX + spec. It's an opportunity for engineering to ask questions of management, UX, + and data science. It's really important for engineering to get a precise and + accurate understanding of how the extension is supposed to behave — right + down to the UI changes — so that no one makes erroneous assumptions during + development. + +4. At some point around this time, the team (usually management) creates a few + artifacts to track the work and facilitate communication with outside teams + involved in shipping experiments. They include: + + * A page on `Experimenter <Experiments_>`__ + * A QA PI (product integrity) request so that QA resources are allocated + * A bug in `Data Science :: Experiment Collaboration`__ so that data science + can track the work and discuss telemetry (engineering might file this one) + + __ https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=normal&bug_status=NEW&bug_type=task&cf_firefox_messaging_system=---&cf_fx_iteration=---&cf_fx_points=---&comment=%23%23%20Brief%20Description%20of%20the%20request%20%28required%29%3A%0D%0A%0D%0A%23%23%20Business%20purpose%20for%20this%20request%20%28required%29%3A%0D%0A%0D%0A%23%23%20Requested%20timelines%20for%20the%20request%20or%20how%20this%20fits%20into%20roadmaps%20or%20critical%20decisions%20%28required%29%3A%0D%0A%0D%0A%23%23%20Links%20to%20any%20assets%20%28e.g%20Start%20of%20a%20PHD%2C%20BRD%3B%20any%20document%20that%20helps%20describe%20the%20project%29%3A%0D%0A%0D%0A%23%23%20Name%20of%20Data%20Scientist%20%28If%20Applicable%29%3A%0D%0A%0D%0A%2APlease%20note%20if%20it%20is%20found%20that%20not%20enough%20information%20has%20been%20given%20this%20will%20delay%20the%20triage%20of%20this%20request.%2A&component=Experiment%20Collaboration&contenttypemethod=list&contenttypeselection=text%2Fplain&filed_via=standard_form&flag_type-4=X&flag_type-607=X&flag_type-800=X&flag_type-803=X&flag_type-936=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Data%20Science&rep_platform=Unspecified&target_milestone=---&version=unspecified + +5. Engineering breaks down the work and files bugs. There's another engineering + meeting to discuss the breakdown, or it's discussed asynchronously. + +6. Engineering sets up a GitHub repo for the extension. See `Implementing + Experiments`_ for an example repo you can clone to get started. Disable + GitHub Issues on the repo so that QA will file bugs in Bugzilla instead of + GitHub. There's nothing wrong with GitHub Issues, but our team's project + management tracks all work through Bugzilla. If it's not there, it's not + captured. + +7. Engineering or management fills out the Add-on section of the Experimenter + page as much as possible at this point. "Active Experiment Name" isn't + necessary, and "Signed Release URL" won't be available until the end of the + process. + +8. Engineering implements the extension and any new WebExtensions APIs it + requires. + +9. When the extension is done, engineering or management clicks the "Ready for + Sign-Off" button on the Experimenter page. That changes the page's status + from "Draft" to "Ready for Sign-Off," which allows QA and other teams to sign + off on their portions of the experiment. + +10. Engineering requests the extension be signed "for testing" (or "for + QA"). Michael (mythmon) from the Experiments team and Rehan (rdalal) from + Services Engineering are good contacts. Build the extension zip file using + web-ext as discussed in Workflow_. Attach it to a bug (a metabug for + implementing the extension, for example), needinfo Michael or Rehan, and ask + him to sign it. He'll attach the signed version to the bug. If neither + Michael nor Rehan is available, try asking in the #ask-experimenter Slack + channel. + +11. Engineering sends QA the link to the signed extension and works with them to + resolve bugs they find. + +12. When QA signs off, engineering asks Michael to sign the extension "for + release" using the same needinfo process described earlier. + +13. Paste the URL of the signed extension in the "Signed Release URL" textbox of + the Add-on section of the Experimenter page. + +14. Other teams sign off as they're ready. + +15. The experiment ships! 🎉 + + +Implementing Experiments +------------------------ + +This section discusses how to implement add-on experiments. Pref-flip +experiments are much simpler and don't need a lot of explanation. You should be +familiar with the concepts discussed in the `Developing Address Bar Extensions`_ +and `Running Address Bar Extensions`_ sections before reading this one. + +The most salient thing about add-on experiments is that they're implemented +simply as privileged extensions. Other than being privileged and possibly +containing bundled experimental APIs, they're similar to all other extensions. + +The `top-sites experiment extension <topSites_>`__ is an example of a real, +shipped experiment. + +.. _topSites: https://github.com/0c0w3/urlbar-top-sites-experiment + +Setup +~~~~~ + +example-addon-experiment_ is a repo you can clone to get started. It's geared +toward urlbar extensions and includes the stub of a browser chrome mochitest. + +.. _example-addon-experiment: https://github.com/0c0w3/example-addon-experiment + +browser.normandyAddonStudy +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As discussed in Experiments_, an experiment typically has more than one branch +so that it can test different UXes. The experiment's extension(s) needs to know +the branch the user is enrolled in so that it can behave appropriately for the +branch: show the user the proper UX, collect the proper telemetry, and so on. + +This is the purpose of the ``browser.normandyAddonStudy`` WebExtensions API. +Like ``browser.urlbar``, it's a privileged API available only to Mozilla +extensions. + +Its schema is normandyAddonStudy.json_. + +It's a very simple API. The primary function is ``getStudy``, which returns the +study the user is currently enrolled in or null if there isn't one. (Recall that +*study* is a synonym for *experiment*.) One of the first things an experiment +extension typically does is to call this function. + +The Normandy client in Firefox will keep an experiment extension installed only +while the experiment is active. Therefore, ``getStudy`` should always return a +non-null study object. Nevertheless, the study object has an ``active`` boolean +property that's trivial to sanity check. (The example extension does.) + +The more important property is ``branch``, the name of the branch that the user +is enrolled in. Your extension should use it to determine the appropriate UX. + +Finally, there's an ``onUnenroll`` event that's fired when the user is +unenrolled in the study. It's not quite clear in what cases an extension would +need to listen for this event given that Normandy automatically uninstalls +extensions on unenrollment. Maybe if they create some persistent state that's +not automatically undone on uninstall by the WebExtensions framework? + +If your extension itself needs to unenroll the user for some reason, call +``endStudy``. + +.. _normandyAddonStudy.json: https://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/normandyAddonStudy.json + +Telemetry +~~~~~~~~~ + +Experiments can capture telemetry in two places: in the product itself and +through the privileged ``browser.telemetry`` WebExtensions API. The API schema +is telemetry.json_. + +The telemetry pings from users running experiments are automatically correlated +with those experiments, no extra work required. That's true regardless of +whether the telemetry is captured in the product or though +``browser.telemetry``. + +The address bar has some in-product, preffed off telemetry that we want to +enable for all our experiments — at least that's the thinking as of August 2019. +It's called `engagement event telemetry`_, and it records user *engagements* +with and *abandonments* of the address bar [source__]. We added a +BrowserSetting_ on ``browser.urlbar`` just to let us flip the pref and enable +this telemetry in our experiment extensions. Call it like this:: + + await browser.urlbar.engagementTelemetry.set({ value: true }); + +.. _telemetry.json: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/schemas/telemetry.json +.. _engagement event telemetry: https://bugzilla.mozilla.org/show_bug.cgi?id=1559136 +__ https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/browser/components/urlbar/UrlbarController.jsm#598 +.. _BrowserSetting: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/types/BrowserSetting + +Engineering Best Practices +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Clear up questions with your UX person early and often. There's often a gap +between what they have in their mind and what you have in yours. Nothing wrong +with that, it's just the nature of development. But misunderstandings can cause +big problems when they're discovered late. This is especially true of UX +behaviors, as opposed to visuals or styling. It's no fun to realize at the end +of a release cycle that you've designed the wrong WebExtensions API because some +UX detail was overlooked. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Related to the previous point, make builds of your extension for your UX person +so they can test it. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Taking the previous point even further, if your experiment will require a +substantial new API(s), you might think about prototyping the experiment +entirely in a custom Firefox build before designing the API at all. Give it to +your UX person. Let them disect it and tell you all the problems with it. Fill +in all the gaps in your understanding, and then design the API. We've never +actually done this, though. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It's a good idea to work on the extension as you're designing and developing the +APIs it'll use. You might even go as far as writing the first draft of the +extension before even starting to implement the APIs. That lets you spot +problems that may not be obvious were you to design the API in isolation. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Your extension's ID should end in ``@shield.mozilla.org``. QA will flag it if it +doesn't. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set ``"hidden": true`` in your extension's manifest.json. That hides it on +about:addons. (It can still be seen on about:studies.) QA will spot this if you +don't. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are drawbacks of hiding features behind prefs and enabling them in +experiment extensions. Consider not doing that if feasible, or at least weigh +these drawbacks against your expected benefits. + +* Prefs stay flipped on in private windows, but experiments often have special + requirements around private-browsing mode (PBM). Usually, they shouldn't be + active in PBM at all, unless of course the point of the experiment is to test + PBM. Extensions also must request PBM access ("incognito" in WebExtensions + terms), and the user can disable access at any time. The result is that part + of your experiment could remain enabled — the part behind the pref — while + other parts are disabled. + +* Prefs stay flipped on in safe mode, even though your extension (like all + extensions) will be disabled. This might be a bug__ in the WebExtensions + framework, though. + + __ https://bugzilla.mozilla.org/show_bug.cgi?id=1576997 |