diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 19:33:14 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 19:33:14 +0000 |
| commit | 36d22d82aa202bb199967e9512281e9a53db42c9 (patch) | |
| tree | 105e8c98ddea1c1e4784a60a5a6410fa416be2de /browser/components/newtab/docs/v2-system-addon | |
| parent | Initial commit. (diff) | |
| download | firefox-esr-upstream.tar.xz firefox-esr-upstream.zip | |
Adding upstream version 115.7.0esr.upstream/115.7.0esrupstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
11 files changed, 2558 insertions, 0 deletions
diff --git a/browser/components/newtab/docs/v2-system-addon/about_home_startup_cache.md b/browser/components/newtab/docs/v2-system-addon/about_home_startup_cache.md new file mode 100644 index 0000000000..584a683366 --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/about_home_startup_cache.md @@ -0,0 +1,86 @@ +# The `about:home` startup cache + +By default, a user's browser session starts with a single window and a single tab, pointed at about:home. This means that it's important to ensure that `about:home` loads as quickly as possible to provide a fast overall startup experience. + +`about:home`, which is functionally identical to `about:newtab`, is generated dynamically by calculating an appropriate state object in the parent process, and passing it down to a content process into the React library in order to render the final interactive page. This is problematic during the startup sequence, as calculating that initial state can be computationally expensive, and requires multiple reads from the disk. + +The `about:home` startup cache is an attempt to address this expense. It works by assuming that between browser sessions, `about:home` _usually_ doesn't need to change. + +## Components of the `about:home` startup cache mechanism + +There are 3 primary components to the cache mechanism: + +### The HTTP Cache + +The HTTP cache is normally used for caching webpages retrieved over the network, but seemed like the right fit for storage of the `about:home` cache as well. + +The HTTP cache is usually queried by the networking stack when browsing the web. The HTTP cache is, however, not typically queried when accessing `chrome://` or `resource://` URLs, so we have to do it ourselves, manually for the `about:home` case. This means giving `about:home` special capabilities for populating and reading from the HTTP cache. In order to avoid potential security issues, this requires that we sequester `about:home` / `about:newtab` in their own special content process. The "privileged about content process" exists for this purpose, and is also used for `about:logins` and `about:certificate`. + +The HTTP cache lives in the parent process, and so any read and write operations need to be initiated in the parent process. Thankfully, however, the HTTP cache accepts data using `nsIOutputStream` and serves it using `nsIInputStream`. We can send `nsIInputStream` over the message manager, and convert an `nsIInputStream` into an `nsIOutputStream`, so we have everything we need to efficiently communicate with the "privileged about content process" to save and retrieve page data. + +The official documentation for the HTTP cache [can be found here](https://developer.mozilla.org/en-US/docs/Mozilla/HTTP_cache). + +### `AboutHomeStartupCache` + +This singleton component lives inside of `BrowserGlue` to avoid having to load yet another JSM out of the `omni.ja` file in the parent process during startup. + +`AboutHomeStartupCache` is responsible for feeding the "privileged about content process" with the `nsIInputStream`'s that it needs to present the initial `about:home` document. It is also responsible for populating the cache with updated versions of `about:home` that are sent by the "privileged about content process". + +Since accessing the HTTP cache is asynchronous, there is an opportunity for a race, where the cache can either be accessed and available before the initial `about:home` is requested, or after. To accommodate for both cases, the `AboutHomeStartupCache` constructs `nsIPipe` instances, which it sends down to the "privileged about content process" as soon as one launches. + +If the HTTP cache entry is already available when the process launches, and cached data is available, we connect the cache to the `nsIPipe`'s to stream the data down to the "privileged about content process". + +If the HTTP cache is not yet available, we hold references to those `nsIPipe` instances, and wait until the cache entry is available. Only then do we connect the cache entry to the `nsIPipe` instances to send the data down to the "privileged about content process". + +### `AboutNewTabService` + +The `AboutNewTabService` is used by the `AboutRedirector` in both the parent and content processes to determine how to handle attempts to load `about:home` and `about:newtab`. + +There are distinct versions of the `AboutNewTabService` - one for the parent process (`BaseAboutNewTabService`), and one for content processes (`AboutNewTabChildService`, which inherits from `BaseAboutNewTabService`). + +The `AboutRedirector`, when running inside of a "privileged about content process" knows to direct attempts to load `about:home` to `AboutNewTabChildService`'s `aboutHomeCacheChannel` method. This method is then responsible for choosing whether or not to return an `nsIChannel` for the cached document, or for the dynamically generated version of `about:home`. + +### `AboutHomeStartupCacheChild` + +This singleton component lives inside of the "privileged about content process", and is initialized as soon as the message is received from the parent that includes the `nsIInputStream`'s that will be used to potentially load from the cache. + +When the `AboutRedirector` in the "privileged about content process" notices that a request has been made to `about:home`, it asks `nsIAboutNewTabService` to return a new `nsIChannel` for that document. The `AboutNewTabChildService` then checks to see if the `AboutHomeStartupCacheChild` can return an `nsIChannel` for any cached content. + +If, at this point, nothing has been streamed from the parent, we fall back to loading the dynamic `about:home` document. This might occur if the cache doesn't exist yet, or if we were too slow to pull it off of the disk. Subsequent attempts to load `about:home` will bypass the cache and load the dynamic document instead. This is true even if the privileged about content process crashes and a new one is created. + +The `AboutHomeStartupCacheChild` will also be responsible for generating the cache periodically. Periodically, the `AboutNewTabService` will send down the most up-to-date state for `about:home` from the parent process, and then the `AboutHomeStartupCacheChild` will generate document markup using ReactDOMServer within a `ChromeWorker`. After that's generated, the "privileged about content process" will send up `nsIInputStream` instances for both the markup and the script for the initial page state. The `AboutHomeStartupCache` singleton inside of `BrowserGlue` is responsible for receiving those `nsIInputStream`'s and persisting them in the HTTP cache for the next start. + +## What is cached? + +Two things are cached: + +1. The raw HTML mark-up of `about:home`. +2. A small chunk of JavaScript that "hydrates" the markup through the React libraries, allowing the page to become interactive after painting. + +The JavaScript being cached cannot be put directly into the HTML mark-up as inline script due to the CSP of `about:home`, which does not allow inline scripting. Instead, we load a script from `about:home?jscache`. This goes through the same mechanism for retrieving the HTML document from the cache, but instead pulls down the cached script. + +If the HTML mark-up is cached, then we presume that the script is also cached. We cannot cache one and not the other. If only one cache exists, or only one has been sent down to the "privileged about content process" by the time the `about:home` document is requested, then we fallback to loading the dynamic `about:home` document. + +## Refreshing the cache + +The cache is refreshed periodically by having `ActivityStreamMessageChannel` tell `AboutHomeStartupCache` when it has sent any messages down to the preloaded `about:newtab`. In general, such messages are a good hint that something visual has updated for the next `about:newtab`, and that the cache should probably be refreshed. + +`AboutHomeStartupCache` debounces notifications about such messages, since they tend to be bursty. + +## Invalidating the cache + +It's possible that the composition or layout of `about:home` will change over time from release to release. When this occurs, it might be desirable to invalidate any pre-existing cache that might exist for a user, so that they don't see an outdated `about:home` on startup. + +To do this, we set a version number on the cache entry, and ensure that the version number is equal to our expectations on startup. If the version number does not match our expectation, then the cache is discarded and the `about:home` document will be rendered dynamically. + +The version number is currently set to the application build ID. This means that when the application updates, the cache is invalidated on the first restart after a browser update is applied. + +## Handling errors + +`about:home` is typically the first thing that the user sees upon starting the browser. It is critically important that it function quickly and correctly. If anything happens to go wrong when retrieving or saving to the cache, we should fall back to generating the document dynamically. + +As an example, it's theoretically possible for the browser to crash while in the midst of saving to the cache. In that case, we might have a partial document saved, or a partial script saved - neither of which is acceptable. + +Thankfully, the HTTP cache was designed with resilience in mind, so partially written entries are automatically discarded, which allows us to fall back to the dynamic page generation mode. + +As additional redundancy to that resilience, we also make sure to create a new nsICacheEntry every time the cache is populated, and write the version metadata as the last step. Since the version metadata is written last, we know that if it's missing when we try to load the cache that the writing of the page and the script did not complete, and that we should fall back to dynamically rendering the page. diff --git a/browser/components/newtab/docs/v2-system-addon/data_dictionary.md b/browser/components/newtab/docs/v2-system-addon/data_dictionary.md new file mode 100644 index 0000000000..35bb084f38 --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/data_dictionary.md @@ -0,0 +1,360 @@ +# Activity Stream Pings + +The Activity Stream system add-on sends various types of pings to the backend (HTTPS POST) [Onyx server](https://github.com/mozilla/onyx) : + +- a `health` ping that reports whether or not a user has a custom about:home or about:newtab page +- a `session` ping that describes the ending of an Activity Stream session (a new tab is closed or refreshed), and +- an `event` ping that records specific data about individual user interactions while interacting with Activity Stream +- an `impression_stats` ping that records data about Pocket impressions and user interactions + +Schema definitions/validations that can be used for tests can be found in `system-addon/test/schemas/pings.js`. + +## Example Activity Stream `health` log + +```js +{ + "addon_version": "20180710100040", + "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e", + "locale": "en-US", + "version": "62.0a1", + "release_channel": "nightly", + "event": "AS_ENABLED", + "value": 10, + + // These fields are generated on the server + "date": "2016-03-07", + "ip": "10.192.171.13", + "ua": "python-requests/2.9.1", + "receive_at": 1457396660000 +} +``` + +## Example Activity Stream `session` Log + +```js +{ + // These fields are sent from the client + "action": "activity_stream_session", + "addon_version": "20180710100040", + "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e", + "locale": "en-US", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "session_duration": 1635, + "session_id": "{12dasd-213asda-213dkakj}", + "profile_creation_date": 14786, + "user_prefs": 7 + + // These fields are generated on the server + "date": "2016-03-07", + "ip": "10.192.171.13", + "ua": "python-requests/2.9.1", + "receive_at": 1457396660000 +} +``` + +## Example Activity Stream `user_event` Log + +```js +{ + "action": "activity_stream_user_event", + "action_position": "3", + "addon_version": "20180710100040", + "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e", + "event": "click or scroll or search or delete", + "locale": "en-US", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "source": "top sites, or bookmarks, or...", + "session_id": "{12dasd-213asda-213dkakj}", + "recommender_type": "pocket-trending", + "metadata_source": "MetadataService or Local or TippyTopProvider", + "user_prefs": 7 + + // These fields are generated on the server + "ip": "10.192.171.13", + "ua": "python-requests/2.9.1", + "receive_at": 1457396660000, + "date": "2016-03-07", +} +``` + +## Example Activity Stream `impression_stats` Logs + +```js +{ + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "addon_version": "20180710100040", + "locale": "en-US", + "source": "pocket", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"] + "tiles": [{"id": 10000}, {"id": 10001}, {"id": 10002}] + "user_prefs": 7, + "window_inner_width": 1000, + "window_inner_height" 900 +} +``` + +```js +{ + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "addon_version": "20180710100040", + "locale": "en-US", + "source": "pocket", + "page": "unknown", + "user_prefs": 7, + "window_inner_width": 1000, + "window_inner_height" 900, + + // "pos" is the 0-based index to record the tile's position in the Pocket section. + // "shim" is a base64 encoded shim attached to spocs, unique to the impression from the Ad server. + "tiles": [{"id": 10000, "pos": 0, "shim": "enuYa1j73z3RzxgTexHNxYPC/b,9JT6w5KB0CRKYEU+"}], + + // A 0-based index to record which tile in the "tiles" list that the user just interacted with. + "click|block|pocket": 0 +} +``` + +## Example Activity Stream `Router` Pings + +```js +{ + "client_id": "n/a", + "action": ["snippets_user_event" | "onboarding_user_event"], + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "source": "pocket", + "addon_version": "20180710100040", + "locale": "en-US", + "source": "NEWTAB_FOOTER_BAR", + "message_id": "some_snippet_id", + "event": "IMPRESSION", + "event_context": "{\"view\":\"application_menu\"}" +} +``` + +```{eval-rst} ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| KEY | DESCRIPTION | | ++============================+======================================================================================================================================================+==================+ +| ``action_position`` | [Optional] The index of the element in the ``source`` that was clicked. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``action`` | [Required] Either ``activity_stream_event`` or ``activity_stream_session``. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``addon_version`` | [Required] Firefox build ID, i.e. ``Services.appinfo.appBuildID``. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``advertiser`` | [Optional] An identifier for the advertiser used by the sponsored TopSites telemetry pings. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``client_id`` | [Required] An identifier for this client. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``card_type`` | [Optional] ("bookmark", "pocket", "trending", "pinned", "search", "spoc", "organic") | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``context_id`` | [Optional] An identifier used by the sponsored TopSites telemetry pings. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``search_vendor`` | [Optional] the vendor of the search shortcut, one of ("google", "amazon", "wikipedia", "duckduckgo", "bing", etc.). This field only exists when | | +| | ``card_type = "search"`` | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``date`` | [Auto populated by Onyx] The date in YYYY-MM-DD format. | :three: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``shield_id`` | [Optional] DEPRECATED: use `experiments` instead. The unique identifier for a specific experiment. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``event`` | [Required] The type of event. Any user defined string ("click", "share", "delete", "more\_items") | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``event_context`` | [Optional] A string to record the context of an AS Router event ping. Compound context values will be stringified by JSON.stringify | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``highlight_type`` | [Optional] Either ["bookmarks", "recommendation", "history"]. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``impression_id`` | [Optional] The unique impression identifier for a specific client. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``ip`` | [Auto populated by Onyx] The IP address of the client. | :two: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``locale`` | [Auto populated by Onyx] The browser chrome's language (eg. en-US). | :two: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``load_trigger_ts`` | [Optional][Server Counter][Server Alert for too many omissions] DOMHighResTimeStamp of the action perceived by the user to trigger the load of this | | +| | page. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``load_trigger_type`` | [Server Counter][Server Alert for too many omissions] Either ["first\_window\_opened", "menu\_plus\_or\_keyboard", "unexpected"]. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``metadata_source`` | [Optional] The source of which we computed metadata. Either (``MetadataService`` or ``Local`` or ``TippyTopProvider``). | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``page`` | [Required] One of ["about:newtab", "about:home", "about:welcome", "unknown" (which either means not-applicable or is a bug)]. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``position`` | [Optional] An integer indicating the placement (1-based) of the sponsored TopSites tile. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``recommender_type`` | [Optional] The type of recommendation that is being shown, if any. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``reporting_url`` | [Optional] A URL used by Mozilla services for impression and click reporting for the sponsored TopSites. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``session_duration`` | [Optional][Server Counter][Server Alert for too many omissions] Time in (integer) milliseconds of the difference between the new tab becoming visible| :one: | +| | and losing focus | | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``session_id`` | [Optional] The unique identifier for a specific session. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``source`` (AS) | [Required] Either ("recent\_links", "recent\_bookmarks", "frecent\_links", "top\_sites", "spotlight", "sidebar") and indicates what ``action``. | :two: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``source`` (CS) | [Optional] Either "newtab" or "urlbar" indicating the location of the TopSites pings for Contextual Services. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``received_at`` | [Auto populated by Onyx] The time in ms since epoch. | :three: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``tile_id`` | [Optional] An integer identifier for a sponsored TopSites tile. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``total_bookmarks`` | [Optional] The total number of bookmarks in the user's places db. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``total_history_size`` | [Optional] The number of history items currently in the user's places db. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``ua`` | [Auto populated by Onyx] The user agent string. | :two: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``unload_reason`` | [Required] The reason the Activity Stream page lost focus. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``url`` | [Optional] The URL of the recommendation shown in one of the highlights spots, if any. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``value`` (event) | [Optional] An object with keys "icon\_type", "card\_type" and "pocket\_logged\_in\_status" to record the extra information for event ping | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``ver`` | [Auto populated by Onyx] The version of the Onyx API the ping was sent to. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``highlights_size`` | [Optional] The size of the Highlights set. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| highlights_data_late_by_ms | [Optional] Time in ms it took for Highlights to become initialized | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| topsites_data_late_by_ms | [Optional] Time in ms it took for TopSites to become initialized | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| topsites_first_painted_ts | [Optional][Service Counter][Server Alert for too many omissions] Timestamp of when the Top Sites element finished painting (possibly with only | | +| | placeholder screenshots) | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``custom_screenshot`` | [Optional] Number of topsites that display a custom screenshot. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``screenshot_with_icon`` | [Optional] Number of topsites that display a screenshot and a favicon. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``screenshot`` | [Optional] Number of topsites that display only a screenshot. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``tippytop`` | [Optional] Number of topsites that display a tippytop icon. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``rich_icon`` | [Optional] Number of topsites that display a high quality favicon. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``no_image`` | [Optional] Number of topsites that have no screenshot. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``topsites_pinned`` | [Optional] Number of topsites that are pinned. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| topsites_search_shortcuts | [Optional] Number of search shortcut topsites. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| visibility_event_rcvd_ts | [Optional][Server Counter][Server Alert for too many omissions] DOMHighResTimeStamp of when the page itself receives an event that | | +| | document.visibilityState == visible. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``tiles`` | [Required] A list of tile objects for the Pocket articles. Each tile object mush have a ID, optionally a "pos" property to indicate the tile | | +| | position, and optionally a "shim" property unique to the impression from the Ad server | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``click`` | [Optional] An integer to record the 0-based index when user clicks on a Pocket tile. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``block`` | [Optional] An integer to record the 0-based index when user blocks a Pocket tile. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``pocket`` | [Optional] An integer to record the 0-based index when user saves a Pocket tile to Pocket. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``window_inner_width`` | [Optional] Amount of vertical space in pixels available to the window. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``window_inner_height`` | [Optional] Amount of horizontal space in pixels available to the window. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``user_prefs`` | [Required] The encoded integer of user's preferences. | :one: & :four: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``is_preloaded`` | [Required] A boolean to signify whether the page is preloaded or not | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``icon_type`` | [Optional] ("tippytop", "rich\_icon", "screenshot\_with\_icon", "screenshot", "no\_image", "custom\_screenshot") | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``region`` | [Optional] A string maps to pref "browser.search.region", which is essentially the two letter ISO 3166-1 country code populated by the Firefox | | +| | search service. Note that: 1). it reports "OTHER" for those regions with smaller Firefox user base (less than 10000) so that users cannot be | | +| | uniquely identified; 2). it reports "UNSET" if this pref is missing; 3). it reports "EMPTY" if the value of this pref is an empty string. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``profile_creation_date`` | [Optional] An integer to record the age of the Firefox profile as the total number of days since the UNIX epoch. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``message_id`` | [required] A string identifier of the message in Activity Stream Router. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``has_flow_params`` | [required] One of [true, false]. A boolean identifier that indicates if Firefox Accounts flow parameters are set or unset. | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``displayed`` | [required] 1: a SPOC is displayed; 0: non-displayed | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``reason`` | [required] The reason if a SPOC is not displayed, "n/a" for the displayed, one of ("frequency\_cap", "blocked\_by\_user", "flight\_duplicate", | | +| | "probability\_selection", "below\_min\_score", "out\_of\_position", "n/a") | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``full_recalc`` | [required] Is it a full SPOCS recalculation: 0: false; 1: true. Recalculation case: 1). fetch SPOCS from Pocket endpoint. Non-recalculation cases: | | +| | 1). An impression updates the SPOCS; 2). Any action that triggers the ``selectLayoutRender`` | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``experiments`` | [Optional] An object to record all active experiments (an empty object will be sent if there is no active experiment). The experiments IDs are | :one: | +| | stored as keys, and the value object stores the branch information. | | +| | `Example: {"experiment_1": {"branch": "control"}, "experiment_2": {"branch": "treatment"}}`. This deprecates the `shield_id` used in Activity Stream | | +| | and Messaging System. | | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``browser_session_id`` | [Optional] The unique identifier for a browser session, retrieved from TelemetrySession | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.source`` | [Optional] Referring partner domain, when install happens via a known partner | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.medium`` | [Optional] Category of the source, such as 'organic' for a search engine | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.campaign`` | [Optional] Identifier of the particular campaign that led to the download of the product | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.content`` | [Optional] Identifier to indicate the particular link within a campaign | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.experiment`` | [Optional] Funnel experiment identifier, see bug 1567339 | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.variantion`` | [Optional] Funnel experiment variant identifier, see bug 1567339 | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +| ``attribution.ua`` | [Optional] Derived user agent, see bug 1595063 | :one: | ++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+ +``` + +**Where:** + +- :one: Firefox data +- :two: HTTP protocol data +- :three: server augmented data +- :four: User preferences encoding table + +Note: the following session-related fields are not yet implemented in the system-addon, +but will likely be added in future versions: + +```js +{ + "total_bookmarks": 19, + "total_history_size": 9, + "highlights_size": 20 +} +``` + +## Encoding and decoding of `user_prefs` + +This encoding mapping was defined in `system-addon/lib/TelemetryFeed.jsm` + +```{eval-rst} ++-------------------+------------------------+ +| Preference | Encoded value (binary) | ++===================+========================+ +| `showSearch` | 1 (00000001) | ++-------------------+------------------------+ +| `showTopSites` | 2 (00000010) | ++-------------------+------------------------+ +| `showTopStories` | 4 (00000100) | ++-------------------+------------------------+ +| `showHighlights` | 8 (00001000) | ++-------------------+------------------------+ +| `showSnippets` | 16 (00010000) | ++-------------------+------------------------+ +| `showSponsored` | 32 (00100000) | ++-------------------+------------------------+ +| `showCFRAddons` | 64 (01000000) | ++-------------------+------------------------+ +| `showCFRFeatures` | 128 (10000000) | ++-------------------+------------------------+ +| `showSponsoredTopSites` | 256 (100000000) | ++-------------------+------------------------+ +``` + +Each item above could be combined with other items through bitwise OR (`|`) operation. + +Examples: + +- Everything is on, `user_prefs = 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 | 256 = 511` +- Everything is off, `user_prefs = 0` +- Only show search and Top Stories, `user_prefs = 1 | 4 = 5` +- Everything except Highlights, `user_prefs = 1 | 2 | 4 | 16 | 32 | 64 | 128 | 256 = 503` + +Likewise, one can use bitwise AND (`&`) for decoding. + +- Check if everything is shown, `user_prefs & (1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 | 256)` or `user_prefs == 511` +- Check if everything is off, `user_prefs == 0` +- Check if search is shown, `user_prefs & 1` +- Check if both Top Sites and Top Stories are shown, `(user_prefs & 2) && (user_prefs & 4)`, or `(user_prefs & (2 | 4)) == (2 | 4)` diff --git a/browser/components/newtab/docs/v2-system-addon/data_events.md b/browser/components/newtab/docs/v2-system-addon/data_events.md new file mode 100644 index 0000000000..5805f8bf9d --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/data_events.md @@ -0,0 +1,1417 @@ +# Metrics we collect + +By default, the about:newtab, about:welcome and about:home pages in Firefox (the pages you see when you open a new tab and when you start the browser), will send data back to Mozilla servers about usage of these pages. The intent is to collect data in order to improve the user's experience while using Activity Stream. Data about your specific browsing behaior or the sites you visit is **never transmitted to any Mozilla server**. At any time, it is easy to **turn off** this data collection by [opting out of Firefox telemetry](https://support.mozilla.org/kb/share-telemetry-data-mozilla-help-improve-firefox). + +Data is sent to our servers in the form of discrete HTTPS 'pings' or messages whenever you do some action on the Activity Stream about:home, about:newtab or about:welcome pages. We try to minimize the amount and frequency of pings by batching them together. Pings are sent in [JSON serialized format](http://www.js.org/). + +At Mozilla, [we take your privacy very seriously](https://www.mozilla.org/privacy/). The Activity Stream page will never send any data that could personally identify you. We do not transmit what you are browsing, searches you perform or any private settings. Activity Stream does not set or send cookies, and uses [Transport Layer Security](https://en.wikipedia.org/wiki/Transport_Layer_Security) to securely transmit data to Mozilla servers. + +Data collected from Activity Stream is retained on Mozilla secured servers for a period of 30 days before being rolled up into an anonymous aggregated format. After this period the raw data is deleted permanently. Mozilla **never shares data with any third party**. + +The following is a detailed overview of the different kinds of data we collect in the Activity Stream. See [data_dictionary.md](data_dictionary.md) for more details for each field. + +## Page takeover ping + +This ping is submitted once upon Activity Stream initialization if either about:home or about:newtab are set to a custom URL. It sends the category of the custom URL. It also includes the web extension id of the extension controlling the home and/or newtab page. + +```js +{ + "event": "PAGE_TAKEOVER_DATA", + "value": { + "home_url_category": [ + "search-engine" | + "search-engine-mozilla-tag" | + "search-engine-other-tag" | + "news-portal" | + "ecommerce" | + "social-media" | + "known-hijacker" | + "other" + ], + "home_extension_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "newtab_url_category": [ + "search-engine" | + "search-engine-mozilla-tag" | + "search-engine-other-tag" | + "news-portal" | + "ecommerce" | + "social-media" | + "known-hijacker" | + "other" + ], + "newtab_extension_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c" + }, + + // Basic metadata + "action": "activity_stream_event", + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "user_prefs": 7 +} +``` + +## User event pings + +These pings are captured when a user **performs some kind of interaction** in the add-on. + +### Basic shape + +A user event ping includes some basic metadata (tab id, addon version, etc.) as well as variable fields which indicate the location and action of the event. + +```js +{ + // This indicates the type of interaction + "event": [ + "CLICK", + "SEARCH", + "BLOCK", + "DELETE", + "IMPRESSION", + "OPEN_NEW_WINDOW", + "OPEN_PRIVATE_WINDOW", + "BOOKMARK_DELETE", + "BOOKMARK_ADD", + "OPEN_NEWTAB_PREFS", + "CLOSE_NEWTAB_PREFS", + "SEARCH_HANDOFF", + "SHOW_PERSONALIZE", + "HIDE_PERSONALIZE" + ], + + // Optional field indicating the UI component type + "source": "TOP_SITES", + + // Optional field if there is more than one of a component type on a page. + // It is zero-indexed. + // For example, clicking the second Highlight would result in an action_position of 1 + "action_position": 1, + + // Basic metadata + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "action": "activity_stream_event", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "user_prefs": 7 +} +``` + +### Types of user interactions + +#### Show/hide Personalization Panel + +```js +{ + "event": ["SHOW_PERSONALIZE" | "HIDE_PERSONALIZE"], // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Performing a search + +```js +{ + "event": "SEARCH", + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Performing a search handoff + +```js +{ + "event": "SEARCH_HANDOFF", + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Clicking a top site item + +```js +{ + "event": "CLICK", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "card_type": ["pinned" | "search" | "spoc"], + "icon_type": ["screenshot_with_icon" | "screenshot" | "tippytop" | "rich_icon" | "no_image" | "custom_screenshot"], + // only exists if its card_type = "search" + "search_vendor": "google" + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Clicking a top story item + +```js +{ + "event": "CLICK", + "source": ["CARDGRID" | "CARDGRID_WIDGET"], + "action_position": 2, + "value": { + // "spoc" for sponsored stories, "organic" for regular stories. + "card_type": ["organic" | "spoc" | "topics_widget"], + // topic and position only exists if its card_type = "topics_widget" + "topic": "entertainment" + // The index position of the topic link within the card + "position_in_card": 0 + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Clicking a popular topic link + +```js +{ + "event": "CLICK", + "source": "POPULAR_TOPICS", + "value": { + "topic": ["must-reads" | "productivity" | "health" | "finance" | "technology" | "more-recommendations"] + } + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Adding a search shortcut + +```js +{ + "event": "SEARCH_EDIT_ADD", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "search_vendor": "google" + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Showing privacy information + +```js +{ + "event": "SHOW_PRIVACY_INFO", + "source": "TOP_SITES", + "action_position": 2, + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Clicking on privacy information link + +```js +{ + "event": "CLICK_PRIVACY_INFO", + "source": "DS_PRIVACY_MODAL", + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Deleting a search shortcut + +```js +{ + "event": "SEARCH_EDIT_DELETE", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "search_vendor": "google" + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Deleting an item from history + +```js +{ + "event": "DELETE", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "card_type": "pinned", + "icon_type": ["screenshot_with_icon" | "screenshot" | "tippytop" | "rich_icon" | "no_image" | "custom_screenshot"] + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Blocking a site + +```js +{ + "event": "BLOCK", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "card_type": ["pinned" | "search" | "spoc"], + "icon_type": ["screenshot_with_icon" | "screenshot" | "tippytop" | "rich_icon" | "no_image" | "custom_screenshot"], + // only exists if its card_type = "search" + "search_vendor": "google" + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Bookmarking a link + +```js +{ + "event": "BOOKMARK_ADD", + "source": "HIGHLIGHTS", + "action_position": 2, + "value": { + "card_type": "trending" + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Removing a bookmark from a link + +```js +{ + "event": "BOOKMARK_DELETE", + "source": "HIGHLIGHTS", + "action_position": 2, + "value": { + "card_type": "bookmark" + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Opening a link in a new window + +```js +{ + "event": "OPEN_NEW_WINDOW", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "card_type": "pinned", + "icon_type": ["screenshot_with_icon" | "screenshot" | "tippytop" | "rich_icon" | "no_image" | "custom_screenshot"] + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Opening a link in a new private window + +```js +{ + "event": "OPEN_PRIVATE_WINDOW", + "source": "TOP_SITES", + "action_position": 2, + "value": { + "card_type": "pinned", + "icon_type": ["screenshot_with_icon" | "screenshot" | "tippytop" | "rich_icon" | "no_image" | "custom_screenshot"] + } + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Opening the new tab preferences pane + +```js +{ + "event": "OPEN_NEWTAB_PREFS", + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Closing the new tab preferences pane + +```js +{ + "event": "CLOSE_NEWTAB_PREFS", + + // Basic metadata + "action": "activity_stream_event", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Changing preferences from about:preferences#home or on the customize menu on the newtab page. + +```js +{ + "event": "PREF_CHANGED", + "source": ["TOP_STORIES" | "POCKET_SPOCS" | "HIGHLIGHTS" | "SNIPPETS" | "TOP_SITES" | "SPONSORED_TOP_SITES"], + "value": "{\"status\":true|false,\"menu_source\":\"ABOUT_PREFERENCES|CUSTOMIZE_MENU\"}" + "release_channel": "default", + "experiments": {}, + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "addon_version": "20180710100040", + "locale": "en-US", + "user_prefs": 7 +} +``` + +#### Pinning a tab + +```js +{ + "event": "TABPINNED", + "source": "TAB_CONTEXT_MENU", + "value": "{\"total_pinned_tabs\":2}", + + // Basic metadata + "action": "activity_stream_user_event", + "client_id": "aabaace5-35f4-7345-a28e-5502147dc93c", + "version": "67.0a1", + "addon_version": "20190218094427", + "locale": "en-US", + "user_prefs": 59, + "page": "n/a", + "session_id": "n/a", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3" +} +``` + +#### Adding or editing a new TopSite + +```js +{ + "event": "TOP_SITES_EDIT", + "source": "TOP_SITES_SOURCE", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + // "-1" Is used for prepending a new TopSite at the front of the list, while + // any other possible value is used for editing an existing TopSite slot. + "action_position": [-1 | "0..TOP_SITES_LENGTH"] +} +``` + +#### Requesting a custom screenshot preview + +```js +{ + "event": "PREVIEW_REQUEST", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "source": "TOP_SITES" +} +``` + +## Session end pings + +When a session ends, the browser will send a `"activity_stream_session"` ping to our metrics servers. This ping contains the length of the session, a unique reason for why the session ended, and some additional metadata. + +### Basic event + +All `"activity_stream_session"` pings have the following basic shape. Some fields are variable. + +```js +{ + "action": "activity_stream_session", + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "session_id": "005deed0-e3e4-4c02-a041-17405fd703f6", + "addon_version": "20180710100040", + "locale": "en-US", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "session_duration": 4199, + "profile_creation_date": 14786, + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "user_prefs": 7 +} +``` + +### What causes a session end? + +Here are different scenarios that cause a session end event to be sent: + +1. After a search +2. Clicking on something that causes navigation (top site, highlight, etc.) +3. Closing the browser +4. Refreshing +5. Navigating to a new URL via the url bar or file menu + +### Session performance data + +This data is held in a child object of the `activity_stream_session` event called `perf`. All fields suffixed by `_ts` are type `DOMHighResTimeStamp` (aka a double of milliseconds, with a 5 microsecond precision) with 0 being the [timeOrigin](https://developer.mozilla.org/en-US/docs/Web/API/DOMHighResTimeStamp#The_time_origin) of the browser's hidden chrome window. + +An example might look like this: + +```js +perf: { + // Timestamp of the action perceived by the user to trigger the load + // of this page. + // + // Not required at least for error cases where the + // observer event doesn't fire + "load_trigger_ts": 1, + + // What was the perceived trigger of the load action: + "load_trigger_type": [ + "first_window_opened" | // home only + "menu_plus_or_keyboard" | // newtab only + "unexpected" // sessions lacking actual start times + ], + + // when the page itself receives an event that document.visibilityStat=visible + "visibility_event_rcvd_ts": 2, + + // When did the topsites element finish painting? Note that, at least for + // the first tab to be loaded, and maybe some others, this will be before + // topsites has yet to receive screenshots updates from the add-on code, + // and is therefore just showing placeholder screenshots. + "topsites_first_painted_ts": 5, + + // The 6 different types of TopSites icons and how many of each kind did the + // user see. + "topsites_icon_stats": { + "custom_screenshot": 0, + "screenshot_with_icon": 2, + "screenshot": 1, + "tippytop": 2, + "rich_icon": 1, + "no_image": 0 + }, + + // The number of Top Sites that are pinned. + "topsites_pinned": 3, + + // The number of search shortcut Top Sites. + "topsites_search_shortcuts": 2, + + // How much longer the data took, in milliseconds, to be ready for display + // than it would have been in the ideal case. The user currently sees placeholder + // cards instead of real cards for approximately this length of time. This is + // sent when the first call of the component's `render()` method happens with + // `this.props.initialized` set to `false`, and the value is the amount of + // time in ms until `render()` is called with `this.props.initialized` set to `true`. + "highlights_data_late_by_ms": 67, + "topsites_data_late_by_ms": 35, + + // Whether the page is preloaded or not. + "is_preloaded": [true|false], +} +``` + +## Top Story pings + +When Top Story (currently powered by Pocket) is enabled in Activity Stream, the browser will send following `activity_stream_impression_stats` to our metrics servers. + +### Impression stats + +This reports all the Pocket recommended articles (a list of `id`s) when the user opens a newtab. + +```js +{ + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "addon_version": "20180710100040", + "locale": "en-US", + "source": "pocket", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "user_prefs": 7, + "window_inner_width": 1000, + "window_inner_height" 900, + "experiments": { + "experiment_1": {"branch": "control"}, + "experiment_2": {"branch": "treatment"} + }, + "tiles": [{"id": 10000}, {"id": 10001}, {"id": 10002}] +} +``` + +### Click/block/save_to_pocket ping + +This reports the user's interaction with those Pocket tiles. + +```js +{ + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "addon_version": "20180710100040", + "locale": "en-US", + "source": "pocket", + "page": ["about:newtab" | "about:home" | "about:welcome" | "unknown"], + "experiments": { + "experiment_1": {"branch": "control"}, + "experiment_2": {"branch": "treatment"} + }, + "user_prefs": 7, + "window_inner_width": 1000, + "window_inner_height" 900, + + // "pos" is the 0-based index to record the tile's position in the Pocket section. + // "shim" is a base64 encoded shim attached to spocs, unique to the impression from the Ad server. + "tiles": [{"id": 10000, "pos": 0, "shim": "enuYa1j73z3RzxgTexHNxYPC/b,9JT6w5KB0CRKYEU+"}], + + // A 0-based index to record which tile in the "tiles" list that the user just interacted with. + "click|block|pocket": 0 +} +``` + +## Save to Pocket button pings + +Right now the save to Pocket button, while technically outside of newtab, has some similarities with the newtab telemetry. + +These pings record user interaction with the save to Pocket button. + +### Click/impression ping + +```js +{ + "locale": "en-US", + "version": "83.0a1", + "release_channel": "default", + "model": "", + "events": [ + { + "action": "click|impression|unpin", + "position": 0, + "source": ["recs_learn_more" | "view_list" | "home_view_list" | "home_topic" | "home_discover" | "save_button" | "home_button" | "on_save_recs" | "learn_more" | "sign_up_1" | "sign_up_2" | "log_in"] + } + ], + "pocket_logged_in_state": true | false, + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "profile_creation_date": 18550 +} +``` + +## Activity Stream Router pings + +These pings record the impression and user interactions within Activity Stream Router. + +### Impression ping + +This reports the impression of Activity Stream Router. + +#### Snippets impression + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "action": "snippets_user_event", + "impression_id": "n/a", + "source": "SNIPPETS", + "addon_version": "20180710100040", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "NEWTAB_FOOTER_BAR", + "message_id": "some_snippet_id", + "event": "IMPRESSION" +} +``` + +CFR impression ping has two forms, in which the message_id could be of different meanings. + +#### CFR impression for all the prerelease channels and shield experiment + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "action": "cfr_user_event", + "impression_id": "n/a", + "addon_version": "20180710100040", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "CFR", + // message_id could be the ID of the recommendation, such as "wikipedia_addon" + "message_id": "wikipedia_addon", + "event": "IMPRESSION" +} +``` + +#### CFR impression for the release channel + +```js +{ + "client_id": "n/a", + "action": "cfr_user_event", + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "addon_version": "20180710100040", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "CFR", + // message_id should be a bucket ID in the release channel, we may not use the + // individual ID, such as addon ID, per legal's request + "message_id": "bucket_id", + "event": "IMPRESSION" +} +``` + +### User interaction pings + +This reports the user's interaction with Activity Stream Router. + +#### Snippets interaction pings + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "action": "snippets_user_event", + "addon_version": "20180710100040", + "impression_id": "n/a", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "NEWTAB_FOOTER_BAR", + "message_id": "some_snippet_id", + "event": ["CLICK_BUTTION" | "BLOCK"] +} +``` + +#### CFR interaction pings for all the prerelease channels and shield experiment + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "action": "cfr_user_event", + "addon_version": "20180710100040", + "impression_id": "n/a", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "CFR", + // message_id could be the ID of the recommendation, such as "wikipedia_addon" + "message_id": "wikipedia_addon", + "event": ["IMPRESSION" | "INSTALL" | "PIN" | "BLOCK" | "DISMISS" | "RATIONALE" | "LEARN_MORE" | "CLICK" | "CLICK_DOORHANGER" | "MANAGE"], + // "modelVersion" records the model identifier for the CFR machine learning experiment, see more detail in Bug 1594422. + // Non-experiment users will not report this field. + "event_context": "{ \"modelVersion\": \"some_model_version_id\" }" +} +``` + +#### CFR interaction pings for release channel + +```js +{ + "client_id": "n/a", + "action": "cfr_user_event", + "addon_version": "20180710100040", + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "CFR", + // message_id should be a bucket ID in the release channel, we may not use the + // individual ID, such as addon ID, per legal's request + "message_id": "bucket_id", + "event": ["IMPRESSION" | "INSTALL" | "PIN" | "BLOCK" | "DISMISS" | "RATIONALE" | "LEARN_MORE" | "CLICK" | "CLICK_DOORHANGER" | "MANAGE"], +} +``` + +### Targeting error pings + +This reports when an error has occurred when parsing/evaluating a JEXL targeting string in a message. + +```js +{ + "client_id": "n/a", + "action": "asrouter_undesired_event", + "addon_version": "20180710100040", + "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "message_id": "some_message_id", + "event": "TARGETING_EXPRESSION_ERROR", + "event_context": ["MALFORMED_EXPRESSION" | "OTHER_ERROR"] +} +``` + +### Remote Settings error pings + +This reports a failure in the Remote Settings loader to load messages for Activity Stream Router. + +```js +{ + "action": "asrouter_undesired_event", + "client_id": "n/a", + "addon_version": "20180710100040", + "locale": "en-US", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "user_prefs": 7, + "event": ["ASR_RS_NO_MESSAGES" | "ASR_RS_ERROR"], + // The value is set to the ID of the message provider. For example: remote-cfr, remote-onboarding, etc. + "event_context": "REMOTE_PROVIDER_ID" +} +``` + +## Toolbar Badge interaction pings + +This reports when a user has seen or clicked a badge/notification in the browser toolbar in a non-PBM window + +```js +{ + "locale": "en-US", + "client_id": "9da773d8-4356-f54f-b7cf-6134726bcf3d", + "version": "70.0a1", + "release_channel": "default", + "addon_version": "20190712095934", + "action": "cfr_user_event", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "CFR", + "message_id": "FXA_ACCOUNTS_BADGE", + "event": ["CLICK" | "IMPRESSION"] +} +``` + +## Panel interaction pings + +This reports when a user opens the panel, views messages and clicks on a message. +For message impressions we concatenate the ids of all messages in the panel. + +```js +{ + "locale": "en-US", + "client_id": "9da773d8-4356-f54f-b7cf-6134726bcf3d", + "version": "70.0a1", + "release_channel": "default", + "addon_version": "20190712095934", + "action": "cfr_user_event", + "experiments": { + "experiment_1": { "branch": "control" }, + "experiment_2": { "branch": "treatment" } + }, + "source": "CFR", + "message_id": "WHATS_NEW_70", + "event": ["CLICK" | "IMPRESSION"], + "value": { "view": ["application_menu" | "toolbar_dropdown"] } +} +``` + +We also report when the panel checkbox (used to allow users to opt out of +notifications) is checked or unchecked. + +```js +{ + ... + "message_id": "n/a", + "event": "WNP_PREF_TOGGLE", + "value": { "prefValue": true } +} +``` + +## Moments page pings + +This reports when a moments page message has set the user preference for +`browser.startup.homepage_override.once`. It goes through the same policy +as other CFR messages. + +```js +// Release ping +{ + "action": "cfr_user_event" + "addon_version": "20200225022813" + "bucket_id": "WNP_THANK_YOU" + "event": "MOMENTS_PAGE_SET" + "impression_id": "{d85d2268-b751-9543-b6d7-aad523bf2b26}" + "experiments": { + "experiment_1": {"branch": "control"}, + "experiment_2": {"branch": "treatment"} + }, + "locale": "en-US" + "message_id": "n/a" + "source": "CFR" +} + +// Beta and Nightly channels +{ + "source": "CFR", + "message_id": "WNP_THANK_YOU", + "bucket_id": "WNP_THANK_YOU", + "event": "MOMENTS_PAGE_SET", + "addon_version": "20200225022813", + "experiments": { + "experiment_1": {"branch": "control"}, + "experiment_2": {"branch": "treatment"} + }, + "locale": "en-US", + "client_id": "21dc1375-b24e-984b-83e9-c8a9660ae4ff" +} +``` + +## InfoBar pings + +This reports when the user interacts with the browser infobar (messaging area +located at the top of the content area). Similar policy applied as for the +What's New panel client_id is reported in all the channels. + +```js +{ + "experiments": { + "exp1": { + "branch": "treatment-a" + } + }, + "addon_version": "20210115035053", + "release_channel": "release", + "locale": "en-US", + "event": [ + "IMPRESSION", + "CLICK_PRIMARY_BUTTON", + "CLICK_SECONDARY_BUTTON", + "DISMISSED" + ], + "client_id": "c4beb4bf-4feb-9c4e-9587-9323b28c2e50", + "version": "86", + "message_id": "INFOBAR_ACTION_86", + "browser_session_id": "93714e76-9919-ca49-b697-5e7c09a1394f" +} +``` + +## Spotlight pings + +This reports when the user interacts with the Messaging System Spotlight component +Similar policy applied as for the Infobar messages: client_id is reported in all +the channels. Currently this is only used in experiments. + +```js +{ + "experiments": { + "exp1": { + "branch": "treatment-a" + } + }, + "addon_version": "20210115035053", + "release_channel": "release", + "locale": "en-US", + "event": ["IMPRESSION" | "CLICK" | "DISMISS"], + "client_id": "c4beb4bf-4feb-9c4e-9587-9323b28c2e50", + "version": "93", + "message_id": "SPOTLIGHT_MESSAGE_93", + "browser_session_id": "93714e76-9919-ca49-b697-5e7c09a1394f" +} +``` + +## ToastNotification pings + +This reports when the user interacts with a toast notification: an OS-level +toast notification UI affordance or a pop-up OS-level window. Similar policy +applied as for the What's New panel: client_id is reported in all the +channels. Currently this is only used in experiments. + +```js +{ + "experiments": { + "exp1": { + "branch": "treatment-a" + } + }, + "addon_version": "20210115035053", + "release_channel": "release", + "locale": "en-US", + "event": ["IMPRESSION"], + "client_id": "c4beb4bf-4feb-9c4e-9587-9323b28c2e50", + "version": "86", + "message_id": "TOAST_NOTIFICATION_EXAMPLE_ID", + "browser_session_id": "93714e76-9919-ca49-b697-5e7c09a1394f" +} +``` + +## Messaging-experiments pings + +As the new experiment platform, the Messaging experiment manager is now managing & operating all the experiments of Firefox Messaging System, including the first-run experience (about:welcome), CFR, Whats-new-panel, Moments Page, and Snippets. + +### Enrollment & Unenrollment pings + +Under the hood, the experiment manager makes use of Normandy API for experiment management (enrollment & unenrollment as well as the corresponding telemetry). Therefore, the enrollment & unenrollment pings are collected through the Normandy counterparts. See [`normandy` category](https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml#441) for more details. + +### Experiment reach ping + +This records whether a branch's targeting is satisfied for Messaging System experiments. All qualified branch ID(s) will be recorded in the `extra_keys` for each active experiment, and the event `value` will be the experiment ID (i.e. slug). + +Unlike other Activity Stream pings, this is a Firefox Events telemetry event, and it is sent only for users enrolled in a Messaging System experiment. + +```js +{ + "category": "messaging_experiments", + "method": "reach", + // any of ["cfr", "whats_new_panel", "moments_page", "snippets", "cfr_fxa"] + "object": "cfr" + "value": "experiment_message_id", + "extra_keys": { + // A semicolon separated string with all the qualified branch IDs + "branches": "control;variant_1;variant_2" + } +} +``` + +### Experiment attribute errors + +This records whether issues were encountered with any of the targeting attributes used in the experiment enrollment or message targeting. +Two different types of events are sent: `attribute_error` and `attribute_timeout` along with the attribute that caused it. An attribute +is a variable inside the JEXL targeting expression that is evaluated client side by the browser. + +```js +{ + "messaging_experiments", + "targeting", + "attribute_error", // event + "foo", // attribute, + "extra_keys": { + "source": "message id or experiment slug", + }, +}, +{ + "messaging_experiments", + "targeting", + "attribute_timeout", // event + "bar", // attribute, + "extra_keys": { + "source": "message id or experiment slug", + }, +} +``` + +## Firefox Onboarding (about:welcome) pings + +These record the telemetry metrics during the Firefox onboarding experience. + +### Onboarding impressions + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "version": "76.0a1", + "locale": "en-US", + "experiments": {}, + "release_channel": "default", + "addon_version": "20200330194034", + "message_id": [ + | "DEFAULT_ABOUTWELCOME" + | "DEFAULT_ABOUTWELCOME_AW_GET_STARTED" + | "DEFAULT_ABOUTWELCOME_SITES" + | "DEFAULT_ABOUTWELCOME_AW_IMPORT_SETTINGS" + | "DEFAULT_ABOUTWELCOME_AW_CHOOSE_THEME" + | "RTAMO_DEFAULT_WELCOME" + ], + "event": "IMPRESSION", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "event_context": { "page": "about:welcome" }, + "attribution": { + "source": "mozilla.org", + "medium": "referral", + "campaign": "Firefox-Brand-US-Mozilla-Org", + "content": "test-addon@github.io", + "experiment": "ua-onboarding", + "variation": "chrome", + "ua": "firefox" + } +} +``` + +### Onboarding button clicks + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "version": "76.0a1", + "locale": "en-US", + "experiments": {}, + "release_channel": "default", + "addon_version": "20200330194034", + "message_id": [ + | "DEFAULT_ABOUTWELCOME_AW_GET_STARTED" + | "DEFAULT_ABOUTWELCOME_AW_IMPORT_SETTINGS" + | "DEFAULT_ABOUTWELCOME_AW_CHOOSE_THEME" + | "RTAMO_DEFAULT_WELCOME" + ], + "event": "CLICK_BUTTION", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "event_context": { "page": "about:welcome", "source": ["primary_button" | "secondary_button"] }, + "attribution": { + "source": "mozilla.org", + "medium": "referral", + "campaign": "Firefox-Brand-US-Mozilla-Org", + "content": "test-addon@github.io", + "experiment": "ua-onboarding", + "variation": "chrome", + "ua": "firefox" + } +} +``` + +### Onboarding Return-To-AMO install ping + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "version": "76.0a1", + "locale": "en-US", + "experiments": {}, + "release_channel": "default", + "addon_version": "20200330194034", + "message_id": "RTAMO_DEFAULT_WELCOME", + "event": "INSTALL", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "event_context": { "page": "about:welcome", "source": "ADD_EXTENSION_BUTTON" }, + "attribution": { + "source": "mozilla.org", + "medium": "referral", + "campaign": "Firefox-Brand-US-Mozilla-Org", + "content": "test-addon@github.io", + "experiment": "ua-onboarding", + "variation": "chrome", + "ua": "firefox" + } +} +``` + +### Onboarding embedded migration interactions + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "version": "76.0a1", + "locale": "en-US", + "experiments": {}, + "release_channel": "default", + "addon_version": "20200330194034", + "message_id": "MR_WELCOME_DEFAULT_0_AW_IMPORT_SETTINGS_EMBEDDED", + "event": "CLICK_BUTTION", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "event_context": { + "page": "about:welcome", + "source": ["primary_button" | "migrate_close"], + } +} +``` + +### Onboarding session end ping + +```js +{ + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "version": "76.0a1", + "locale": "en-US", + "experiments": {}, + "release_channel": "default", + "addon_version": "20200330194034", + "message_id": "DEFAULT_ABOUTWELCOME", + "event": "SESSION_END", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3", + "event_context": { + "page": "about:welcome", + "reason": [ + | "welcome-window-closed" + | "welcome-tab-closed" + | "app-shut-down" + | "address-bar-navigated" + ] + }, + "attribution": { + "source": "mozilla.org", + "medium": "referral", + "campaign": "Firefox-Brand-US-Mozilla-Org", + "content": "test-addon@github.io", + "experiment": "ua-onboarding", + "variation": "chrome", + "ua": "firefox" + } +} +``` + +## Feature Callout pings + +Most interactions and impressions in Feature Callouts are handled by the Onboarding pings above, but since Feature Callouts are inserted into other content, they have some additional capabilities. + +### Page event ping for Feature Callouts + +Feature Callout messages can include properties to listen for specific events on the host page and perform an action when fired. For example, Feature Callouts pointing to a button can be dismissed when that button is clicked. This metric records the event type, event target, and action. + +```js +{ + "experiments": {}, + "locale": "en-US", + "version": "107.0a1", + "release_channel": "default", + "event": "PAGE_EVENT", + "event_context": { + "action": "DISMISS", + "reason": "CLICK", + "source": "button.primary#some-button", + "page": "about:firefoxview" + }, + "message_id": "FIREFOX_VIEW_TAB_PICKUP_REMINDER", + "addon_version": "20221013100028", + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3" +} +``` + +### Feature Callout dismissal + +A Feature Callout can be dismissed by clicking its dismiss button directly or pressing the Escape key. With a page event listener, it can also be configured to dismiss the callout when an event in the page happens, such as clicking the target of the callout (see the page event ping above). + +```js +{ + "experiments": {}, + "locale": "en-US", + "version": "108.0a1", + "release_channel": "default", + "event": "DISMISS", + "event_context": { + "source": [ + "dismiss_button", + "KEY_Escape", + "PAGE_EVENT:button.primary#some-button" + ], + "page": ["about:firefoxview", "chrome://browser/content/browser.xhtml"] + }, + "message_id": "some_feature_callout_id", + "addon_version": "20221013100028", + "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c", + "browser_session_id": "e7e52665-7db3-f348-9918-e93160eb2ef3" +} +``` + +## Sponsored TopSites pings + +These record the impression and click pings for the Sponsored TopSites. + +### Impression ping for the sponsored TopSites + +```js +{ + "context_id": "{94642acb-4996-034b-916c-147da723cc41}", + "advertiser": "test-advertiser", + "tile_id": 42, + "source": "newtab", + "position": 1, + "reporting_url": "https://test.reporting.net/static?id=7RHzfOIWHG7kJnEYgGeuJm7vJ%3DEkiFXwxYIZjF8XgClWfC 8%2B7R4dHQ8zjmENj%3DNwxwz%2FJmdw7R4dHQfz4Z2ZfplnHG3Z5FwqgCfX4ZLafC2mfBINI9HuiF2 z4Z2ZfplnHmcux%3DcvImauiF2zfQlWfpDX7R3%2Bx%3DDuiF2zfLyy", + "version": "76.0a1", + "locale": "en-US", + "release_channel": "default", + "experiments": { + "exp_id_foo": { + "branch": "control" + }, + "exp_id_bar": { + "branch": "treatment" + } + } +} +``` + +### Click ping for the sponsored TopSites + +```js +{ + "context_id": "{94642acb-4996-034b-916c-147da723cc41}", + "tile_id": 42, + "source": "newtab", + "position": 1, + "advertiser": "test-advertiser", + "reporting_url": "https://test.reporting.net/ctp?version=16.0.0&key=1598991514900100001.1&ci=15 98991514792.12747&ctag=1598291839300100006&aespFlag=altinst", + "version": "76.0a1", + "locale": "en-US", + "release_channel": "default", + "experiments": { + "exp_id_foo": { + "branch": "control" + }, + "exp_id_bar": { + "branch": "treatment" + } + } +} +``` + +## Glean "newtab" ping + +Unlike the other data collections, this is a +[Glean Ping](https://mozilla.github.io/glean/book/user/pings/index.html) +that batches events and metadata about newtab sessions. + +You can find full documentation about this ping and its contents in +[its Glean Dictionary entry](https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/pings/newtab). diff --git a/browser/components/newtab/docs/v2-system-addon/geo_locale.md b/browser/components/newtab/docs/v2-system-addon/geo_locale.md new file mode 100644 index 0000000000..a83275489e --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/geo_locale.md @@ -0,0 +1,23 @@ +# Custom `geo`, `locale`, and update channels + +There are instances where you may need to change your local build's locale, geo, and update channel (such as changes to the visibility of Discovery Stream on a per-geo/locale basis in `ActivityStream.jsm`). + +## Changing update channel + +- Change `app.update.channel` to desired value (eg: `release`) by editing `LOCAL_BUILD/Contents/Resources/defaults/pref/channel-prefs.js`. (**NOTE:** Changing pref `app.update.channel` from `about:config` seems to have no effect!) + +## Changing geo + +- Set `browser.search.region` to desired geo (eg `CA`) + +## Changing locale + +*Note: These prefs are only configurable on a nightly or local build.* + +- Toggle `extensions.langpacks.signatures.required` to `false` +- Toggle `xpinstall.signatures.required` to `false` +- Toggle `intl.multilingual.downloadEnabled` to `true` +- Toggle `intl.multilingual.enabled` to `true` +- For Mac and Linux builds, open the [langpack](https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/linux-x86_64/xpi/) for target locale in your local build (eg `firefox-70.0a1.en-CA.langpack.xpi` if you want an `en-CA` locale). +- For Windows, use [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/](https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/) +- In `about:preferences` click "Set Alternatives" under "Language", move desired locale to the top position, click OK, click "Apply And Restart" diff --git a/browser/components/newtab/docs/v2-system-addon/mochitests.md b/browser/components/newtab/docs/v2-system-addon/mochitests.md new file mode 100644 index 0000000000..c0b1b7b2a6 --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/mochitests.md @@ -0,0 +1,28 @@ +# Mochitests + +We use [mochitests](https://firefox-source-docs.mozilla.org/testing/mochitest-plain/) to do functional (and possibly integration) testing. Mochitests are part of Firefox and allow us to test activity stream literally as you would use it. + +Mochitests live in `test/functional/mochitest`, and as of this writing, they + are all the [`browser-chrome`](https://firefox-source-docs.mozilla.org/testing/chrome-tests/) flavor of mochitests. They currently only run against the bootstrapped version of the add-on in system-addon, not the test pilot version at the top level directory. + +## Adding New Tests + +If you add new tests, make sure to list them in the `browser.ini` file. You will see the other tests there. Add a new entry with the same format as the others. You can also add new JS or HTML files by listing in under `support-files`. Make sure to start your test name with "browser_", so that the test suite knows the pick it up. E.g: "browser_as_my_new_test.js". + +## Writing Tests + +Here are a few tips for writing mochitests: + +* Only write mochitests for testing the interaction of multiple components on the page and to make sure that the protocol is working. +* If you need to access the content page, use `ContentTask.spawn`: + +```js +ContentTask.spawn(gBrowser.selectedBrowser, null, function* () { + content.wrappedJSObject.foo(); +}); +``` + +The above calls the function `foo` that exists in the page itself. You can also access the DOM this way: `content.document.querySelector`, if you want to click a button or do other things. You can even you use assertions inside this callback to check DOM state. + +* If you run into problems running tests in e10s, refer to the [wiki](https://wiki.mozilla.org/Electrolysis/e10s_test_tips) for tips +* Nobody likes to see intermittent oranges in their tests, so read the [docs on how to avoid them](https://firefox-source-docs.mozilla.org/testing/intermittent/)! diff --git a/browser/components/newtab/docs/v2-system-addon/preferences.md b/browser/components/newtab/docs/v2-system-addon/preferences.md new file mode 100644 index 0000000000..cda103b421 --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/preferences.md @@ -0,0 +1,278 @@ +# Preferences + +## Preference branch + +The preference branch for activity stream is `browser.newtabpage.activity-stream.`. +Any preferences defined in the preference configuration will be relative to that +branch. For example, if a preference is defined with the name `foo`, the full +preference as it is displayed in `about:config` will be `browser.newtabpage.activity-stream.foo`. + +## Defining new preferences + +All preferences for Activity Stream should be defined in the `PREFS_CONFIG` Array +found in `lib/ActivityStream.jsm`. +The configuration object should have a `name` (the name of the pref), a `title` +that describes the functionality of the pref, and a `value`, the default value +of the pref. Optionally a `getValue` function can be provided to dynamically +generate a default pref value based on args, e.g., geo and locale. For +developers-specific defaults, an optional `value_local_dev` will be used instead +of `value`. For example: + +```js +{ + name: "telemetry.log", + title: "Log telemetry events in the console", + value: false, + value_local_dev: true, + getValue: ({geo}) => geo === "CA" +} +``` + +### IMPORTANT: Setting test-specific values for Mozilla Central + +If a feed or feature behind a pref makes any network calls or would other be +disruptive for automated tests and that pref is on by default, make sure you +disable it for tests in Mozilla Central. + +You should create a bug in Bugzilla and a patch that adds lines to turn off your +pref in the following files: + +- layout/tools/reftest/reftest-preferences.js +- testing/profiles/prefs_general.js +- testing/talos/talos/config.py + +You can see an example in [this patch](https://github.com/mozilla/activity-stream/pull/2977). + +## Reading, setting, and observing preferences from `.jsm`s + +To read/set/observe Activity Stream preferences, construct a `Prefs` instance found in `lib/ActivityStreamPrefs.jsm`. + +```js +// Import Prefs +XPCOMUtils.defineLazyModuleGetter( + this, + "Prefs", + "resource://activity-stream/lib/ActivityStreamPrefs.jsm" +); + +// Create an instance +const prefs = new Prefs(); +``` + +The `Prefs` utility will set the Activity Stream branch for you by default, so you +don't need to worry about prefixing every pref with `browser.newtabpage.activity-stream.`: + +```js +const prefs = new Prefs(); + +// This will return the value of browser.newtabpage.activity-stream.foo +prefs.get("foo"); + +// This will set the value of browser.newtabpage.activity-stream.foo to true +prefs.set("foo", true); + +// This will call aCallback when browser.newtabpage.activity-stream.foo is changed +prefs.observe("foo", aCallback); + +// This will stop listening to browser.newtabpage.activity-stream.foo +prefs.ignore("foo", aCallback); +``` + +See :searchfox:`toolkit/modules/Preferences.sys.mjs <source/toolkit/modules/Preferences.sys.mjs>` +for more information about what methods are available. + +## Discovery Stream Preferences + +Preferences specific to the Discovery Stream are nested under the sub-branch `browser.newtabpage.activity-stream.discoverystream` (with the exception of `browser.newtabpage.blocked`). + +### `browser.newtabpage.activity-stream.discoverystream.flight.blocks` + +- Type: `string (JSON)` +- Default: `{}` +- Pref Type: AS + +Not intended for user configuration, but is programmatically updated. Used for tracking blocked flight IDs when a user dismisses a SPOC. Keys are flight IDs. Values don't have a specific meaning. + +### `browser.newtabpage.blocked` + +- Type: `string (JSON)` +- Default: `null` +- Pref Type: AS + +Not intended for user configuration, but is programmatically updated. Used for tracking blocked story IDs when a user dismisses one. Keys are story IDs. Values don't have a specific meaning. + +### `browser.newtabpage.activity-stream.discoverystream.config` + +- Type `string (JSON)` +- Default: + ```json + { + "api_key_pref": "extensions.pocket.oAuthConsumerKey", + "collapsible": true, + "enabled": true, + "show_spocs": true, + "hardcoded_layout": true, + "personalized": true, + "layout_endpoint": "https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic" + } + ``` + - `api_key_pref` (string): The name of a variable containing the key for the Pocket API. + - `collapsible` (boolean): Controls whether the sections in new tab can be collapsed. + - `enabled` (boolean): Controls whether DS is turned on and is programmatically set based on a user's locale. DS enablement is a logical `AND` of this and the value of `browser.newtabpage.activity-stream.discoverystream.enabled`. + - `show_spocs` (boolean): Show sponsored content in new tab. + - `hardcoded_layout` (boolean): When this is true, a hardcoded layout shipped with Firefox will be used instead of a remotely fetched layout definition. + - `personalized` (boolean): When this is `true` personalized content based on browsing history will be displayed. + - `layout_endpoint` (string): The URL for a remote layout definition that will be used if `hardcoded_layout` is `false`. + - `unused_key` (string): This is not set by default and is unused by this codebase. It's a standardized way to differentiate configurations to prevent experiment participants from being unenrolled. + +### `browser.newtabpage.activity-stream.discoverystream.enabled` + +- Type: `boolean` +- Default: `true` +- Pref Type: Firefox + +When this is set to `true` the Discovery Stream experience will show up if `enabled` is also `true` on `browser.newtabpage.activity-stream.discoverystream.config`. Otherwise the old Activity Stream experience will be shown. + +### `browser.newtabpage.activity-stream.discoverystream.endpointSpocsClear` + +- Type: `string (URL)` +- Default: `https://spocs.getpocket.com/user` +- Pref Type: AS + +Endpoint for when a user opts-out of sponsored content to delete the corresponding data from the ad server. + +### `browser.newtabpage.activity-stream.discoverystream.endpoints` + +- Type: `string (URLs, CSV)` +- Default: `https://getpocket.cdn.mozilla.net/,https://spocs.getpocket.com/` +- Pref Type: AS + +A list of endpoints that are allowed to be used by Discovery Stream for remote content (eg: story metadata) and configuration (eg: remote layout definitions for experimentation). + +### `browser.newtabpage.activity-stream.discoverystream.hardcoded-basic-layout` + +- Type: `boolean` +- Default: `false` +- Pref Type: Firefox + +If this is `false` the default hardcoded layout is used, and if it's `true` then an alternate hardcoded layout (that currently simulates the older AS experience) is used. + +### `browser.newtabpage.activity-stream.discoverystream.rec.impressions` + +- Type: `string (JSON)` +- Default: `{}` +- Pref Type: AS + +Programmatically generated hash table where the keys are recommendation IDs and the values are timestamps representing the first impression. + +### `browser.newtabpage.activity-stream.discoverystream.spoc.impressions` + +- Type: `string (JSON)` +- Default: `{}` +- Pref Type: AS + +Programmatically generated hash table where the keys are sponsored content IDs and the values are arrays of timestamps for every impression. + +### `browser.newtabpage.activity-stream.discoverystream.locale-list-config` + +- Type: `string (CSV, locales)` +- Default: `null` +- Pref Type: Firefox + +A comma separated list of locales that by default have stories enabled in newtab. It overrides what might be in region-stories-config. So if I set this to "en-US,en-CA,en-GB", all users with a English browser would see newtab stories, even if their region was not in region-stories-config list. + +### `browser.newtabpage.activity-stream.discoverystream.region-stories-config` + +- Type: `string (CSV, regions)` +- Default: `US,DE,CA,GB,IE,CH,AT,BE` +- Pref Type: Firefox + +A comma separated list of geos that by default have stories enabled in newtab. It matches the client's geo with that list, then looks for a matching locale. + +### `browser.newtabpage.activity-stream.discoverystream.region-spocs-config` + +- Type: `string (CSV, regions)` +- Default: `US,CA,DE` +- Pref Type: Firefox + +A comma separated list of geos that by default have spocs enabled in newtab. It matches the client's geo with that list. + +### `browser.newtabpage.activity-stream.discoverystream.region-layout-config` + +- Type: `string (CSV, regions)` +- Default: `US,CA,GB,DE,IE,CH,AT,BE` +- Pref Type: Firefox + +A comma separated list of geos that have 7 rows of stories enabled in newtab. It matches the client's geo with that list. + +### `browser.newtabpage.activity-stream.discoverystream.region-basic-layout` + +- Type: `boolean` +- Default: false +- Pref Type: AS + +If this is `true` newtabs with stories enabled see 1 row. It is set programmatically based on the result from region-layout-config. + +### `browser.newtabpage.activity-stream.discoverystream.spocs-endpoint` + +- Type: `string (URL)` +- Default: `null` +- Pref Type: Firefox + +Override to specify endpoint for SPOCs. Will take precedence over remote and hardcoded layout SPOC endpoints. + +### `browser.newtabpage.activity-stream.discoverystream.personalization.version` + +- Type: `integer` +- Default: `1` +- Pref Type: Firefox + +This controls what version of personalization we should use to score newtab stories. + +### `browser.newtabpage.activity-stream.discoverystream.personalization.modelKeys` + +- Type: `string (CSV)` +- Default: `nb_model_arts_and_entertainment, nb_model_autos_and_vehicles, nb_model_beauty_and_fitness, nb_model_blogging_resources_and_services, nb_model_books_and_literature, nb_model_business_and_industrial, nb_model_computers_and_electronics, nb_model_finance, nb_model_food_and_drink, nb_model_games, nb_model_health, nb_model_hobbies_and_leisure, nb_model_home_and_garden, nb_model_internet_and_telecom, nb_model_jobs_and_education, nb_model_law_and_government, nb_model_online_communities, nb_model_people_and_society, nb_model_pets_and_animals, nb_model_real_estate, nb_model_reference, nb_model_science, nb_model_shopping, nb_model_sports, nb_model_travel` +- Pref Type: Firefox + +This is a configuration for personalization version 2. It is a list of topics the algorithm uses to score stories by. + +### `browser.newtabpage.activity-stream.discoverystream.recs.personalized` + +- Type: `boolean` +- Default: false +- Pref Type: Firefox + +This controls if newtab story personalization includes regular stories or not. See spocs.personalized for sponsored content. + +### `browser.newtabpage.activity-stream.discoverystream.spocs.personalized` + +- Type: `boolean` +- Default: true +- Pref Type: Firefox + +This controls if newtab story personalization includes sponsored content or not. See recs.personalized for regular stories. + +### `browser.newtabpage.activity-stream.discoverystream.isCollectionDismissible` + +- Type: `boolean` +- Default: true +- Pref Type: Firefox + +This controls if newtab story collections are dismissible or not. + +### `browser.newtabpage.activity-stream.feeds.section.topstories` + +- Type: `boolean` +- Default: true +- Pref Type: Firefox + +This controls if the user should see newtab stories or not. It is set by the user via about:preferences#home + +### `browser.newtabpage.activity-stream.feeds.system.topstories` + +- Type: `boolean` +- Default: false +- Pref Type: AS + +Not intended for user configuration, but is programmatically set. It also controls if the user should see newtab stories or not. It is set at run time, and computed based on the locale/region. diff --git a/browser/components/newtab/docs/v2-system-addon/remote_cfr.md b/browser/components/newtab/docs/v2-system-addon/remote_cfr.md new file mode 100644 index 0000000000..6f3ef344ee --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/remote_cfr.md @@ -0,0 +1,82 @@ +# Remote CFR Messages +Starting in Firefox 68, CFR messages will be defined using [Remote Settings](https://remote-settings.readthedocs.io/en/latest/index.html). In this document, we'll cover how to set up a dev environment. + +## Using a dev server for Remote CFR + +> Note: Since Novembre 2021, Remote Settings has a proper DEV instance, which is +> reachable without VPN, but has the same config (openid, multi-signoff, ...) +> and collections as STAGE/PROD. + +**1. Obtain your Bearer Token** + +Until [Bug 1630651](https://bugzilla.mozilla.org/show_bug.cgi?id=1630651) happens, the easiest way to obtain your OpenID credentials is to use the admin interface. + +1. [Login on the Admin UI](https://remote-settings-dev.allizom.org/v1/admin/) using your LDAP identity +2. Copy the authentication header (📋 icon in the top bar) +3. Test your credentials with ``curl``. When reaching out the server root URL with this bearer token you should see a ``user`` entry whose ``id`` field is ``ldap:<you>@mozilla.com``. + +```bash +SERVER=https://settings.dev.mozaws.net/v1 +BEARER_TOKEN="Bearer uLdb-Yafefe....2Hyl5_w" + +curl -s ${SERVER}/ -H "Authorization:${BEARER_TOKEN}" | jq .user +``` + +**2. Create/Update/Delete CFR entries** + +> The messages can also be created manually using the [admin interface](https://settings.dev.mozaws.net/v1/admin/). + +In following example, we will create a new entry using the REST API (reusing `SERVER` and `BEARER_TOKEN` from previous step). + +```bash +CID=cfr + +# post a message +curl -X POST ${SERVER}/buckets/main-workspace/collections/${CID}/records \ + -d '{"data":{"id":"PIN_TAB","template":"cfr_doorhanger","content":{"category":"cfrFeatures","bucket_id":"CFR_PIN_TAB","notification_text":{"string_id":"cfr-doorhanger-extension-notification"},"heading_text":{"string_id":"cfr-doorhanger-pintab-heading"},"info_icon":{"label":{"string_id":"cfr-doorhanger-extension-sumo-link"},"sumo_path":"extensionrecommendations"},"text":{"string_id":"cfr-doorhanger-pintab-description"},"descriptionDetails":{"steps":[{"string_id":"cfr-doorhanger-pintab-step1"},{"string_id":"cfr-doorhanger-pintab-step2"},{"string_id":"cfr-doorhanger-pintab-step3"}]},"buttons":{"primary":{"label":{"string_id":"cfr-doorhanger-pintab-ok-button"},"action":{"type":"PIN_CURRENT_TAB"}},"secondary":[{"label":{"string_id":"cfr-doorhanger-extension-cancel-button"},"action":{"type":"CANCEL"}},{"label":{"string_id":"cfr-doorhanger-extension-never-show-recommendation"}},{"label":{"string_id":"cfr-doorhanger-extension-manage-settings-button"},"action":{"type":"OPEN_PREFERENCES_PAGE","data":{"category":"general-cfrfeatures"}}}]}},"targeting":"locale == \"en-US\" && !hasPinnedTabs && recentVisits[.timestamp > (currentDate|date - 3600 * 1000 * 1)]|length >= 3","frequency":{"lifetime":3},"trigger":{"id":"frequentVisits","params":["docs.google.com","www.docs.google.com","calendar.google.com","messenger.com","www.messenger.com","web.whatsapp.com","mail.google.com","outlook.live.com","facebook.com","www.facebook.com","twitter.com","www.twitter.com","reddit.com","www.reddit.com","github.com","www.github.com","youtube.com","www.youtube.com","feedly.com","www.feedly.com","drive.google.com","amazon.com","www.amazon.com","messages.android.com"]}}}' \ + -H 'Content-Type:application/json' \ + -H "Authorization:${BEARER_TOKEN}" +``` + +The collection was modified and now with pending changes in the workspace. We will now request a review, so that the changes become visible in the **preview** bucket. + +```bash +# request review +curl -X PATCH ${SERVER}/buckets/main-workspace/collections/${CID} \ + -H 'Content-Type:application/json' \ + -d '{"data": {"status": "to-review"}}' \ + -H "Authorization:${BEARER_TOKEN}" +``` + +Now this new record should be listed here: https://settings.dev.mozaws.net/v1/buckets/main-preview/collections/cfr/records + +**3. Set Remote Settings prefs to use the dev server.** + +Until [support for the DEV environment](https://github.com/mozilla-extensions/remote-settings-devtools/issues/66) is added to the [Remote Settings dev tools](https://github.com/mozilla-extensions/remote-settings-devtools/), we'll change the preferences manually. + +> These are critical preferences, you should use a dedicated Firefox profile for development. + +```javascript + Services.prefs.setCharPref("services.settings.loglevel", "debug"); + Services.prefs.setCharPref("services.settings.server", "https://settings.dev.mozaws.net/v1"); + // Pull data from the preview bucket. + RemoteSettings.enablePreviewMode(true); +``` + +**3. Set ASRouter CFR pref to use Remote Settings provider and enable asrouter devtools.** + +```javascript +Services.prefs.setStringPref("browser.newtabpage.activity-stream.asrouter.providers.cfr", JSON.stringify({"id":"cfr-remote","enabled":true,"type":"remote-settings","collection":"cfr"})); +Services.prefs.setBoolPref("browser.newtabpage.activity-stream.asrouter.devtoolsEnabled", true); +``` + +**4. Go to `about:newtab#devtools`** +There should be a "cfr-remote" provider listed. + +## Using the staging server for Remote CFR + +If your message is published in the staging environment the easiest way to test is using the [Remote Settings Devtools](https://github.com/mozilla/remote-settings-devtools/releases) addon. You can install this by going to `about:debugging` and using the `Load Temporary Addon` feature. +The devtools allow you to switch your profile between production and staging and takes care of correctly flipping all the required preferences. + +## Remote l10n +By default, all CFR messages are localized with the remote Fluent files hosted in `ms-language-packs` on Remote Settings. For local test and development, you can force ASRouter to use the local Fluent files by flipping the pref `browser.newtabpage.activity-stream.asrouter.useRemoteL10n`. diff --git a/browser/components/newtab/docs/v2-system-addon/sections.md b/browser/components/newtab/docs/v2-system-addon/sections.md new file mode 100644 index 0000000000..fae2a6508f --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/sections.md @@ -0,0 +1,82 @@ +# Sections + +Each section in Activity Stream displays data from a corresponding section feed +in a standardised `Section` UI component. Each section feed is responsible for +listening to events and updating the section options such as the title, icon, +and rows (the cards for the section to display). + +The `Section` UI component displays the rows provided by the section feed. If no +rows are available it displays an empty state consisting of an icon and a +message. Optionally, the section may have a info option menu that is displayed +when users hover over the info icon. + +On load, `SectionsManager` and `SectionsFeed` in `SectionsManager.jsm` add the +sections configured in the `BUILT_IN_SECTIONS` map to the state. These sections +are initially disabled, so aren't visible. The section's feed may use the +methods provided by the `SectionsManager` to enable its section and update its +properties. + +The section configuration in `BUILT_IN_SECTIONS` consists of a generator +function keyed by the pref name for the section feed. The generator function +takes an `options` argument as the only parameter, which is passed the object +stored as serialised JSON in the pref `{feed_pref_name}.options`, or the empty +object if this doesn't exist. The generator returns a section configuration +object which may have the following properties: + +```{eval-rst} ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| Property | Type | Description | ++====================+=====================+===================================================================================================================+ +| id | String | Non-optional unique id. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| title | Localisation object | Has property `id`, the string localisation id, and optionally a `values` object to fill in placeholders. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| icon | String | Icon id. New icons should be added in icons.scss. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| maxRows | Integer | Maximum number of rows of cards to display. Should be >= 1. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| contextMenuOptions | Array of strings | The menu options to provide in the card context menus. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| shouldHidePref | Boolean | If true, will the section preference in the preferences pane will not be shown. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| pref | Object | Configures the section preference to show in the preferences pane. Has properties `titleString` and `descString`. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +| emptyState | Object | Configures the empty state of the section. Has properties `message` and `icon`. | ++--------------------+---------------------+-------------------------------------------------------------------------------------------------------------------+ +``` + +## Section feeds + +Each section feed should be controlled by the pref `feeds.section.{section_id}`. + +### Enabling the section + +The section feed must listen for the events `INIT` (dispatched when Activity +Stream is initialised) and `FEED_INIT` (dispatched when a feed is re-enabled +having been turned off, with the feed id as the `data`). On these events it must +call `SectionsManager.enableSection(id)`. Care should be taken that this happens +only once `SectionsManager` has also initialised; the feed can use the method +`SectionsManager.onceInitialized()`. + +### Disabling the section + +The section feed must have an `uninit` method. This is called when the section +feed is disabled by turning the section's pref off. In `uninit` the feed must +call `SectionsManager.disableSection(id)`. This will remove the section's UI +component from every existing Activity Stream page. + +### Updating the section rows + +The section feed can call `SectionsManager.updateSection(id, options)` to update +section options. The `rows` array property of `options` stores the cards of +sites to display. Each card object may have the following properties: + +```js +{ + type, // One of the types in Card/types.js, e.g. "Trending" + title, // Title string + description, // Description string + image, // Image url + url // Site url +} +``` diff --git a/browser/components/newtab/docs/v2-system-addon/telemetry.md b/browser/components/newtab/docs/v2-system-addon/telemetry.md new file mode 100644 index 0000000000..4d76d0e615 --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/telemetry.md @@ -0,0 +1,13 @@ +# Telemetry checklist + +Adding telemetry generally involves a few steps: + +1. File a "user story" bug about who wants what question answered. This will be used to track the client-side implementation as well as the data review request. If the server side changes are needed, ask Nan (:nanj / @ncloudio) if in doubt, bugs will be filed separately as dependencies. +1. Implement as usual... +1. Update `system-addon/test/schemas/pings.js` with a commented JOI schema of your changes, and add tests to system-addon/test/unit/TelemetryFeed.test.js to exercise the ping creation. +1. Update [data_events.md](data_events.md) with an example of the data in question. +1. Update any fields that you've added, deleted, or changed in [data_dictionary.md](data_dictionary.md). +1. Get review from Nan on the data schema and the documentation changes. +1. Request `data-review` of your documentation changes from a [data steward](https://wiki.mozilla.org/Firefox/Data_Collection) to ensure suitability for collection controlled by the opt-out `datareporting.healthreport.uploadEnabled` pref. Download and fill out the [data review request form](https://github.com/mozilla/data-review/blob/master/request.md) and then attach it as a text file on Bugzilla so you can r? a data steward. We've been working with Chris H-C (:chutten) for the Firefox specific telemetry, and Kenny Long (kenny@getpocket.com) for the Pocket specific telemetry, they are the best candidates for the review work as they know well about the context. +1. After landing the implementation, check with Nan to make sure the pings are making it to the database. +1. Once data flows in, you can build dashboard for the new telemetry on [Redash](https://sql.telemetry.mozilla.org/dashboards). If you're looking for some help about Redash or dashboard building, Nan is the guy for that. diff --git a/browser/components/newtab/docs/v2-system-addon/tippytop.md b/browser/components/newtab/docs/v2-system-addon/tippytop.md new file mode 100644 index 0000000000..37111135c7 --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/tippytop.md @@ -0,0 +1,40 @@ +# TippyTop in Activity Stream +TippyTop, a collection of icons from the Alexa top sites, provides high quality images for the Top Sites in Activity Stream. The TippyTop manifest is hosted on S3, and then moved to [Remote Settings](https://remote-settings.readthedocs.io/en/latest/index.html) since Firefox 63. In this document, we'll cover how we produce and manage TippyTop manifest for Activity Stream. + +## TippyTop manifest production +TippyTop manifest is produced by [tippy-top-sites](https://github.com/mozilla/tippy-top-sites). + +```sh +# set up the environment, only needed for the first time +$ pip install -r requirements.txt +$ python make_manifest.py --count 2000 > icons.json # Alexa top 2000 sites +``` + +Because the manifest is hosted remotely, we use another repo [tippytop-service](https://github.com/mozilla-services/tippytop-service) for the version control and deployment. Ask :nanj or :r1cky for permission to access this private repo. + +## TippyTop manifest publishing +For each new manifest release, firstly you should tag it in the tippytop-service repo, then publish it as follows: + +### For Firefox 62 and below +File a deploy bug with the tagged version at Bugzilla as [Activity Streams: Application Servers](https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&component=Activity%20Streams%3A%20Application%20Servers), assign it to our system engineer :jbuck, he will take care of the rest. + +### For Firefox 63 and beyond +Activity Stream started using Remote Settings to manage TippyTop manifest since Firefox 63. To be able to publish new manifest, you need to be in the author&reviewer group of Remote Settings. See more details in this [mana page](https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=66655528). You can also ask :nanj or :leplatram to get this set up for you. +To publish the manifest to Remote Settings, go to the tippytop-service repo, and run the script as follows, + +```sh +# set up the remote setting, only needed for the first time +$ python3 -m venv .venv +$ source .venv/bin/activate +$ pip install -r requirements.txt + +# publish it to prod +$ source .venv/bin/activate +# It will ask you for your LDAP user name and password. +$ ./upload2remotesettings.py prod +``` + +After uploading it to Remote Setting, you can request for review in the [dashboard](https://remote-settings.mozilla.org/v1/admin/). Note that you will need to log in the Mozilla LDAP VPN for both uploading and accessing Remote Setting's dashboard. Once your request gets approved by the reviewer, the new manifest will be content signed and published to production. + +## TippyTop Viewer +You can use this [viewer](https://mozilla.github.io/tippy-top-sites/manifest-viewer/) to load all the icons in the current manifest. diff --git a/browser/components/newtab/docs/v2-system-addon/unit_testing_guide.md b/browser/components/newtab/docs/v2-system-addon/unit_testing_guide.md new file mode 100644 index 0000000000..c3dd369a2d --- /dev/null +++ b/browser/components/newtab/docs/v2-system-addon/unit_testing_guide.md @@ -0,0 +1,149 @@ +# Unit testing + +## Overview + +Our unit tests in Activity Stream are written with mocha, chai, and sinon, and run +with karma. They include unit tests for both content code (React components, etc.) +and `.jsm`s. + +You can find unit tests in `tests/unit`. + +## Execution + +To run the unit tests once, execute `npm test`. + +To run unit tests continuously (i.e. in "test-driven development" mode), you can +run `npm run tddmc`. + +## Debugging + +To debug tests, you should run them in continuous mode with `npm run tddmc`. +In the Firefox window that is opened (it should say "Karma... - connected"), +click the "debug" button and open your console to see test output, set +breakpoints, etc. + +Unfortunately, source maps for tests do not currently work in Firefox. If you need +to see line numbers, you can run the tests with Chrome by running +`npm install --save-dev karma-chrome-launcher && npm run tddmc -- --browsers Chrome` + +## Where to put new tests + +If you are creating a new test, add it to a subdirectory of the `tests/unit` +that corresponds to the file you are testing. Tests should end with `.test.js` or +`.test.jsx` if the test includes any jsx. + +For example, if the file you are testing is `lib/Foo.jsm`, the test +file should be `test/unit/lib/Foo.test.js` + +## Mocha tests + +All our unit tests are written with [mocha](https://mochajs.org), which injects +globals like `describe`, `it`, `beforeEach`, and others. It can be used to write +synchronous or asynchronous tests: + +```js +describe("FooModule", () => { + // A synchronous test + it("should create an instance", () => { + assert.instanceOf(new FooModule(), FooModule); + }); + describe("#meaningOfLife", () => { + // An asynchronous test + it("should eventually get the meaning of life", async () => { + const foo = new FooModule(); + const result = await foo.meaningOfLife(); + assert.equal(result, 42); + }); + }); +}); +``` + +## Assertions + +To write assertions, use the globally available `assert` object (this is provided +by karma-chai, so you do not need to `require` it). + +For example: + +```js +assert.equal(foo, 3); +assert.propertyVal(someObj, "foo", 3); +assert.calledOnce(someStub); +``` + +You can use any of the assertions from: + +- [`chai`](http://chaijs.com/api/assert/). +- [`sinon-chai`](https://github.com/domenic/sinon-chai#assertions) + +### Custom assertions + +We have some custom assertions for checking various types of actions: + +#### `.isUserEventAction(action)` + +Asserts that a given `action` is a valid User Event, i.e. that it contains only +expected/valid properties for User Events in Activity Stream. + +```js +// This will pass +assert.isUserEventAction(ac.UserEvent({event: "CLICK"})); + +// This will fail +assert.isUserEventAction({type: "FOO"}); + +// This will fail because BLOOP is not a valid event type +assert.isUserEventAction(ac.UserEvent({event: "BLOOP"})); +``` + +## Overriding globals in `.jsm`s + +Most `.jsm`s you will be testing use `Cu.import` or `XPCOMUtils` to inject globals. +In order to add mocks/stubs/fakes for these globals, you should use the `GlobalOverrider` +utility in `test/unit/utils`: + +```js +const {GlobalOverrider} = require("test/unit/utils"); +describe("MyModule", () => { + let globals; + let sandbox; + beforeEach(() => { + globals = new GlobalOverrider(); + sandbox = globals.sandbox; // this is a sinon sandbox + // This will inject a "AboutNewTab" global before each test + globals.set("AboutNewTab", {override: sandbox.stub()}); + }); + // globals.restore() clears any globals you added as well as the sinon sandbox + afterEach(() => globals.restore()); +}); +``` + +## Testing React components + +You should use the [enzyme](https://github.com/airbnb/enzyme) suite of test utilities +to test React Components for Activity Stream. + +Where possible, use the [shallow rendering method](https://github.com/airbnb/enzyme/blob/master/docs/api/shallow.md) (this will avoid unnecessarily +rendering child components): + +```js +const React = require("react"); +const {shallow} = require("enzyme"); + +describe("<Foo>", () => { + it("should be hidden by default", () => { + const wrapper = shallow(<Foo />); + assert.isTrue(wrapper.find(".wrapper").props().hidden); + }); +}); +``` + +If you need to, you can also do [Full DOM rendering](https://github.com/airbnb/enzyme/blob/master/docs/api/mount.md) +with enzyme's `mount` utility. + +```js +const React = require("react"); +const {mount} = require("enzyme"); +... +const wrapper = mount(<Foo />); +``` |