diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 19:33:14 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 19:33:14 +0000 |
| commit | 36d22d82aa202bb199967e9512281e9a53db42c9 (patch) | |
| tree | 105e8c98ddea1c1e4784a60a5a6410fa416be2de /toolkit/components/telemetry/docs/obsolete | |
| parent | Initial commit. (diff) | |
| download | firefox-esr-36d22d82aa202bb199967e9512281e9a53db42c9.tar.xz firefox-esr-36d22d82aa202bb199967e9512281e9a53db42c9.zip | |
Adding upstream version 115.7.0esr.upstream/115.7.0esr
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'toolkit/components/telemetry/docs/obsolete')
13 files changed, 3649 insertions, 0 deletions
diff --git a/toolkit/components/telemetry/docs/obsolete/activation-ping.rst b/toolkit/components/telemetry/docs/obsolete/activation-ping.rst new file mode 100644 index 0000000000..a2c241aec4 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/activation-ping.rst @@ -0,0 +1,69 @@ + +"activation" ping (obsolete) +============================ + +This mobile-specific ping is intendend to track activations of distributions of mobile products +with a small error rate. + +This will be sent at Startup. Minimally, we want to get this ping at least once from every client. + +Submission will be per the Edge server specification:: + + /submit/mobile/activation/v/docId + +This is a modern “structure ingestion” ping (the namespace is not telemetry). For structured +ingestion, we capture the schema version as one of the URI parameters, so we don’t need to +include it in the body of the message. + +* ``v`` is the ping format version +* ``docId`` is a UUID for deduping + +Structure: + +.. code-block:: js + + { + "identifier": <string>, // Googled Ad ID hashed using bcrypt + "clientId": <string>, // client id, e.g. "c641eacf-c30c-4171-b403-f077724e848a" + //included only if identifier was unabled to be retrieved + "manufacturer": <string>, // Build.MANUFACTURER + "model": <string>, // Build.MODEL + "locale": <string>, // application locale, e.g. "en-US" + "os": <string>, // OS name. + "osversion": <string>, // OS version. + "created": <string>, // date the ping was created + // in local time, "yyyy-mm-dd" + "tz": <integer>, // timezone offset (in minutes) of the + // device when the ping was created + "app_name": <string>, // "Fennec" + "channel": <string>, // Android package name e.g. "org.mozilla.firefox" + "distributionId": <string> // Distribution identifier (optional) + } + + +Field details +------------- + +identifier +~~~~~~~~~~ +The ``identifier`` field is the Google Ad ID hashed using bcrypt. Ideally we want to send this instead of the +client_id but not all distributions have Google Play Services enabled. + +client_id +~~~~~~~~~~ +The ``client_id`` field represents the telemetry client id and it is only included if the identifier is empty. + +channel +~~~~~~~ +The ``channel`` field represents the Android package name. + +Version history +--------------- +* v1: initial version - shipped in `Fennec 68 <https://bugzilla.mozilla.org/show_bug.cgi?id=1534451>`_. + +Android implementation notes +---------------------------- +On Android, the uploader has a high probability of delivering the complete data +for a given client but not a 100% probability. This was a conscious decision to +keep the code simple. Even if we drop a ping, it will be resent on future startups +until we have confirmation that it has been uploaded. diff --git a/toolkit/components/telemetry/docs/obsolete/core-ping.rst b/toolkit/components/telemetry/docs/obsolete/core-ping.rst new file mode 100644 index 0000000000..6fee237805 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/core-ping.rst @@ -0,0 +1,510 @@ + +"core" ping (obsolete) +====================== + +This mobile-specific ping is intended to provide the most critical +data in a concise format, allowing for frequent uploads. + +Since this ping is used to measure retention, it should be sent +each time the browser is opened. + +Submission will be per the Edge server specification:: + + /submit/telemetry/docId/docType/appName/appVersion/appUpdateChannel/appBuildID + +* ``docId`` is a UUID for deduping +* ``docType`` is “core” +* ``appName`` is “Fennec” +* ``appVersion`` is the version of the application (e.g. "46.0a1") +* ``appUpdateChannel`` is “release”, “beta”, etc. +* ``appBuildID`` is the build number + +Note: Counts below (e.g. search & usage times) are “since the last +ping”, not total for the whole application lifetime. + +Structure: + +.. code-block:: + + { + "v": 10, // ping format version + "clientId": <string>, // client id, e.g. + // "c641eacf-c30c-4171-b403-f077724e848a" + "seq": <positive integer>, // running ping counter, e.g. 3 + "locale": <string>, // application locale, e.g. "en-US" + "os": <string>, // OS name. + "osversion": <string>, // OS version. + "device": <string>, // Build.MANUFACTURER + " - " + Build.MODEL + // where manufacturer is truncated to 12 characters + // & model is truncated to 19 characters + "arch": <string>, // e.g. "arm", "x86" + "profileDate": <pos integer>, // Profile creation date in days since + // UNIX epoch. + "defaultSearch": <string>, // Identifier of the default search engine, + // e.g. "yahoo". + "displayVersion": <string>, // Version displayed to user, e.g. 57.0b3 (optional) + "distributionId": <string>, // Distribution identifier (optional) + "campaignId": <string>, // Adjust's campaign identifier (optional) + "created": <string>, // date the ping was created + // in local time, "yyyy-mm-dd" + "tz": <integer>, // timezone offset (in minutes) of the + // device when the ping was created + "sessions": <integer>, // number of sessions since last upload + "durations": <integer>, // combined duration, in seconds, of all + // sessions since last upload + "searches": <object>, // Optional, object of search use counts in the + // format: { "engine.source": <pos integer> } + // e.g.: { "yahoo.suggestion": 3, "other.listitem": 1 } + "experiments": [<string>, /* … */], // Optional, array of identifiers + // for the active experiments + "accessibilityServices": [<string>, /* … */], // Optional, array of identifiers for + // enabled accessibility services that + // interact with our android products. + "flashUsage": <integer>, // number of times flash plugin is played since last upload + "defaultBrowser": <boolean> // true if the user has set Firefox as default browser + "bug_1501329_affected": <boolean> // true if Firefox previously used canary clientId + // when submitting telemetry + "fennec": <object> // Fennec only. + // Block of a variety of fields of different types. + // Used to understand the usage of Fennec features in the release population + // to understand when Fenix is ready to support Fennec users. + { + "new_tab": { + "top_sites_clicked": <int>, // Number of times a Top Site was opened from the Awesome Screen. + // Resets after each sent core ping. + "pocket_stories_clicked": <int>, // Number of time a Pocket Recommended website was opened + // from the Awesome Screen. + // Resets after each sent core ping. + } + "settings_advanced": { + "restore_tabs": <boolean>, // State of the "Settings/Advanced/Restore Tabs" setting + "show_images": <string>, // State of the "Settings/Advanced/Show images" setting + // Value will be be "user-specified" for any non-default values + "show_web_fonts": <boolean>, // State of the "Settings/Advanced/Show web fonts" setting + }, + "settings_general": { + "full_screen_browsing": <boolean>, // Current state of the + // "Settings/General/Full-screen browsing" setting. + "tab_queue": <boolean>, // State of the "Settings/General/Tab queue" setting. + "tab_queue_usage_count": <int>, // Number of tabs opened through Tab Queue. + // Resets after each sent core ping. + "compact_tabs": <boolean>, // State of the "Settings/General/Compact tabs" setting. + "homepage": { + "custom_homepage": <boolean>, // "true" if not "about:home". + "custom_homepage_use_for_newtab": <boolean>, // If the "Settings/General/Home/Also use for new tabs" + // setting is enabled. + "topsites_enabled": <boolean>, // If the "Settings/General/Home/Top Sites" + // setting is set to "Show". + "pocket_enabled": <boolean>, // If the "Settings/General/Home/Top Sites/Recommended by Pocket" + // setting is enabled. + "recent_bookmarks_enabled": <boolean>, // If the "Settings/General/Home/Top Sites/ + // Additional Content/Recent Bookmarks" + // setting is enabled. + "visited_enabled": <boolean>, // If the "Settings/General/Home/Top Sites/Additional Content/Visited" + // setting is enabled. + bookmarks_enabled": <boolean>, // If the "Settings/General/Home/Bookmarks" setting is set to "Show". + "history_enabled": <boolean>, // If the "Settings/General/Home/History" setting is set to "Show". + } + }, + "settings_privacy": { + "do_not_track": <boolean>, // If the "Settings/Privacy/Do not track" is enabled. + "master_password": <boolean>, // If the "Settings/Privacy/Use master password" is enabled. + "master_password_usage_count": <int>, // Number of times the user has entered their master password. + // Resets after each sent core ping. + }, + "settings_notifications": { + "product_feature_tips": <boolean>, // If the "Settings/Notifications/Product and feature tips" + // setting is enabled. + }, + "addons": { + "active": [addon_id_1, addon_id_2, …, ], // From all installed addons, which ones are active. + "disabled": [addon_id_1, addon_id_2, …], // From all installed addons, which ones are disabled. + }, + "page_options": { + "save_as_pdf": <int>, // Number of times the user has used "Page/Save to PDF". + // Resets after each sent core ping. + "print": <int>, // Number of times the user has used the "Page/Print". + // Resets after each sent core ping. + "total_added_search_engines": <int>, // The absolute number of user added search engines, + // not just those added during this session. + "total_sites_pinned_to_topsites": <int>, // The absolute number of times the user has used + // the "Pin Site" functionality. + // Not just those added during this session. + "view_source": <int>, // Number of times the user has used the "Page/View Page Source". + // Resets after each sent core ping. + "bookmark_with_star": <int>, // The number of times the user has used the "Menu / <Star>". + // Resets after each sent core ping. + "current_pwas_count": <int>, // On Android >=25 - a positive number of PWAs currently on + // homescreen, installed from this app. + // On Android <25 - a default of "-1". + }, + "sync": { + "only_over_wifi": <boolean>, // "true" if the "Settings/Sync/Sync only over Wi-Fi" + // setting is enabled. + // null if the user is not signed into Sync. + } + } + } + +Field details +------------- + +device +~~~~~~ +The ``device`` field is filled in with information specified by the hardware +manufacturer. As such, it could be excessively long and use excessive amounts +of limited user data. To avoid this, we limit the length of the field. We're +more likely have collisions for models within a manufacturer (e.g. "Galaxy S5" +vs. "Galaxy Note") than we are for shortened manufacturer names so we provide +more characters for the model than the manufacturer. + +distributionId +~~~~~~~~~~~~~~ +The ``distributionId`` contains the distribution ID as specified by +preferences.json for a given distribution. More information on distributions +can be found `here <https://wiki.mozilla.org/Mobile/Distribution_Files>`_. + +It is optional. + +campaignId +~~~~~~~~~~~~~~ +The ``campaignId`` contains the campaign identifier like '3ly8t0'. +It's generated by `Adjust <https://docs.adjust.com/en/tracker-generation/#segmenting-users-dynamically-with-campaign-structure-parameters>`_, +It can only used to identify a campaign, but can't target to a specific user. + +It is optional because not everyone has a campaign to begin with. + +defaultSearch +~~~~~~~~~~~~~ +On Android, this field may be ``null``. To get the engine, we rely on +``SearchEngineManager#getDefaultEngine``, which searches in several places in +order to find the search engine identifier: + +* Shared Preferences +* The distribution (if it exists) +* The localized default engine + +If the identifier could not be retrieved, this field is ``null``. If the +identifier is retrieved, we attempt to create an instance of the search +engine from the search plugins (in order): + +* In the distribution +* From the localized plugins shipped with the browser +* The third-party plugins that are installed in the profile directory + +If the plugins fail to create a search engine instance, this field is also +``null``. + +This field can also be ``null`` when a custom search engine is set as the +default. + +sessions & durations +~~~~~~~~~~~~~~~~~~~~ +On Android, a session is the time when Firefox is focused in the foreground. +`sessions` tracks the number of sessions since the last upload and +`durations` is the accumulated duration in seconds of all of these +sessions. Note that showing a dialog (including a Firefox dialog) will +take Firefox out of focus & end the current session. + +An implementation that records a session when Firefox is completely hidden is +preferable (e.g. to avoid the dialog issue above), however, it's more complex +to implement and so we chose not to, at least for the initial implementation. + +profileDate +~~~~~~~~~~~ +On Android, this value is created at profile creation time and retrieved or, +for legacy profiles, taken from the package install time (note: this is not the +same exact metric as profile creation time but we compromised in favor of ease +of implementation). + +Additionally on Android, this field may be ``null`` in the unlikely event that +all of the following events occur: + +#. The times.json file does not exist +#. The package install date could not be persisted to disk + +The reason we don't just return the package install time even if the date could +not be persisted to disk is to ensure the value doesn't change once we start +sending it: we only want to send consistent values. + +searches +~~~~~~~~ +This describes the search engine usage(count). The format is { "<engine identifier>.<source>"" : count } +This is optional because the users may have never used the search feature. +There's no difference if extended telemetry is enabled (prerelease builds) or not. + +Possible value : + +.. code-block:: js + + { + "yahoo.listitem":2, + "duckduckgo.listitem":1, + "google.suggestion":1 + } + +**<engine identifier>**: the identifier of the the search engine. The identifier is collected the way same as desktop. +we only record the search engine name when: + +* builtin or suggested search engines with an ID (includes partner search engines in various distribution scenarios). + If it's not a built-in engine, we show "null" or "other". +* If the user has "Health Report" and core ping enabled. + +**<sources>**: it's from one of the 'method's in UI telemetry. Possible values: + +* actionbar: the user types in the url bar and hits enter to use the default + search engine +* listitem: the user selects a search engine from the list of secondary search + engines at the bottom of the screen +* suggestion: the user clicks on a search suggestion or, in the case that + suggestions are disabled, the row corresponding with the main engine + +accessibilityServices +~~~~~~~~~~~~~~~~~~~~~ +This describes which accessibility services are currently enabled on user's device and could be interacting with our +products. This is optional because users often do not have any accessibility services enabled. If present, the value is +a list of accessibility service ids. + +fennec.new_tab.top_sites_clicked +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `top_sites_clicked` field contains the number of times a top site was +opened from the new tab page since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + + +Fennec.new_tab.pocket_stories_clicked +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `pocket_stories_clicked` contains the number of times a pocket story was +opened from the new tab page since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + +Fennec.settings_advanced.restore_tabs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `restore_tabs` field contains state of the "Settings/Advanced/Restore Tabs" +setting. It is true for "Always Restore" and false for "Don’t restore after +quitting Firefox". +The value is determined at the time of sending the core ping. + +Fennec.settings_advanced.show_images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `show_images` field contains the state of the +"Settings/Advanced/Show images" settings. +It is a string value set to "default" if the setting is "Always", or +"user~specified" for any of the other options. +The value is determined at the time of sending the core ping. + +Fennec.settings_advanced.show_web_fonts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `show_web_fonts` field is a boolean that contains the current state of the +"Settings/Advanced/Show web fonts" setting. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.full_screen_browsing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `full_screen_browsing` field is a boolean that contains the current state +of the "Settings/General/Full~screen browsing" setting. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.tab_queue +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `tab_queue` field is a boolean that contains the current state of the +"Settings/General/Tab queue" setting. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.tab_queue_usage_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `tab_queue_usage_count` is a counter that increments with the number of +tabs opened through the tab queue. +It contains the total number of queued tabs opened since the last time the +Core Ping was sent. +This counter is reset when the core ping has been sent. + +Fennec.settings_general.compact_tabs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `compact_tabs` field is a boolean that contains the current state of the +"Settings/General/Compact tabs" setting. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.custom_homepage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `custom_homepage` field is set to true if the homepage is not set to the +the default `about:home`. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.custom_homepage_use_for_newtab +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `custom_homepage_use_for_newtab` field is set to true if the +"Settings/General/Home/Also use for new tabs" setting is enabled. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.topsites_enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `topsites_enabled` setting is true if the "Settings/General/Home/Top Sites" +setting is set to "Show". +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.pocket_enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `pocket_enabled` setting is true if the +"Settings/General/Home/Top Sites/Recommended by Pocket" setting is enabled. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.recent_bookmarks_enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `recent_bookmarks_enabled` setting is true if the +"Settings/General/Home/Top Sites/Additional Content/Recent Bookmarks" setting +is enabled. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.visited_enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `visited_enabled` setting is true if the +"Settings/General/Home/Top Sites/Additional Content/Visited" setting is +enabled. +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.bookmarks_enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `bookmarks_enabled` setting is true if the +"Settings/General/Home/Bookmarks" setting is set to "Show". +The value is determined at the time of sending the core ping. + +Fennec.settings_general.homepage.history_enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `history_enabled` setting is true if the "Settings/General/Home/History" +setting is set to "Show". +The value is determined at the time of sending the core ping. + +Fennec.settings_privacy.do_not_track +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `do_not_track` setting is true if the "Settings/Privacy/Do not track" is +enabled. +The value is determined at the time of sending the core ping. + +Fennec.settings_privacy.master_password +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `master_password` setting is true if the +"Settings/Privacy/Use master password" is enabled. +The value is determined at the time of sending the core ping. + +Fennec.settings_privacy.master_password_usage_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `master_password_usage_count` field contains the number of times the user +has entered their master password since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + +Fennec.settings_notifications.product_feature_tips +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `product_feature_tips` setting is true if the +"Settings/Notifications/Product and feature tips" setting is enabled. +The value is determined at the time of sending the core ping. + +fennec.page_options.save_as_pdf +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `save_as_pdf` field contains the number of times the user has used the +"Page/Save to PDF" feature since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + +fennec.page_options.print +~~~~~~~~~~~~~~~~~~~~~~~~~ +The `print` field contains the number of times the user has used the +"Page/Print" feature since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + +fennec.page_options.total_added_search_engines +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `total_added_search_engines` is an absolute value that contains the number +of search engines the user has added manually. +The value is determined at the time of sending the core ping and never reset +to zero. + +fennec.page_options.total_sites_pinned_to_topsites +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `total_sites_pinned_to_topsites` is an absolute value that contains the +number of sites the user has pinned to top sites. +The value is determined at the time of sending the core ping and never reset +to zero. + +fennec.page_options.view_source +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `view_source` field contains the number of times the user has used the +"Page/View Page Source" feature since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + +Fennec.page_options.bookmark_with_star +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `bookmark_with_star` field contains the number of times the user has used +the "Menu / <Star>"" feature since the last time the core ping was sent. +This counter is reset when the core ping has been sent. + +Fennec.page_options.current_pwas_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `current_pwas_count` field contains the number of currently installed PWAs +from this application. +As Android APIs for querying this are only available on Android >=25 for lower +versions of Android the value of this key will be "-1". +The value is determined at the time of sending the core ping. + +Fennec.sync.only_over_wifi +~~~~~~~~~~~~~~~~~~~~~~~~~~ +The `only_over_wifi` setting is true if the +"Settings/Sync/Sync only over Wi~Fi" setting is enabled. +The value is determined at the time of sending the core ping. +If the user is not signed into sync, then this value is set to `null`. +The value is determined at the time of sending the core ping. + +Other parameters +---------------- + +HTTP "Date" header +~~~~~~~~~~~~~~~~~~ +This header is used to track the submission date of the core ping in the format +specified by +`rfc 2616 sec 14.18 <https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18>`_, +et al (e.g. "Tue, 01 Feb 2011 14:00:00 GMT"). + + +Version history +--------------- +* v10: added ``bug_1501329_affected`` +* v9: + + - Apr 2017: changed ``arch`` to contain device arch rather than the one we + built against & ``accessibilityServices`` + - Dec 2017: added ``defaultBrowser`` to know if the user has set Firefox as + default browser (Dec 2017) + - May 2018: added (optional) ``displayVersion`` to distinguish Firefox beta versions easily + +* v8: added ``flashUsage`` +* v7: added ``sessionCount`` & ``sessionDuration`` & ``campaignId`` +* v6: added ``searches`` +* v5: added ``created`` & ``tz`` +* v4: ``profileDate`` will return package install time when times.json is not available +* v3: added ``defaultSearch`` +* v2: added ``distributionId`` +* v1: initial version - shipped in `Fennec 45 <https://bugzilla.mozilla.org/show_bug.cgi?id=1205835>`_. + +Notes +~~~~~ + +* ``distributionId`` (v2) actually landed after ``profileDate`` (v4) but was + uplifted to 46, whereas ``profileDate`` landed on 47. The version numbers in + code were updated to be increasing (bug 1264492) and the version history docs + rearranged accordingly. + +Android implementation notes +---------------------------- +On Android, the uploader has a high probability of delivering the complete data +for a given client but not a 100% probability. This was a conscious decision to +keep the code simple. The cases where we can lose data: + +* Resetting the field measurements (including incrementing the sequence number) + and storing a ping for upload are not atomic. Android can kill our process + for memory pressure in between these distinct operations so we can just lose + a ping's worth of data. That sequence number will be missing on the server. +* If we exceed some number of pings on disk that have not yet been uploaded, + we remove old pings to save storage space. For those pings, we will lose + their data and their sequence numbers will be missing on the server. + +Note: we never expect to drop data without also dropping a sequence number so +we are able to determine when data loss occurs. diff --git a/toolkit/components/telemetry/docs/obsolete/deletion-ping.rst b/toolkit/components/telemetry/docs/obsolete/deletion-ping.rst new file mode 100644 index 0000000000..dfa7fefa00 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/deletion-ping.rst @@ -0,0 +1,26 @@ + +"deletion" ping (obsolete) +========================== + +This ping is generated when a user turns off FHR upload from the Preferences panel, changing the related ``datareporting.healthreport.uploadEnabled`` preference. This requests that all associated data from that user be deleted. + +This ping contains the client id and no environment data. + +Structure: + +.. code-block:: js + + { + version: 4, + type: "deletion", + ... common ping data + clientId: <UUID>, + payload: { } + } + +Version History +--------------- + +- Firefox 63: + + - Replaced by "optout" ping (`bug 1445921 <https://bugzilla.mozilla.org/show_bug.cgi?id=1445921>`_). diff --git a/toolkit/components/telemetry/docs/obsolete/ecosystem-telemetry.rst b/toolkit/components/telemetry/docs/obsolete/ecosystem-telemetry.rst new file mode 100644 index 0000000000..e9902ff89c --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/ecosystem-telemetry.rst @@ -0,0 +1,109 @@ +Ecosystem Telemetry (obsolete) +============================== + +This module transmits Ecosystem Telemetry from Firefox Desktop. +It is only sent for Firefox Account users, using a single ping type +"account-ecosystem" + +.. note:: + + You might like to read the `background information on Ecosystem + Telemetry <https://mozilla.github.io/ecosystem-platform/docs/features/firefox-accounts/ecosystem-telemetry/>`_ + +The existing telemetry client id is **not** submitted with the ping, but an +"ecosystem client id" is - this has the same semantics as the existing +client id, although is a different value, and is not sent in any other ping. + +An anonymized user ID is submitted with each ping - `read more about these +IDs and how they're designed to safeguard user privacy <https://mozilla.github.io/ecosystem-platform/docs/features/firefox-accounts/ecosystem-telemetry/>`_ + +A reduced Telemetry environment is submitted in the ping, as described below. + +Environment +----------- + +In an effort to reduce the possibility of fingerprinting, we only provide the +following environment subset: + +.. code-block:: js + + { + settings: { + locale: <string>, // e.g. "it", null on failure + }, + system: { + memoryMB: <number>, + os: { + name: <string>, // e.g. "Windows_NT", null on failure + version: <string>, // e.g. "6.1", null on failure + locale: <string>, // "en" or null on failure + }, + cpu: { + speedMHz: <number>, // cpu clock speed in MHz + } + }, + profile: { + creationDate: <integer>, // integer days since UNIX epoch, e.g. 16446 + firstUseDate: <integer>, // integer days since UNIX epoch, e.g. 16446 - optional + } + } + +account-ecosystem ping +---------------------- + +.. code-block:: js + + { + "type": "account-ecosystem", + ... common ping data + "environment": { ... }, // as above + "payload": { + "reason": <string>, // Why the ping was submitted + "ecosystemAnonId": <string>, // The anonymized ID, as described above. + "ecosystemClientId": <guid>, // The ecosystem client ID as described above. + "duration": <number>, // duration since ping was last sent or since the beginning of the Firefox session in seconds + "histograms": {...}, + "keyedHistograms": {...}, + "scalars": {...}, + "keyedScalars": {...}, + } + } + +reason +~~~~~~ +The ``reason`` field contains the information about why the "account-ecosystem" ping was submitted: + +* ``periodic`` - Sent roughly every 24 hours +* ``shutdown`` - Sent on shutdown +* ``logout`` - Sent when the user logs out + +histograms and keyedHistograms +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This section contains the :doc:`../collection/histograms` that are valid for the account-ecosystem ping, per process. +The recorded histograms are described in `Histograms.json <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json>`_, marked with the `account-ecosystem` store. + +scalars and keyedScalars +~~~~~~~~~~~~~~~~~~~~~~~~ +This section contains the :doc:`../collection/scalars` that are valid for the account-ecosystem ping, per process. +Scalars are only submitted if data was added to them. +The recorded scalars are described in `Scalars.yaml <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml>`_, marked with the `account-ecosystem` store. + +Send behavior +------------- + +Without an account +~~~~~~~~~~~~~~~~~~ + +Never. + +When a user logs into Firefox Accounts, this ping is submitted as described in +"With an account" below. No ping is immediately sent. + +With an account +~~~~~~~~~~~~~~~ + +The ping is submitted; roughly every 24 hours with reason *periodic*. On +shutdown this ping is submitted with reason *shutdown*. + +If the user logs out and disconnects the account, this ping is submitted with +reason *logout*. While logged out, no pings will be submitted. diff --git a/toolkit/components/telemetry/docs/obsolete/fhr/architecture.rst b/toolkit/components/telemetry/docs/obsolete/fhr/architecture.rst new file mode 100644 index 0000000000..2e9c37f3d3 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/fhr/architecture.rst @@ -0,0 +1,226 @@ +.. _healthreport_architecture: + +============ +Architecture +============ + +``healthreporter.jsm`` contains the main interface for FHR, the +``HealthReporter`` type. An instance of this is created by the +``data_reporting_service``. + +``providers.jsm`` contains numerous ``Metrics.Provider`` and +``Metrics.Measurement`` used for collecting application metrics. If you +are looking for the FHR probes, this is where they are. + +Storage +======= + +Firefox Health Report stores data in 3 locations: + +* Metrics measurements and provider state is stored in a SQLite database + (via ``Metrics.Storage``). +* Service state (such as the IDs of documents uploaded) is stored in a + JSON file on disk (via OS.File). +* Lesser state and run-time options are stored in preferences. + +Preferences +=========== + +Preferences controlling behavior of Firefox Health Report live in the +``datareporting.healthreport.*`` branch. + +Service and Data Control +------------------------ + +The follow preferences control behavior of the service and data upload. + +service.enabled + Controls whether the entire health report service runs. The overall + service performs data collection, storing, and submission. + + This is the primary kill switch for Firefox Health Report + outside of the build system variable. i.e. if you are using an + official Firefox build and wish to disable FHR, this is what you + should set to false to prevent FHR from not only submitting but + also collecting data. + +uploadEnabled + Whether uploading of data is enabled. This is the preference the + checkbox in the preferences UI reflects. If this is + disabled, FHR still collects data - it just doesn't upload it. + +service.loadDelayMsec + How long (in milliseconds) after initial application start should FHR + wait before initializing. + + FHR may initialize sooner than this if the FHR service is requested. + This will happen if e.g. the user goes to ``about:healthreport``. + +service.loadDelayFirstRunMsec + How long (in milliseconds) FHR should wait to initialize on first + application run. + + FHR waits longer than normal to initialize on first application run + because first-time initialization can use a lot of I/O to initialize + the SQLite database and this I/O should not interfere with the + first-run user experience. + +documentServerURI + The URI of a Bagheera server that FHR should interface with for + submitting documents. + + You typically do not need to change this. + +documentServerNamespace + The namespace on the document server FHR should upload documents to. + + You typically do not need to change this. + +infoURL + The URL of a page containing more info about FHR, it's privacy + policy, etc. + +about.reportUrl + The URL to load in ``about:healthreport``. + +about.reportUrlUnified + The URL to load in ``about:healthreport``. This is used instead of ``reportUrl`` for UnifiedTelemetry when it is not opt-in. + +service.providerCategories + A comma-delimited list of category manager categories that contain + registered ``Metrics.Provider`` records. Read below for how provider + registration works. + +If the entire service is disabled, you lose data collection. This means +that **local** data analysis won't be available because there is no data +to analyze! Keep in mind that Firefox Health Report can be useful even +if it's not submitting data to remote servers! + +Logging +------- + +The following preferences allow you to control the logging behavior of +Firefox Health Report. + +logging.consoleEnabled + Whether to write log messages to the web console. This is true by + default. + +logging.consoleLevel + The minimum log level FHR messages must have to be written to the + web console. By default, only FHR warnings or errors will be written + to the web console. During normal/expected operation, no messages of + this type should be produced. + +logging.dumpEnabled + Whether to write log messages via ``dump()``. If true, FHR will write + messages to stdout/stderr. + + This is typically only enabled when developing FHR. + +logging.dumpLevel + The minimum log level messages must have to be written via + ``dump()``. + +State +----- + +currentDaySubmissionFailureCount + How many submission failures the client has encountered while + attempting to upload the most recent document. + +lastDataSubmissionFailureTime + The time of the last failed document upload. + +lastDataSubmissionRequestedTime + The time of the last document upload attempt. + +lastDataSubmissionSuccessfulTime + The time of the last successful document upload. + +nextDataSubmissionTime + The time the next data submission is scheduled for. FHR will not + attempt to upload a new document before this time. + +pendingDeleteRemoteData + Whether the client currently has a pending request to delete remote + data. If true, the client will attempt to delete all remote data + before an upload is performed. + +FHR stores various state in preferences. + +Registering Providers +===================== + +Firefox Health Report providers are registered via the category manager. +See ``HealthReportComponents.manifest`` for providers defined in this +directory. + +Essentially, the category manager receives the name of a JS type and the +URI of a JSM to import that exports this symbol. At run-time, the +providers registered in the category manager are instantiated. + +Providers are registered via the category manager to make registration +simple and less prone to errors. Any XPCOM component can create a +category manager entry. Therefore, new data providers can be added +without having to touch core Firefox Health Report code. Additionally, +category manager registration means providers are more likely to be +registered on FHR's terms, when it wants. If providers were registered +in code at application run-time, there would be the risk of other +components prematurely instantiating FHR (causing a performance hit if +performed at an inopportune time) or semi-complicated code around +observers or listeners. Category manager entries are only 1 line per +provider and leave FHR in control: they are simple and safe. + +Document Generation and Lifecycle +================================= + +FHR will attempt to submit a JSON document containing data every 24 wall +clock hours. + +At upload time, FHR will query the database for **all** information from +the last 180 days and assemble this data into a JSON document. We +attempt to upload this JSON document with a client-generated UUID to the +configured server. + +Before we attempt upload, the generated UUID is stored in the JSON state +file on local disk. At this point, the client assumes the document with +that UUID has been successfully stored on the server. + +If the client is aware of other document UUIDs that presumably exist on +the server, those UUIDs are sent with the upload request so the client +can request those UUIDs be deleted. This helps ensure that each client +only has 1 document/UUID on the server at any one time. + +Importance of Persisting UUIDs +------------------------------ + +The choices of how, where, and when document UUIDs are stored and updated +are very important. One should not attempt to change things unless she +has a very detailed understanding of why things are the way they are. + +The client is purposefully very conservative about forgetting about +generated UUIDs. In other words, once a UUID is generated, the client +deliberately holds on to that UUID until it's very confident that UUID +is no longer stored on the server. The reason we do this is because +*orphaned* documents/UUIDs on the server can lead to faulty analysis, +such as over-reporting the number of Firefox installs that stop being +used. + +When uploading a new UUID, we update the state and save the state file +to disk *before* an upload attempt because if the upload succeeds but +the response never makes it back to the client, we want the client to +know about the uploaded UUID so it can delete it later to prevent an +orphan. + +We maintain a list of UUIDs locally (not simply the last UUID) because +multiple upload attempts could fail the same way as the previous +paragraph describes and we have no way of knowing which (if any) +actually succeeded. The safest approach is to assume every document +produced managed to get uploaded some how. + +We store the UUIDs on a file on disk and not anywhere else because we +want storage to be robust. We originally stored UUIDs in preferences, +which only flush to disk periodically. Writes to preferences were +apparently getting lost. We switched to writing directly to files to +eliminate this window. diff --git a/toolkit/components/telemetry/docs/obsolete/fhr/dataformat.rst b/toolkit/components/telemetry/docs/obsolete/fhr/dataformat.rst new file mode 100644 index 0000000000..730d7514b8 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/fhr/dataformat.rst @@ -0,0 +1,1998 @@ +.. _healthreport_dataformat: + +============== +Payload Format +============== + +Currently, the Firefox Health Report is submitted as a compressed JSON +document. The root JSON element is an object. A *version* field defines +the version of the payload which in turn defines the expected contents +the object. + +As of 2013-07-03, desktop submits Version 2, and Firefox for Android submits +Version 3 payloads. + +Version 3 +========= + +Version 3 is a complete rebuild of the document format. Events are tracked in +an "environment". Environments are computed from a large swath of local data +(e.g., add-ons, CPU count, versions), and a new environment comes into being +when one of its attributes changes. + +Client documents, then, will include descriptions of many environments, and +measurements will be attributed to one particular environment. + +A map of environments is present at the top level of the document, with the +current named "current" in the map. Each environment has a hash identifier and +a set of attributes. The current environment is completely described, and has +its hash present in a "hash" attribute. All other environments are represented +as a tree diff from the current environment, with their hash as the key in the +"environments" object. + +A removed add-on has the value 'null'. + +There is no "last" data at present. + +Daily data is hierarchical: by day, then by environment, and then by +measurement, and is present in "data", just as in v2. + +Leading by example:: + + { + "lastPingDate": "2013-06-29", + "thisPingDate": "2013-07-03", + "version": 3, + "environments": { + "current": { + "org.mozilla.sysinfo.sysinfo": { + "memoryMB": 1567, + "cpuCount": 4, + "architecture": "armeabi-v7a", + "_v": 1, + "version": "4.1.2", + "name": "Android" + }, + "org.mozilla.profile.age": { + "_v": 1, + "profileCreation": 15827 + }, + "org.mozilla.addons.active": { + "QuitNow@TWiGSoftware.com": { + "appDisabled": false, + "userDisabled": false, + "scope": 1, + "updateDay": 15885, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15885, + "version": "1.18.02" + }, + "{dbbf9331-b713-6eda-1006-205efead09dc}": { + "appDisabled": false, + "userDisabled": "askToActivate", + "scope": 8, + "updateDay": 15779, + "foreignInstall": true, + "blocklistState": 0, + "type": "plugin", + "installDay": 15779, + "version": "11.1 r115" + }, + "desktopbydefault@bnicholson.mozilla.org": { + "appDisabled": false, + "userDisabled": true, + "scope": 1, + "updateDay": 15870, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15870, + "version": "1.1" + }, + "{6e092a7f-ba58-4abb-88c1-1a4e50b217e4}": { + "appDisabled": false, + "userDisabled": false, + "scope": 1, + "updateDay": 15828, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15828, + "version": "1.1.0" + }, + "{46551EC9-40F0-4e47-8E18-8E5CF550CFB8}": { + "appDisabled": false, + "userDisabled": true, + "scope": 1, + "updateDay": 15879, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15879, + "version": "1.3.2" + }, + "_v": 1 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 3, + "appLocale": "en_us", + "osLocale": "en_us", + "distribution": "", + "acceptLangIsUserSet": 0, + "isTelemetryEnabled": 1, + "isBlocklistEnabled": 1 + }, + "geckoAppInfo": { + "updateChannel": "nightly", + "id": "{aa3c5121-dab2-40e2-81ca-7ea25febc110}", + "os": "Android", + "platformBuildID": "20130703031323", + "platformVersion": "25.0a1", + "vendor": "Mozilla", + "name": "fennec", + "xpcomabi": "arm-eabi-gcc3", + "appBuildID": "20130703031323", + "_v": 1, + "version": "25.0a1" + }, + "hash": "tB4Pnnep9yTxnMDymc3dAB2RRB0=", + "org.mozilla.addons.counts": { + "extension": 4, + "plugin": 1, + "_v": 1, + "theme": 0 + } + }, + "k2O3hlreMeS7L1qtxeMsYWxgWWQ=": { + "geckoAppInfo": { + "platformBuildID": "20130630031138", + "appBuildID": "20130630031138", + "_v": 1 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 2, + } + }, + "1+KN9TutMpzdl4TJEl+aCxK+xcw=": { + "geckoAppInfo": { + "platformBuildID": "20130626031100", + "appBuildID": "20130626031100", + "_v": 1 + }, + "org.mozilla.addons.active": { + "QuitNow@TWiGSoftware.com": null, + "{dbbf9331-b713-6eda-1006-205efead09dc}": null, + "desktopbydefault@bnicholson.mozilla.org": null, + "{6e092a7f-ba58-4abb-88c1-1a4e50b217e4}": null, + "{46551EC9-40F0-4e47-8E18-8E5CF550CFB8}": null, + "_v": 1 + }, + "org.mozilla.addons.counts": { + "extension": 0, + "plugin": 0, + "_v": 1 + } + } + }, + "data": { + "last": {}, + "days": { + "2013-07-03": { + "tB4Pnnep9yTxnMDymc3dAB2RRB0=": { + "org.mozilla.appSessions": { + "normal": [ + { + "r": "P", + "d": 2, + "sj": 653 + }, + { + "r": "P", + "d": 22 + }, + { + "r": "P", + "d": 5 + }, + { + "r": "P", + "d": 0 + }, + { + "r": "P", + "sg": 3560, + "d": 171, + "sj": 518 + }, + { + "r": "P", + "d": 16 + }, + { + "r": "P", + "d": 1079 + } + ], + "_v": "4" + } + }, + "k2O3hlreMeS7L1qtxeMsYWxgWWQ=": { + "org.mozilla.appSessions": { + "normal": [ + { + "r": "P", + "d": 27 + }, + { + "r": "P", + "d": 19 + }, + { + "r": "P", + "d": 55 + } + ], + "_v": "4" + }, + "org.mozilla.searches.counts": { + "bartext": { + "google": 1 + }, + "_v": "4" + }, + "org.mozilla.experiment": { + "lastActive": "some.experiment.id" + "_v": "1" + } + } + } + } + } + } + +App sessions in Version 3 +------------------------- + +Sessions are divided into "normal" and "abnormal". Session objects are stored as discrete JSON:: + + "org.mozilla.appSessions": { + _v: 4, + "normal": [ + {"r":"P", "d": 123}, + ], + "abnormal": [ + {"r":"A", "oom": true, "stopped": false} + ] + } + +Keys are: + +"r" + reason. Values are "P" (activity paused), "A" (abnormal termination). +"d" + duration. Value in seconds. +"sg" + Gecko startup time (msec). Present if this is a clean launch. This + corresponds to the telemetry timer *FENNEC_STARTUP_TIME_GECKOREADY*. +"sj" + Java activity init time (msec). Present if this is a clean launch. This + corresponds to the telemetry timer *FENNEC_STARTUP_TIME_JAVAUI*, + and includes initialization tasks beyond initial + *onWindowFocusChanged*. + +Abnormal terminations will be missing a duration and will feature these keys: + +"oom" + was the session killed by an OOM exception? +"stopped" + was the session stopped gently? + +Version 3.2 +----------- + +As of Firefox 35, the search counts measurement is now bumped to v6, including the *activity* location for the search activity. + +Version 3.1 +----------- + +As of Firefox 27, *appinfo* is now bumped to v3, including *osLocale*, +*appLocale* (currently always the same as *osLocale*), *distribution* (a string +containing the distribution ID and version, separated by a colon), and +*acceptLangIsUserSet*, an integer-boolean that describes whether the user set +an *intl.accept_languages* preference. + +The search counts measurement is now at version 5, which indicates that +non-partner searches are recorded. You'll see identifiers like "other-Foo Bar" +rather than "other". + + +Version 3.2 +----------- + +In Firefox 32, Firefox for Android includes a device configuration section +in the environment description:: + + "org.mozilla.device.config": { + "hasHardwareKeyboard": false, + "screenXInMM": 58, + "screenLayout": 2, + "uiType": "default", + "screenYInMM": 103, + "_v": 1, + "uiMode": 1 + } + +Of these, the only keys that need explanation are: + +uiType + One of "default", "smalltablet", "largetablet". +uiMode + A mask of the Android *Configuration.uiMode* value, e.g., + *UI_MODE_TYPE_CAR*. +screenLayout + A mask of the Android *Configuration.screenLayout* value. One of the + *SCREENLAYOUT_SIZE_* constants. + +Note that screen dimensions can be incorrect due to device inaccuracies and platform limitations. + +Other notable differences from Version 2 +---------------------------------------- + +* There is no default browser indicator on Android. +* Add-ons include a *blocklistState* attribute, as returned by AddonManager. +* Searches are now version 4, and are hierarchical: how the search was started + (bartext, barkeyword, barsuggest), and then counts per provider. + +Version 2 +========= + +Version 2 is the same as version 1 with the exception that it has an additional +top-level field, *geckoAppInfo*, which contains basic application info. + +geckoAppInfo +------------ + +This field is an object that is a simple map of string keys and values +describing basic application metadata. It is very similar to the *appinfo* +measurement in the *last* section. The difference is this field is almost +certainly guaranteed to exist whereas the one in the data part of the +payload may be omitted in certain scenarios (such as catastrophic client +error). + +Its keys are as follows: + +appBuildID + The build ID/date of the application. e.g. "20130314113542". + +version + The value of nsXREAppData.version. This is the application's version. e.g. + "21.0.0". + +vendor + The value of nsXREAppData.vendor. Can be empty an empty string. For + official Mozilla builds, this will be "Mozilla". + +name + The value of nsXREAppData.name. For official Firefox builds, this + will be "Firefox". + +id + The value of nsXREAppData.ID. + +platformVersion + The version of the Gecko platform (as opposed to the app version). For + Firefox, this is almost certainly equivalent to the *version* field. + +platformBuildID + The build ID/date of the Gecko platfor (as opposed to the app version). + This is commonly equivalent to *appBuildID*. + +os + The name of the operating system the application is running on. + +xpcomabi + The binary architecture of the build. + +updateChannel + The name of the channel used for application updates. Official Mozilla + builds have one of the values {release, beta, aurora, nightly}. Local and + test builds have *default* as the channel. + +Version 1 +========= + +Top-level Properties +-------------------- + +The main JSON object contains the following properties: + +lastPingDate + UTC date of the last upload. If this is the first upload from this client, + this will not be present. + +thisPingDate + UTC date when this payload was constructed. + +version + Integer version of this payload format. Currently only 1 is defined. + +clientID + An identifier that identifies the client that is submitting data. + + This property may not be present in older clients. + + See :ref:`healthreport_identifiers` for more info on identifiers. + +clientIDVersion + Integer version associated with the generation semantics for the + ``clientID``. + + If the value is ``1``, ``clientID`` is a randomly-generated UUID. + + This property may not be present in older clients. + +data + Object holding data constituting health report. + +Data Properties +--------------- + +The bulk of the health report is contained within the *data* object. This +object has the following keys: + +days + Object mapping UTC days to measurements from that day. Keys are in the + *YYYY-MM-DD* format. e.g. "2013-03-14" + +last + Object mapping measurement names to their values. + + +The value of *days* and *last* are objects mapping measurement names to that +measurement's values. The values are always objects. Each object contains +a *_v* property. This property defines the version of this measurement. +Additional non-underscore-prefixed properties are defined by the measurement +itself (see sections below). + +Example +------- + +Here is an example JSON document for version 1:: + + { + "version": 1, + "thisPingDate": "2013-03-11", + "lastPingDate": "2013-03-10", + "data": { + "last": { + "org.mozilla.addons.active": { + "masspasswordreset@johnathan.nightingale": { + "userDisabled": false, + "appDisabled": false, + "version": "1.05", + "type": "extension", + "scope": 1, + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 14973, + "updateDay": 15317 + }, + "places-maintenance@bonardo.net": { + "userDisabled": false, + "appDisabled": false, + "version": "1.3", + "type": "extension", + "scope": 1, + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 15268, + "updateDay": 15379 + }, + "_v": 1 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "appBuildID": "20130309030841", + "distributionID": "", + "distributionVersion": "", + "hotfixVersion": "", + "id": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", + "locale": "en-US", + "name": "Firefox", + "os": "Darwin", + "platformBuildID": "20130309030841", + "platformVersion": "22.0a1", + "updateChannel": "nightly", + "vendor": "Mozilla", + "version": "22.0a1", + "xpcomabi": "x86_64-gcc3" + }, + "org.mozilla.profile.age": { + "_v": 1, + "profileCreation": 12444 + }, + "org.mozilla.appSessions.current": { + "_v": 3, + "startDay": 15773, + "activeTicks": 522, + "totalTime": 70858, + "main": 1245, + "firstPaint": 2695, + "sessionRestored": 3436 + }, + "org.mozilla.sysinfo.sysinfo": { + "_v": 1, + "cpuCount": 8, + "memoryMB": 16384, + "architecture": "x86-64", + "name": "Darwin", + "version": "12.2.1" + } + }, + "days": { + "2013-03-11": { + "org.mozilla.addons.counts": { + "_v": 1, + "extension": 15, + "plugin": 12, + "theme": 1 + }, + "org.mozilla.places.places": { + "_v": 1, + "bookmarks": 757, + "pages": 104858 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "isDefaultBrowser": 1 + } + }, + "2013-03-10": { + "org.mozilla.addons.counts": { + "_v": 1, + "extension": 15, + "plugin": 12, + "theme": 1 + }, + "org.mozilla.places.places": { + "_v": 1, + "bookmarks": 757, + "pages": 104857 + }, + "org.mozilla.searches.counts": { + "_v": 1, + "google.urlbar": 4 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "isDefaultBrowser": 1 + } + } + } + } + } + +Measurements +============ + +The bulk of payloads consists of measurement data. An individual measurement +is merely a collection of related values e.g. *statistics about the Places +database* or *system information*. + +Each measurement has an integer version number attached. When the fields in +a measurement or the semantics of data within that measurement change, the +version number is incremented. + +All measurements are defined alphabetically in the sections below. + +org.mozilla.addons.addons +------------------------- + +This measurement contains information about the currently-installed add-ons. + +Version 2 +^^^^^^^^^ + +This version adds the human-readable fields *name* and *description*, both +coming directly from the Addon instance as most properties in version 1. +Also, all plugin details are now in org.mozilla.addons.plugins. + +Version 1 +^^^^^^^^^ + +The measurement object is a mapping of add-on IDs to objects containing +add-on metadata. + +Each add-on contains the following properties: + +* userDisabled +* appDisabled +* version +* type +* scope +* foreignInstall +* hasBinaryComponents +* installDay +* updateDay + +With the exception of *installDay* and *updateDay*, all these properties +come direct from the Addon instance. See https://developer.mozilla.org/en-US/docs/Addons/Add-on_Manager/Addon. +*installDay* and *updateDay* are the number of days since UNIX epoch of +the add-ons *installDate* and *updateDate* properties, respectively. + +Notes +^^^^^ + +Add-ons that have opted out of AMO updates via the +*extensions._id_.getAddons.cache.enabled* preference are, since Bug 868306 +(Firefox 24), included in the list of submitted add-ons. + +Example +^^^^^^^ +:: + + "org.mozilla.addons.addons": { + "_v": 2, + "{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}": { + "userDisabled": false, + "appDisabled": false, + "name": "Adblock Plus", + "version": "2.4.1", + "type": "extension", + "scope": 1, + "description": "Ads were yesterday!", + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 16093, + "updateDay": 16093 + }, + "{e4a8a97b-f2ed-450b-b12d-ee082ba24781}": { + "userDisabled": true, + "appDisabled": false, + "name": "Greasemonkey", + "version": "1.14", + "type": "extension", + "scope": 1, + "description": "A User Script Manager for Firefox", + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 16093, + "updateDay": 16093 + } + } + +org.mozilla.addons.plugins +-------------------------- + +This measurement contains information about the currently-installed plugins. + +Version 1 +^^^^^^^^^ + +The measurement object is a mapping of plugin IDs to objects containing +plugin metadata. + +The plugin ID is constructed of the plugins filename, name, version and +description. Every plugin has at least a filename and a name. + +Each plugin contains the following properties: + +* name +* version +* description +* blocklisted +* disabled +* clicktoplay +* mimeTypes +* updateDay + +With the exception of *updateDay* and *mimeTypes*, all these properties come +directly from ``nsIPluginTag`` via ``nsIPluginHost``. +*updateDay* is the number of days since UNIX epoch of the plugins last modified +time. +*mimeTypes* is the list of mimetypes the plugin supports, see +``nsIPluginTag.getMimeTypes()``. + +Example +^^^^^^^ + +:: + + "org.mozilla.addons.plugins": { + "_v": 1, + "Flash Player.plugin:Shockwave Flash:12.0.0.38:Shockwave Flash 12.0 r0": { + "mimeTypes": [ + "application/x-shockwave-flash", + "application/futuresplash" + ], + "name": "Shockwave Flash", + "version": "12.0.0.38", + "description": "Shockwave Flash 12.0 r0", + "blocklisted": false, + "disabled": false, + "clicktoplay": false + }, + "Default Browser.plugin:Default Browser Helper:537:Provides information about the default web browser": { + "mimeTypes": [ + "application/apple-default-browser" + ], + "name": "Default Browser Helper", + "version": "537", + "description": "Provides information about the default web browser", + "blocklisted": false, + "disabled": true, + "clicktoplay": false + } + } + +org.mozilla.addons.counts +------------------------- + +This measurement contains information about historical add-on counts. + +Version 1 +^^^^^^^^^ + +The measurement object consists of counts of different add-on types. The +properties are: + +extension + Integer count of installed extensions. +plugin + Integer count of installed plugins. +theme + Integer count of installed themes. +lwtheme + Integer count of installed lightweight themes. + +Notes +^^^^^ + +Add-ons opted out of AMO updates are included in the counts. This differs from +the behavior of the active add-ons measurement. + +If no add-ons of a particular type are installed, the property for that type +will not be present (as opposed to an explicit property with value of 0). + +Example +^^^^^^^ + +:: + + "2013-03-14": { + "org.mozilla.addons.counts": { + "_v": 1, + "extension": 21, + "plugin": 4, + "theme": 1 + } + } + + + +org.mozilla.appInfo.appinfo +--------------------------- + +This measurement contains basic XUL application and Gecko platform +information. It is reported in the *last* section. + +Version 2 +^^^^^^^^^ + +In addition to fields present in version 1, this version has the following +fields appearing in the *days* section: + +isBlocklistEnabled + Whether the blocklist ping is enabled. This is an integer, 0 or 1. + This does not indicate whether the blocklist ping was sent but merely + whether the application will try to send the blocklist ping. + +isTelemetryEnabled + Whether Telemetry is enabled. This is an integer, 0 or 1. + +Version 1 +^^^^^^^^^ + +The measurement object contains mostly string values describing the +current application and build. The properties are: + +* vendor +* name +* id +* version +* appBuildID +* platformVersion +* platformBuildID +* os +* xpcomabi +* updateChannel +* distributionID +* distributionVersion +* hotfixVersion +* locale +* isDefaultBrowser + +Notes +^^^^^ + +All of the properties appear in the *last* section except for +*isDefaultBrowser*, which appears under *days*. + +Example +^^^^^^^ + +This example comes from an official macOS Nightly build:: + + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "appBuildID": "20130311030946", + "distributionID": "", + "distributionVersion": "", + "hotfixVersion": "", + "id": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", + "locale": "en-US", + "name": "Firefox", + "os": "Darwin", + "platformBuildID": "20130311030946", + "platformVersion": "22.0a1", + "updateChannel": "nightly", + "vendor": "Mozilla", + "version": "22.0a1", + "xpcomabi": "x86_64-gcc3" + }, + +org.mozilla.appInfo.update +-------------------------- + +This measurement contains information about the application update mechanism +in the application. + +Version 1 +^^^^^^^^^ + +The following daily values are reported: + +enabled + Whether automatic application update checking is enabled. 1 for yes, + 0 for no. +autoDownload + Whether automatic download of available updates is enabled. + +Notes +^^^^^ + +This measurement was merged to mozilla-central for JS FHR on 2013-07-15. + +Example +^^^^^^^ + +:: + + "2013-07-15": { + "org.mozilla.appInfo.update": { + "_v": 1, + "enabled": 1, + "autoDownload": 1, + } + } + +org.mozilla.appInfo.versions +---------------------------- + +This measurement contains a history of application version numbers. + +Version 2 +^^^^^^^^^ + +Version 2 reports more fields than version 1 and is not backwards compatible. +The following fields are present in version 2: + +appVersion + An array of application version strings. +appBuildID + An array of application build ID strings. +platformVersion + An array of platform version strings. +platformBuildID + An array of platform build ID strings. + +When the application is upgraded, the new version and/or build IDs are +appended to their appropriate fields. + +Version 1 +^^^^^^^^^ + +When the application version (*version* from *org.mozilla.appinfo.appinfo*) +changes, we record the new version on the day the change was seen. The new +versions for a day are recorded in an array under the *version* property. + +Notes +^^^^^ + +If the application isn't upgraded, this measurement will not be present. +This means this measurement will not be present for most days if a user is +on the release channel (since updates are typically released every 6 weeks). +However, users on the Nightly and Aurora channels will likely have a lot +of these entries since those builds are updated every day. + +Values for this measurement are collected when performing the daily +collection (typically occurs at upload time). As a result, it's possible +the actual upgrade day may not be attributed to the proper day - the +reported day may lag behind. + +The app and platform versions and build IDs should be identical for most +clients. If they are different, we are possibly looking at a *Frankenfox*. + +Example +^^^^^^^ + +:: + + "2013-03-27": { + "org.mozilla.appInfo.versions": { + "_v": 2, + "appVersion": [ + "22.0.0" + ], + "appBuildID": [ + "20130325031100" + ], + "platformVersion": [ + "22.0.0" + ], + "platformBuildID": [ + "20130325031100" + ] + } + } + +org.mozilla.appSessions.current +------------------------------- + +This measurement contains information about the currently running XUL +application's session. + +Version 3 +^^^^^^^^^ + +This measurement has the following properties: + +startDay + Integer days since UNIX epoch when this session began. +activeTicks + Integer count of *ticks* the session was active for. Gecko periodically + sends out a signal when the session is active. Session activity + involves keyboard or mouse interaction with the application. Each tick + represents a window of 5 seconds where there was interaction. +totalTime + Integer seconds the session has been alive. +main + Integer milliseconds it took for the Gecko process to start up. +firstPaint + Integer milliseconds from process start to first paint. +sessionRestored + Integer milliseconds from process start to session restore. + +Example +^^^^^^^ + +:: + + "org.mozilla.appSessions.current": { + "_v": 3, + "startDay": 15775, + "activeTicks": 4282, + "totalTime": 249422, + "main": 851, + "firstPaint": 3271, + "sessionRestored": 5998 + } + +org.mozilla.appSessions.previous +-------------------------------- + +This measurement contains information about previous XUL application sessions. + +Version 3 +^^^^^^^^^ + +This measurement contains per-day lists of all the sessions started on that +day. The following properties may be present: + +cleanActiveTicks + Active ticks of sessions that were properly shut down. +cleanTotalTime + Total number of seconds for sessions that were properly shut down. +abortedActiveTicks + Active ticks of sessions that were not properly shut down. +abortedTotalTime + Total number of seconds for sessions that were not properly shut down. +main + Time in milliseconds from process start to main process initialization. +firstPaint + Time in milliseconds from process start to first paint. +sessionRestored + Time in milliseconds from process start to session restore. + +Notes +^^^^^ + +Sessions are recorded on the date on which they began. + +If a session was aborted/crashed, the total time may be less than the actual +total time. This is because we don't always update total time during periods +of inactivity and the abort/crash could occur after a long period of idle, +before we've updated the total time. + +The lengths of the arrays for {cleanActiveTicks, cleanTotalTime}, +{abortedActiveTicks, abortedTotalTime}, and {main, firstPaint, sessionRestored} +should all be identical. + +The length of the clean sessions plus the length of the aborted sessions should +be equal to the length of the {main, firstPaint, sessionRestored} properties. + +It is not possible to distinguish the main, firstPaint, and sessionRestored +values from a clean vs aborted session: they are all lumped together. + +For sessions spanning multiple UTC days, it's not possible to know which +days the session was active for. It's possible a week long session only +had activity for 2 days and there's no way for us to tell which days. + +Example +^^^^^^^ + +:: + + "org.mozilla.appSessions.previous": { + "_v": 3, + "cleanActiveTicks": [ + 78, + 1785 + ], + "cleanTotalTime": [ + 4472, + 88908 + ], + "main": [ + 32, + 952 + ], + "firstPaint": [ + 2755, + 3497 + ], + "sessionRestored": [ + 5149, + 5520 + ] + } + +org.mozilla.crashes.crashes +--------------------------- + +This measurement contains a historical record of application crashes. + +Version 6 +^^^^^^^^^ + +This version adds tracking for out-of-memory (OOM) crashes in the main process. +An OOM crash will be counted as both main-crash and main-crash-oom. + +This measurement will be reported on each day there was a crash or crash +submission. Records may contain the following fields, whose values indicate +the number of crashes, hangs, or submissions that occurred on the given day: + +* content-crash +* content-crash-submission-succeeded +* content-crash-submission-failed +* content-hang +* content-hang-submission-succeeded +* content-hang-submission-failed +* gmplugin-crash +* gmplugin-crash-submission-succeeded +* gmplugin-crash-submission-failed +* main-crash +* main-crash-oom +* main-crash-submission-succeeded +* main-crash-submission-failed +* main-hang +* main-hang-submission-succeeded +* main-hang-submission-failed +* plugin-crash +* plugin-crash-submission-succeeded +* plugin-crash-submission-failed +* plugin-hang +* plugin-hang-submission-succeeded +* plugin-hang-submission-failed + +Version 5 +^^^^^^^^^ + +This version adds support for Gecko media plugin (GMP) crashes. + +This measurement will be reported on each day there was a crash or crash +submission. Records may contain the following fields, whose values indicate +the number of crashes, hangs, or submissions that occurred on the given day: + +* content-crash +* content-crash-submission-succeeded +* content-crash-submission-failed +* content-hang +* content-hang-submission-succeeded +* content-hang-submission-failed +* gmplugin-crash +* gmplugin-crash-submission-succeeded +* gmplugin-crash-submission-failed +* main-crash +* main-crash-submission-succeeded +* main-crash-submission-failed +* main-hang +* main-hang-submission-succeeded +* main-hang-submission-failed +* plugin-crash +* plugin-crash-submission-succeeded +* plugin-crash-submission-failed +* plugin-hang +* plugin-hang-submission-succeeded +* plugin-hang-submission-failed + +Version 4 +^^^^^^^^^ + +This version follows up from version 3, adding submissions which are now +tracked by the :ref:`crashes_crashmanager`. + +This measurement will be reported on each day there was a crash or crash +submission. Records may contain the following fields, whose values indicate +the number of crashes, hangs, or submissions that occurred on the given day: + +* main-crash +* main-crash-submission-succeeded +* main-crash-submission-failed +* main-hang +* main-hang-submission-succeeded +* main-hang-submission-failed +* content-crash +* content-crash-submission-succeeded +* content-crash-submission-failed +* content-hang +* content-hang-submission-succeeded +* content-hang-submission-failed +* plugin-crash +* plugin-crash-submission-succeeded +* plugin-crash-submission-failed +* plugin-hang +* plugin-hang-submission-succeeded +* plugin-hang-submission-failed + +Version 3 +^^^^^^^^^ + +This version follows up from version 2, building on improvements to +the :ref:`crashes_crashmanager`. + +This measurement will be reported on each day there was a +crash. Records may contain the following fields, whose values indicate +the number of crashes or hangs that occurred on the given day: + +* main-crash +* main-hang +* content-crash +* content-hang +* plugin-crash +* plugin-hang + +Version 2 +^^^^^^^^^ + +The switch to version 2 coincides with the introduction of the +:ref:`crashes_crashmanager`, which provides a more robust source of +crash data. + +This measurement will be reported on each day there was a crash. The +following fields may be present in each record: + +mainCrash + The number of main process crashes that occurred on the given day. + +Yes, version 2 does not track submissions like version 1. It is very +likely submissions will be re-added later. + +Also absent from version 2 are plugin crashes and hangs. These will be +re-added, likely in version 3. + +Version 1 +^^^^^^^^^ + +This measurement will be reported on each day there was a crash. The +following properties are reported: + +pending + The number of crash reports that haven't been submitted. +submitted + The number of crash reports that were submitted. + +Notes +^^^^^ + +Main process crashes are typically submitted immediately after they +occur (by checking a box in the crash reporter, which should appear +automatically after a crash). If the crash reporter submits the crash +successfully, we get a submitted crash. Else, we leave it as pending. + +A pending crash does not mean it will eventually be submitted. + +Pending crash reports can be submitted post-crash by going to +about:crashes. + +If a pending crash is submitted via about:crashes, the submitted count +increments but the pending count does not decrement. This is because FHR +does not know which pending crash was just submitted and therefore it does +not know which day's pending crash to decrement. + +Example +^^^^^^^ + +:: + + "org.mozilla.crashes.crashes": { + "_v": 1, + "pending": 1, + "submitted": 2 + }, + "org.mozilla.crashes.crashes": { + "_v": 2, + "mainCrash": 2 + } + "org.mozilla.crashes.crashes": { + "_v": 4, + "main-crash": 2, + "main-crash-submission-succeeded": 1, + "main-crash-submission-failed": 1, + "main-hang": 1, + "plugin-crash": 2 + } + +org.mozilla.healthreport.submissions +------------------------------------ + +This measurement contains a history of FHR's own data submission activity. +It was added in Firefox 23 in early May 2013. + +Version 2 +^^^^^^^^^ + +This is the same as version 1 except an additional field has been added. + +uploadAlreadyInProgress + A request for upload was initiated while another upload was in progress. + This should not occur in well-behaving clients. It (along with a lock + preventing simultaneous upload) was added to ensure this never occurs. + +Version 1 +^^^^^^^^^ + +Daily counts of upload events are recorded. + +firstDocumentUploadAttempt + An attempt was made to upload the client's first document to the server. + These are uploads where the client is not aware of a previous document ID + on the server. Unless the client had disabled upload, there should be at + most one of these in the history of the client. + +continuationUploadAttempt + An attempt was made to upload a document that replaces an existing document + on the server. Most upload attempts should be attributed to this as opposed + to *firstDocumentUploadAttempt*. + +uploadSuccess + The upload attempt recorded by *firstDocumentUploadAttempt* or + *continuationUploadAttempt* was successful. + +uploadTransportFailure + An upload attempt failed due to transport failure (network unavailable, + etc). + +uploadServerFailure + An upload attempt failed due to a server-reported failure. Ideally these + are failures reported by the FHR server itself. However, intermediate + proxies, firewalls, etc may trigger this depending on how things are + configured. + +uploadClientFailure + An upload attempt failued due to an error/exception in the client. + This almost certainly points to a bug in the client. + +The result for an upload attempt is always attributed to the same day as +the attempt, even if the result occurred on a different day from the attempt. +Therefore, the sum of the result counts should equal the result of the attempt +counts. + +org.mozilla.hotfix.update +------------------------- + +This measurement contains results from the Firefox update hotfix. + +The Firefox update hotfix bypasses the built-in application update mechanism +and installs a modern Firefox. + +Version 1 +^^^^^^^^^ + +The fields in this measurement are dynamically created based on which +versions of the update hotfix state file are found on disk. + +The general format of the fields is ``<version>.<thing>`` where ``version`` +is a hotfix version like ``v20140527`` and ``thing`` is a key from the +hotfix state file, e.g. ``upgradedFrom``. Here are some of the ``things`` +that can be defined. + +upgradedFrom + String identifying the Firefox version that the hotfix upgraded from. + e.g. ``16.0`` or ``17.0.1``. + +uninstallReason + String with enumerated values identifying why the hotfix was uninstalled. + Value will be ``STILL_INSTALLED`` if the hotfix is still installed. + +downloadAttempts + Integer number of times the hotfix started downloading an installer. + Download resumes are part of this count. + +downloadFailures + Integer count of times a download supposedly completed but couldn't + be validated. This likely represents something wrong with the network + connection. The ratio of this to ``downloadAttempts`` should be low. + +installAttempts + Integer count of times the hotfix attempted to run the installer. + This should ideally be 1. It should only be greater than 1 if UAC + elevation was canceled or not allowed. + +installFailures + Integer count of total installation failures this client experienced. + Can be 0. ``installAttempts - installFailures`` implies install successes. + +notificationsShown + Integer count of times a notification was displayed to the user that + they are running an older Firefox. + +org.mozilla.places.places +------------------------- + +This measurement contains information about the Places database (where Firefox +stores its history and bookmarks). + +Version 1 +^^^^^^^^^ + +Daily counts of items in the database are reported in the following properties: + +bookmarks + Integer count of bookmarks present. +pages + Integer count of pages in the history database. + +Example +^^^^^^^ + +:: + + "org.mozilla.places.places": { + "_v": 1, + "bookmarks": 388, + "pages": 94870 + } + +org.mozilla.profile.age +----------------------- + +This measurement contains information about the current profile's age (and +in version 2, the profile's most recent reset date) + +Version 2 +^^^^^^^^^ + +*profileCreation* and *profileReset* properties are present. Both define +the integer days since UNIX epoch that the current profile was created or +reset accordingly. + +Version 1 +^^^^^^^^^ + +A single *profileCreation* property is present. It defines the integer +days since UNIX epoch that the current profile was created. + +Notes +^^^^^ + +It is somewhat difficult to obtain a reliable *profile born date* due to a +number of factors, but since Version 2, improvements have been made - on a +"profile reset" we copy the profileCreation date from the old profile and +record the time of the reset in profileReset. + +Example +^^^^^^^ + +:: + + "org.mozilla.profile.age": { + "_v": 2, + "profileCreation": 15176 + "profileReset": 15576 + } + +org.mozilla.searches.counts +--------------------------- + +This measurement contains information about searches performed in the +application. + +Version 6 (mobile) +^^^^^^^^^^^^^^^^^^ + +This adds two new search locations: *widget* and *activity*, corresponding to the search widget and search activity respectively. + +Version 2 +^^^^^^^^^ + +This behaves like version 1 except we added all search engines that +Mozilla has a partner agreement with. Like version 1, we concatenate +a search engine ID with a search origin. + +Another difference with version 2 is we should no longer misattribute +a search to the *other* bucket if the search engine name is localized. + +The set of search engine providers is: + +* amazon-co-uk +* amazon-de +* amazon-en-GB +* amazon-france +* amazon-it +* amazon-jp +* amazondotcn +* amazondotcom +* amazondotcom-de +* aol-en-GB +* aol-web-search +* bing +* eBay +* eBay-de +* eBay-en-GB +* eBay-es +* eBay-fi +* eBay-france +* eBay-hu +* eBay-in +* eBay-it +* google +* google-jp +* google-ku +* google-maps-zh-TW +* mailru +* mercadolibre-ar +* mercadolibre-cl +* mercadolibre-mx +* seznam-cz +* twitter +* twitter-de +* twitter-ja +* yahoo +* yahoo-NO +* yahoo-answer-zh-TW +* yahoo-ar +* yahoo-bid-zh-TW +* yahoo-br +* yahoo-ch +* yahoo-cl +* yahoo-de +* yahoo-en-GB +* yahoo-es +* yahoo-fi +* yahoo-france +* yahoo-fy-NL +* yahoo-id +* yahoo-in +* yahoo-it +* yahoo-jp +* yahoo-jp-auctions +* yahoo-mx +* yahoo-sv-SE +* yahoo-zh-TW +* yandex +* yandex-ru +* yandex-slovari +* yandex-tr +* yandex.by +* yandex.ru-be + +And of course, *other*. + +The sources for searches remain: + +* abouthome +* contextmenu +* searchbar +* urlbar + +The measurement will only be populated with providers and sources that +occurred that day. + +If a user switches locales, searches from default providers on the older +locale will still be supported. However, if that same search engine is +added by the user to the new build and is *not* a default search engine +provider, its searches will be attributed to the *other* bucket. + +Version 1 +^^^^^^^^^ + +We record counts of performed searches grouped by search engine and search +origin. Only search engines with which Mozilla has a business relationship +are explicitly counted. All other search engines are grouped into an +*other* bucket. + +The following search engines are explicitly counted: + +* Amazon.com +* Bing +* Google +* Yahoo +* Other + +The following search origins are distinguished: + +about:home + Searches initiated from the search text box on about:home. +context menu + Searches initiated from the context menu (highlight text, right click, + and select "search for...") +search bar + Searches initiated from the search bar (the text field next to the + Awesomebar) +url bar + Searches initiated from the awesomebar/url bar. + +Due to the localization of search engine names, non en-US locales may wrongly +attribute searches to the *other* bucket. This is fixed in version 2. + +Example +^^^^^^^ + +:: + + "org.mozilla.searches.counts": { + "_v": 1, + "google.searchbar": 3, + "google.urlbar": 7 + }, + +org.mozilla.searches.engines +---------------------------- + +This measurement contains information about search engines. + +Version 1 +^^^^^^^^^ + +This version debuted with Firefox 31 on desktop. It contains the +following properties: + +default + Daily string identifier or name of the default search engine provider. + + This field will only be collected if Telemetry is enabled. If + Telemetry is enabled and then later disabled, this field may + disappear from future days in the payload. + + The special value ``NONE`` could occur if there is no default search + engine. + + The special value ``UNDEFINED`` could occur if a default search + engine exists but its identifier could not be determined. + + This field's contents are + ``Services.search.defaultEngine.identifier`` (if defined) or + ``"other-"`` + ``Services.search.defaultEngine.name`` if not. + In other words, search engines without an ``.identifier`` + are prefixed with ``other-``. + +Version 2 +^^^^^^^^^ + +Starting with Firefox 40, there is an additional optional value: + +cohort + Daily cohort string identifier, recorded if the user is part of + search defaults A/B testing. + +org.mozilla.sync.sync +--------------------- + +This daily measurement contains information about the Sync service. + +Values should be recorded for every day FHR measurements occurred. + +Version 1 +^^^^^^^^^ + +This version debuted with Firefox 30 on desktop. It contains the following +properties: + +enabled + Daily numeric indicating whether Sync is configured and enabled. 1 if so, + 0 otherwise. + +preferredProtocol + String version of the maximum Sync protocol version the client supports. + This will be ``1.1`` for for legacy Sync and ``1.5`` for clients that + speak the Firefox Accounts protocol. + +actualProtocol + The actual Sync protocol version the client is configured to use. + + This will be ``1.1`` if the client is configured with the legacy Sync + service or if the client only supports ``1.1``. + + It will be ``1.5`` if the client supports ``1.5`` and either a) the + client is not configured b) the client is using Firefox Accounts Sync. + +syncStart + Count of sync operations performed. + +syncSuccess + Count of sync operations that completed successfully. + +syncError + Count of sync operations that did not complete successfully. + + This is a measure of overall sync success. This does *not* reflect + recoverable errors (such as record conflict) that can occur during + sync. This is thus a rough proxy of whether the sync service is + operating without error. + +org.mozilla.sync.devices +------------------------ + +This daily measurement contains information about the device type composition +for the configured Sync account. + +Version 1 +^^^^^^^^^ + +Version 1 was introduced with Firefox 30. + +Field names are dynamic according to the client-reported device types from +Sync records. All fields are daily last seen integer values corresponding to +the number of devices of that type. + +Common values include: + +desktop + Corresponds to a Firefox desktop client. + +mobile + Corresponds to a Fennec client. + +org.mozilla.sync.migration +-------------------------- + +This daily measurement contains information about sync migration (that is, the +semi-automated process of migrating a legacy sync account to an FxA account.) + +Measurements will start being recorded after a migration is offered by the +sync server and stop after migration is complete or the user elects to "unlink" +their sync account. In other words, it is expected that users with Sync setup +for FxA or with sync unconfigured will not collect data, and that for users +where data is collected, the collection will only be for a relatively short +period. + +Version 1 +^^^^^^^^^ + +Version 1 was introduced with Firefox 37 and includes the following properties: + +state + Corresponds to either a STATE_USER_* string or a STATE_INTERNAL_* string in + FxaMigration.jsm. This reflects a state where we are waiting for the user, + or waiting for some internal process to complete on the way to completing + the migration. + +declined + Corresponds to the number of times the user closed the migration infobar. + +unlinked + Set if the user declined to migrate and instead "unlinked" Sync from the + browser. + +accepted + Corresponds to the number of times the user explicitly elected to start or + continue the migration - it counts how often the user clicked on any UI + created specifically for migration. The "ideal" UX for migration would see + this at exactly 1, some known edge-cases (eg, browser restart required to + finish) could expect this to be 2, and anything more means we are doing + something wrong. + +org.mozilla.sysinfo.sysinfo +--------------------------- + +This measurement contains basic information about the system the application +is running on. + +Version 2 +^^^^^^^^^ + +This version debuted with Firefox 29 on desktop. + +A single property was introduced. + +isWow64 + If present, this property indicates whether the machine supports WoW64. + This property can be used to identify whether the host machine is 64-bit. + + This property is only present on Windows machines. It is the preferred way + to identify 32- vs 64-bit support in that environment. + +Version 1 +^^^^^^^^^ + +The following properties may be available: + +cpuCount + Integer number of CPUs/cores in the machine. +memoryMB + Integer megabytes of memory in the machine. +manufacturer + The manufacturer of the device. +device + The name of the device (like model number). +hardware + Unknown. +name + OS name. +version + OS version. +architecture + OS architecture that the application is built for. This is not the + actual system architecture. + +Example +^^^^^^^ + +:: + + "org.mozilla.sysinfo.sysinfo": { + "_v": 1, + "cpuCount": 8, + "memoryMB": 8192, + "architecture": "x86-64", + "name": "Darwin", + "version": "12.2.0" + } + + +org.mozilla.translation.translation +----------------------------------- + +This daily measurement contains information about the usage of the translation +feature. It is a special telemetry measurement which will only be recorded in +FHR if telemetry is enabled. + +Version 1 +^^^^^^^^^ + +Daily counts are reported in the following properties: + +translationOpportunityCount + Integer count of the number of opportunities there were to translate a page. +missedTranslationOpportunityCount + Integer count of the number of missed opportunities there were to translate a page. + A missed opportunity is when the page language is not supported by the translation + provider. +pageTranslatedCount + Integer count of the number of pages translated. +charactersTranslatedCount + Integer count of the number of characters translated. +detectedLanguageChangedBefore + Integer count of the number of times the user manually adjusted the detected + language before translating. +detectedLanguageChangedAfter + Integer count of the number of times the user manually adjusted the detected + language after having first translated the page. +targetLanguageChanged + Integer count of the number of times the user manually adjusted the target + language. +deniedTranslationOffer + Integer count of the number of times the user opted-out offered + page translation, either by the Not Now button or by the notification's + close button in the "offer" state. +autoRejectedTranlationOffer + Integer count of the number of times the user is not offered page + translation because they had previously clicked "Never translate this + language" or "Never translate this site". +showOriginalContent + Integer count of the number of times the user activated the Show Original + command. + +Additional daily counts broken down by language are reported in the following +properties: + +translationOpportunityCountsByLanguage + A mapping from language to count of opportunities to translate that + language. +missedTranslationOpportunityCountsByLanguage + A mapping from language to count of missed opportunities to translate that + language. +pageTranslatedCountsByLanguage + A mapping from language to the counts of pages translated from that + language. Each language entry will be an object containing a "total" member + along with individual counts for each language translated to. + +Other properties: + +detectLanguageEnabled + Whether automatic language detection is enabled. This is an integer, 0 or 1. +showTranslationUI + Whether the translation feature UI will be shown. This is an integer, 0 or 1. + +Example +^^^^^^^ + +:: + + "org.mozilla.translation.translation": { + "_v": 1, + "detectLanguageEnabled": 1, + "showTranslationUI": 1, + "translationOpportunityCount": 134, + "missedTranslationOpportunityCount": 32, + "pageTranslatedCount": 6, + "charactersTranslatedCount": "1126", + "detectedLanguageChangedBefore": 1, + "detectedLanguageChangedAfter": 2, + "targetLanguageChanged": 0, + "deniedTranslationOffer": 3, + "autoRejectedTranlationOffer": 1, + "showOriginalContent": 2, + "translationOpportunityCountsByLanguage": { + "fr": 100, + "es": 34 + }, + "missedTranslationOpportunityCountsByLanguage": { + "it": 20, + "nl": 10, + "fi": 2 + }, + "pageTranslatedCountsByLanguage": { + "fr": { + "total": 6, + "es": 5, + "en": 1 + } + } + } + + +org.mozilla.experiments.info +---------------------------------- + +Daily measurement reporting information about the Telemetry Experiments service. + +Version 1 +^^^^^^^^^ + +Property: + +lastActive + ID of the final Telemetry Experiment that is active on a given day, if any. + + +Version 2 +^^^^^^^^^ + +Adds an additional optional property: + +lastActiveBranch + If the experiment uses branches, the branch identifier string. + +Example +^^^^^^^ + +:: + + "org.mozilla.experiments.info": { + "_v": 2, + "lastActive": "some.experiment.id", + "lastActiveBranch": "control" + } + +org.mozilla.uitour.treatment +---------------------------- + +Daily measurement reporting information about treatment tagging done +by the UITour module. + +Version 1 +^^^^^^^^^ + +Daily text values in the following properties: + +<tag>: + Array of discrete strings corresponding to calls for setTreatmentTag(tag, value). + +Example +^^^^^^^ + +:: + + "org.mozilla.uitour.treatment": { + "_v": 1, + "treatment": [ + "optin", + "optin-DNT" + ], + "another-tag": [ + "foobar-value" + ] + } + +org.mozilla.passwordmgr.passwordmgr +----------------------------------- + +Daily measurement reporting information about the Password Manager + +Version 1 +^^^^^^^^^ + +Property: + +numSavedPasswords + number of passwords saved in the Password Manager + +enabled + Whether or not the user has disabled the Password Manager in preferences + +Example +^^^^^^^ + +:: + + "org.mozilla.passwordmgr.passwordmgr": { + "_v": 1, + "numSavedPasswords": 5, + "enabled": 0, + } + +Version 2 +^^^^^^^^^ + +More detailed measurements of login forms & their behavior + +numNewSavedPasswordsInSession + Number of passwords saved to the password manager this session. + +numSuccessfulFills + Number of times the password manager filled in password fields for user this session. + +numTotalLoginsEncountered + Number of times a login form was encountered by the user in the session. + +Example +^^^^^^^ + +:: + + "org.mozilla.passwordmgr.passwordmgr": { + "_v": 2, + "numSavedPasswords": 32, + "enabled": 1, + "numNewSavedPasswords": 5, + "numSuccessfulFills": 11, + "numTotalLoginsEncountered": 23, + } diff --git a/toolkit/components/telemetry/docs/obsolete/fhr/identifiers.rst b/toolkit/components/telemetry/docs/obsolete/fhr/identifiers.rst new file mode 100644 index 0000000000..82ad0e49e6 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/fhr/identifiers.rst @@ -0,0 +1,83 @@ +.. _healthreport_identifiers: + +=========== +Identifiers +=========== + +Firefox Health Report records some identifiers to keep track of clients +and uploaded documents. + +Identifier Types +================ + +Document/Upload IDs +------------------- + +A random UUID called the *Document ID* or *Upload ID* is generated when the FHR +client creates or uploads a new document. + +When clients generate a new *Document ID*, they persist this ID to disk +**before** the upload attempt. + +As part of the upload, the client sends all old *Document IDs* to the server +and asks the server to delete them. In well-behaving clients, the server +has a single record for each client with a randomly-changing *Document ID*. + +Client IDs +---------- + +A *Client ID* is an identifier that **attempts** to uniquely identify an +individual FHR client. Please note the emphasis on *attempts* in that last +sentence: *Client IDs* do not guarantee uniqueness. + +The *Client ID* is generated when the client first runs or as needed. + +The *Client ID* is transferred to the server as part of every upload. The +server is thus able to affiliate multiple document uploads with a single +*Client ID*. + +Client ID Versions +^^^^^^^^^^^^^^^^^^ + +The semantics for how a *Client ID* is generated are versioned. + +Version 1 + The *Client ID* is a randomly-generated UUID. + +History of Identifiers +====================== + +In the beginning, there were just *Document IDs*. The thinking was clients +would clean up after themselves and leave at most 1 active document on the +server. + +Unfortunately, this did not work out. Using brute force analysis to +deduplicate records on the server, a number of interesting patterns emerged. + +Orphaning + Clients would upload a new payload while not deleting the old payload. + +Divergent records + Records would share data up to a certain date and then the data would + almost completely diverge. This appears to be indicative of profile + copying. + +Rollback + Records would share data up to a certain date. Each record in this set + would contain data for a day or two but no extra data. This could be + explained by filesystem rollback on the client. + +A significant percentage of the records on the server belonged to +misbehaving clients. Identifying these records was extremely resource +intensive and error-prone. These records were undermining the ability +to use Firefox Health Report data. + +Thus, the *Client ID* was born. The intent of the *Client ID* was to +uniquely identify clients so the extreme effort required and the +questionable reliability of deduplicating server data would become +problems of the past. + +The *Client ID* was originally a randomly-generated UUID (version 1). This +allowed detection of orphaning and rollback. However, these version 1 +*Client IDs* were still susceptible to use on multiple profiles and +machines if the profile was copied. diff --git a/toolkit/components/telemetry/docs/obsolete/fhr/index.rst b/toolkit/components/telemetry/docs/obsolete/fhr/index.rst new file mode 100644 index 0000000000..d114a02814 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/fhr/index.rst @@ -0,0 +1,34 @@ +================================ +Firefox Health Report (Obsolete) +================================ + +**Firefox Health Report (FHR) is obsolete and no longer ships with Firefox. +This documentation will live here for a few more cycles.** + +Firefox Health Report is a background service that collects application +metrics and periodically submits them to a central server. The core +parts of the service are implemented in this directory. However, the +actual XPCOM service is implemented in the +``data_reporting_service``. + +The core types can actually be instantiated multiple times and used to +power multiple data submission services within a single Gecko +application. In other words, everything in this directory is effectively +a reusable library. However, the terminology and some of the features +are very specific to what the Firefox Health Report feature requires. + +.. toctree:: + :maxdepth: 1 + + architecture + dataformat + identifiers + +Legal and Privacy Concerns +========================== + +Because Firefox Health Report collects and submits data to remote +servers and is an opt-out feature, there are legal and privacy +concerns over what data may be collected and submitted. **Additions or +changes to submitted data should be signed off by responsible +parties.** diff --git a/toolkit/components/telemetry/docs/obsolete/geckoview-streaming.rst b/toolkit/components/telemetry/docs/obsolete/geckoview-streaming.rst new file mode 100644 index 0000000000..10dfcdf860 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/geckoview-streaming.rst @@ -0,0 +1,26 @@ +GeckoView Streaming API +======================= + +As an alternative to the normal mode where Firefox Desktop records and sends data, +Telemetry can instead route Histogram samples and Scalar values out of Gecko to a Telemetry Delegate. + +To do this, ``toolkit.telemetry.geckoview.streaming`` must be set to true, +and Gecko must have been built with ``MOZ_WIDGET_ANDROID`` defined. + +See :doc:`this guide <../start/report-gecko-telemetry-in-glean>` +for how to collect data in this mode. + +Details +======= + +Samples accumulated on Histograms and values set +(``ScalarAdd`` and ``ScalarSetMaximum`` operations are not supported) +on Scalars that have ``products`` lists that include ``geckoview_streaming`` +will be redirected to a small batching service in +``toolkit/components/telemetry/geckoview/streaming``. +The batching service +(essentially just tables of histogram/scalar names to lists of samples/values) +will hold on to these lists of samples/values paired to the histogram/scalar names for a length of time +(``toolkit.telemetry.geckoview.batchDurationMS`` (default 5000)) +after which the next accumulation or ``ScalarSet`` will trigger the whole batch +(all lists) to be passed over to the ``StreamingTelemetryDelegate``. diff --git a/toolkit/components/telemetry/docs/obsolete/hybrid-content.rst b/toolkit/components/telemetry/docs/obsolete/hybrid-content.rst new file mode 100644 index 0000000000..9eb0b00341 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/hybrid-content.rst @@ -0,0 +1,374 @@ +=================================== +Hybrid Content Telemetry (obsolete) +=================================== + +Hybrid content is web content that is loaded as part of Firefox, appears as part of +Firefox to the user and is primarily intended to be used in Firefox. This can be +either a page that ships with Firefox or that can be loaded dynamically from our hosted +services. Hybrid content telemetry allows Mozilla pages to check whether data +collection is enabled and to submit Telemetry data. + +.. important:: + + Every new or changed data collection in Firefox (including hybrid content) needs a `data collection review <https://wiki.mozilla.org/Firefox/Data_Collection>`__ from a Data Steward. + +The recorded data will be sent to Mozilla servers by Firefox, if the collection is enabled, with the :doc:`main-ping <../data/main-ping>`. + +Adding content data collection +============================== +Telemetry can be sent from web content by: + +1. granting the web content's host privileges in the Firefox codebase; +2. including the ``HybridContentTelemetry-lib.js`` file in the page; +3. registering the probes after the library is loaded; +4. using the API to send Telemetry. + +Granting the privileges +----------------------- +For security/privacy reasons `Mozilla.ContentTelemetry` will only work on a list of allowed secure origins. +The list of allowed origins can be found in +`browser/app/permissions <https://searchfox.org/mozilla-central/source/browser/app/permissions>`_ . +A host needs to be given the ``hc_telemetry`` permission in order to be allowed to use the API. + +Example: + +:: + + origin hc_telemetry 1 https://discovery.addons.mozilla.org + +Adding an entry to the ``permissions`` file requires riding the trains. If "go-faster" content requires +granting permissions to a Mozilla page, it can do so by using the `permission manager <https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIPermissionManager>`_ + +.. code-block:: js + + function addonInit() { + // The following code must be called before attempting to load a page that uses + // hybrid content telemetry on https://example.mozilla.org. + let principal = Services.scriptSecurityManager.createContentPrincipalFromOrigin("https://example.mozilla.org"); + Services.perms.addFromPrincipal(principal, "hc_telemetry", Services.perms.ALLOW_ACTION); + } + + function addonCleanup() { + // The permission must be removed if no longer needed (e.g. the add-on is shut down). + let principal = Services.scriptSecurityManager.createContentPrincipalFromOrigin("https://example.mozilla.org"); + Services.perms.removeFromPrincipal(principal, "hc_telemetry"); + } + +.. important:: + + Granted permissions do not disappear when a "go-faster" add-on is uninstalled but are cleared when the browser is closed. If permissions need to be cleaned without closing the browser, it must be done manually. Moreover, permissions are keyed by origin: ``http://mozilla.com`` and ``https://mozilla.com`` are different things. + +Including the library +--------------------- +To use hybrid content telemetry the related content JS library needs to be included in the page. We have different integration options: + +* Add ``mozilla-hybrid-content-telemetry`` as a dependency to the project and require it in the code. +* Load it directly from the `external unpkg CDN <https://unpkg.com/mozilla-hybrid-content-telemetry/HybridContentTelemetry-lib.js>`_. +* Manually fetch the latest version from the `main repository <https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/hybrid-content/HybridContentTelemetry-lib.js>`_ and add it to the page repository. Then this file can be deployed along with the page. + +Example (manual inclusion): +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: html + + <!DOCTYPE html> + <html> + <head> + <!-- Other head stuff --> + <script type="application/javascript" src="HybridContentTelemetry-lib.js"></script> + </head> + <body> <!-- Other body stuff --> </body> + </html> + +Example (NPM dependency): +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add the dependency to your project: + +.. code-block:: shell + + npm install --save mozilla-hybrid-content-telemetry@1.0.0 + +In your app load the module and use the :ref:`API <the-api>`: + +.. code-block:: js + + const ContentTelemetry = require("mozilla-hybrid-content-telemetry"); + + ContentTelemetry.registerEvents("page.interaction", { + "click": { + methods: ["click"], + objects: ["red_button", "blue_button"], + } + }); + + // Now events can be recorded. + ContentTelemetry.recordEvent("page.interaction", "click", "red_button"); + +.. note:: + + The following examples assume the manual inclusion of the JS library. + +Registering the probes +---------------------- +Probe registration can happen at any time after the library is loaded in the page, but registering early enough ensures that the definition is available once a recording attempt is made. + +Example: + +.. code-block:: html + + <!DOCTYPE html> + <html> + <head> + <!-- Other head stuff --> + <script type="application/javascript"> + window.onload = function() { + if (!Mozilla || !Mozilla.ContentTelemetry) { + // .. uh-oh, was library loaded? Report the error. + return; + } + // Register the probe. + Mozilla.ContentTelemetry.registerEvents("page.interaction", { + "click": { + methods: ["click"], + objects: ["red_button", "blue_button"], + } + }); + }; + </script> + </head> + <body> <!-- Other body stuff --> </body> + </html> + +Recording the data +------------------ +Data recording can happen at any time after a probe has been registered. The data will be recorded and sent by Firefox if permitted by the Telemetry :doc:`preferences <../internals/preferences>`. + +Example: + +.. code-block:: html + + <!DOCTYPE html> + <html> + <head> + <!-- Other head stuff --> + <script type="application/javascript"> + function triggerEvent() { + if (!Mozilla || !Mozilla.ContentTelemetry) { + // .. uh-oh, was library loaded? Report the error. + return; + } + Mozilla.ContentTelemetry.recordEvent("page.interaction", "click", "red_button"); + }; + </script> + </head> + <body> + <!-- Other body stuff --> + <div id="content"> + <button id='event-recording' onclick="triggerEvent();"> + Trigger Recording + </button> + </div> + </body> + </html> + +Checking if upload is enabled +----------------------------- +Mozilla pages can check if data upload is enabled, as reported by Telemetry :doc:`preferences <../internals/preferences>`. This is useful for pages which are not using Telemetry to collect data, but +need to comply to our data policy for the collection. + +Example: + +.. code-block:: html + + <!DOCTYPE html> + <html> + <head> + <!-- Other head stuff --> + <script type="application/javascript"> + function recordData() { + if (!Mozilla || !Mozilla.ContentTelemetry) { + // .. uh-oh, was library loaded? Report the error. + return; + } + + if (!Mozilla.ContentTelemetry.canUpload()) { + // User has opted-out of Telemetry. No collection must take place. + return; + } + + // ... perform the collection without Telemetry below this point. + }; + </script> + </head> + <body> + <!-- Other body stuff --> + <div id="content"> + <button id='event-recording' onclick="recordData();"> + Trigger Recording + </button> + </div> + </body> + </html> + + +.. _the-api: + +The API +======= +The hybrid content API is available to the web content through the inclusion of the `HybridContentTelemetry-lib.js <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/hybrid-content/HybridContentTelemetry-lib.js>`_ library. + +The initial implementation of the API allows the registration and the recording of events. + +JS API +------ +Authorized content can use the following functions: + +.. code-block:: js + + Mozilla.ContentTelemetry.canUpload(); + Mozilla.ContentTelemetry.initPromise(); + Mozilla.ContentTelemetry.registerEvents(category, eventData); + Mozilla.ContentTelemetry.recordEvent(category, method, object, value, extra); + +These functions will not throw. If an unsupported operation is performed (e.g. recording an unknown event) an error will be logged to the browser console. + +.. note:: + + Data collected using this API will always respect the user Telemetry preferences: if a user has chosen to not send Telemetry data to Mozilla servers, Telemetry from hybrid content pages will not be sent either. + Like other Telemetry data, it will still be recorded locally and available through ``about:telemetry``. + +``Mozilla.ContentTelemetry.canUpload()`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: js + + Mozilla.ContentTelemetry.canUpload(); + +This function returns true if the browser is allowed to send collected data to Mozilla servers (i.e. ``datareporting.healthreport.uploadEnabled`` is ``true``), false otherwise. See :doc:`preferences <../internals/preferences>`. + +.. note:: + + The page should use this function to check if it is allowed to collect data. This is only needed in case the Telemetry system is not be being used for collection. If Telemetry is used, then this is taken care of internally by the Telemetry API. The page should not cache the returned value: users can opt in or out from the Data Collection at any time and so the returned value may change. + +Example: + +.. code-block:: js + + if (Mozilla.ContentTelemetry.canUpload()) { + // ... perform the data collection here using another measurement system. + } + +``Mozilla.ContentTelemetry.initPromise()`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: js + + Mozilla.ContentTelemetry.initPromise(); + +This function returns a Promise that gets resolved as soon as Hybrid Content Telemetry is correctly initialized and the value from ``canUpload`` can be reliably read. The promise will reject if Hybrid Content Telemetry is disabled or the host doesn't have enough privileges to use the API. + +``Mozilla.ContentTelemetry.registerEvents()`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: js + + Mozilla.ContentTelemetry.registerEvents(category, eventData); + +Register new dynamic events from the content. This accepts the same parameters and is subject to the same limitation as ``Services.telemetry.registerEvents()``. See the `events` documentation for the definitive reference. + +.. note:: + + Make sure to call this before recording events, as soon as the library is loaded (e.g. `window load event <https://developer.mozilla.org/en-US/docs/Web/Events/load>`_). This will make sure that the definition will be ready when recording. + +The data recorded into events registered with this function will end up in the ``dynamic`` process section of the main ping. + +Example: + +.. code-block:: js + + Mozilla.ContentTelemetry.registerEvents("page.interaction", { + "click": { + methods: ["click"], + objects: ["red_button", "blue_button"], + } + }); + // Now events can be recorded. + Mozilla.ContentTelemetry.recordEvent("page.interaction", "click", "red_button"); + +``Mozilla.ContentTelemetry.recordEvent()`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: js + + Mozilla.ContentTelemetry.recordEvent(category, method, object, value, extra); + +Record a registered event. This accepts the same parameters and is subject to the same limitation as ``Services.telemetry.recordEvent()``. See the `events` documentation for the definitive reference. + +Example: + +.. code-block:: js + + Mozilla.ContentTelemetry.recordEvent("ui", "click", "reload-btn"); + // event: [543345, "ui", "click", "reload-btn"] + Mozilla.ContentTelemetry.recordEvent("ui", "search", "search-bar", "google"); + // event: [89438, "ui", "search", "search-bar", "google"] + Mozilla.ContentTelemetry.recordEvent("ui", "completion", "search-bar", "yahoo", + {"querylen": "7", "results": "23"}); + // event: [982134, "ui", "completion", "search-bar", "yahoo", + // {"qerylen": "7", "results": "23"}] + +Data Review +=========== + +Adding the ``hc_telemetry`` permission for a new domain in `browser/app/permissions <https://searchfox.org/mozilla-central/source/browser/app/permissions>`_ +requires `Data Collection Review <https://wiki.mozilla.org/Firefox/Data_Collection>`_ as we are enabling a new method of data collection. + +Giving a domain permission to use Hybrid Content Telemetry also gives any Extensions running on this domain permission to use Hybrid Content Telemetry. +If the domain is already on the `list of restricted domains <https://hg.mozilla.org/integration/mozilla-inbound/file/39e131181d44/modules/libpref/init/all.js#l5120>`_ +(configured by the ``extensions.webextensions.restrictedDomains`` preference), Extensions don't run on this domain and therefore cannot access the Hybrid Content Telemetry API. +No additional approval is necessary. + +If the domain is not on that list, you need additional privacy review. In that case request help from the Telemetry team. + +Testing +======= + +In order to test Hybrid Content Telemetry integrations, the permission API can be used to enable certain hosts. +The ``Services.perms.addFromPrincipal`` API is available in the Browser Console as well as in ``xpcshell`` and ``mochi`` tests with access to the ``Services.*`` APIs. + +The respective ``hc_telemetry`` permission needs to be set before any pages on that host load the ``HybridContentTelemetry-lib.js`` file. + +Manual testing +-------------- + +After starting the browser, open the Browser Console (Tools -> Web Developer -> Browser Console). +To enable Hybrid Content Telemetry on ``https://example.mozilla.org``, execute this code snippet in the console: + +.. code-block:: js + + let principal = Services.scriptSecurityManager.createContentPrincipalFromOrigin("https://example.mozilla.org"); + Services.perms.addFromPrincipal(principal, "hc_telemetry", Services.perms.ALLOW_ACTION); + +Afterwards load the page on ``https://example.mozilla.org`` and it will be able to record Telemetry data. + +.. note:: + + Manual testing requires a host that handles HTTPS connections, as this kind of collection is only allowed on secure hosts. To allow for local testing, a local proxy capable of handling HTTPS connection is required. + +Automated testing +----------------- + +In test frameworks with privileged access the permission can be set in the ``head.js`` or during test setup. +Add the code snippet in your ``head.js`` to enable Hybrid Content ContentTelemetry on ``https://example.mozilla.org``: + +.. code-block:: js + + let principal = Services.scriptSecurityManager.createContentPrincipalFromOrigin("https://example.mozilla.org"); + Services.perms.addFromPrincipal(principal, "hc_telemetry", Services.perms.ALLOW_ACTION); + +Version History +=============== + +- Firefox 59: Initial hybrid content telemetry support (`bug 1417473 <https://bugzilla.mozilla.org/show_bug.cgi?id=1417473>`_). +- Firefox 71: Hybrid Content Telemetry removed (`bug 1520491 <https://bugzilla.mozilla.org/show_bug.cgi?id=1520491>`_). diff --git a/toolkit/components/telemetry/docs/obsolete/index.rst b/toolkit/components/telemetry/docs/obsolete/index.rst new file mode 100644 index 0000000000..fc2acf7615 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/index.rst @@ -0,0 +1,15 @@ +.. _telemetry_obsolete: + +====================== +Obsolete Documentation +====================== + +.. toctree:: + :glob: + :maxdepth: 5 + :titlesonly: + + uitelemetry/index + fhr/index + hybrid-content + * diff --git a/toolkit/components/telemetry/docs/obsolete/optout-ping.rst b/toolkit/components/telemetry/docs/obsolete/optout-ping.rst new file mode 100644 index 0000000000..910bb41220 --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/optout-ping.rst @@ -0,0 +1,33 @@ + +"optout" ping (obsolete) +======================== + +This ping is generated when a user turns off FHR upload from the Preferences panel, changing the related ``datareporting.healthreport.uploadEnabled`` :doc:`preference <../internals/preferences>`. + +This ping contains no client id and no environment data. + +Structure: + +.. code-block:: js + + { + version: 4, + type: "optout", + ... common ping data + payload: { } + } + +Expected behaviours +------------------- +The following is a list of expected behaviours for the ``optout`` ping: + +- Sending the "optout" ping is best-effort. Telemetry tries to send the ping once and discards it immediately if sending fails. +- The ping might be delayed if ping sending is throttled (e.g. around midnight). +- The ping can be lost if the browser is shutdown before the ping is sent out the "optout" ping is discarded. + +Version History +--------------- + +- Firefox 63: + + - "optout" ping replaced the "deletion" ping (`bug 1445921 <https://bugzilla.mozilla.org/show_bug.cgi?id=1445921>`_). diff --git a/toolkit/components/telemetry/docs/obsolete/uitelemetry/index.rst b/toolkit/components/telemetry/docs/obsolete/uitelemetry/index.rst new file mode 100644 index 0000000000..89447cc9eb --- /dev/null +++ b/toolkit/components/telemetry/docs/obsolete/uitelemetry/index.rst @@ -0,0 +1,146 @@ +.. _uitelemetry: + +================================== +UITelemetry data format (obsolete) +================================== + +.. note:: + + ``UITelemetry`` is deprecated. As of Firefox 61, ``UITelemetry`` is no longer reported. + +UI Telemetry sends its data as a JSON blob. This document describes the different parts +of the JSON blob. + +``toolbars`` +============ + +This tracks the state of the user's UI customizations. It has the following properties: + +- ``sizemode`` - string indicating whether the window is in maximized, normal (restored) or + fullscreen mode; +- ``bookmarksBarEnabled`` - boolean indicating whether the bookmarks bar is visible; +- ``menuBarEnabled`` - boolean indicating whether the menu bar is visible (always false on OS X); +- ``titleBarEnabled`` - boolean indicating whether the (real) titlebar is visible (rather than + having tabs in the titlebar); +- ``defaultKept`` - list of strings identifying toolbar buttons and items that are still in their + default position. Only the IDs of builtin widgets are sent (ie not add-on widgets); +- ``defaultMoved`` - list of strings identifying toolbar buttons and items that are no longer in + their default position, but have not been removed to the palette. Only the IDs of builtin widgets + are sent (ie not add-on widgets); +- ``nondefaultAdded`` - list of strings identifying toolbar buttons and items that have been added + from the palette. Only the IDs of builtin widgets are sent (ie not add-on widgets); +- ``defaultRemoved`` - list of strings identifying toolbar buttons and items that are in the + palette that are elsewhere by default. Only the IDs of builtin widgets are sent + (ie not add-on widgets); +- ``addonToolbars`` - the number of non-default toolbars that are customizable. 1 by default + because it counts the add-on bar shim; +- ``visibleTabs`` - array of the number of visible tabs per window; +- ``hiddenTabs`` - array of the number of hidden tabs per window (ie tabs in panorama groups which + are not the current group); +- ``countableEvents`` - please refer to the next section. +- ``durations`` - an object mapping descriptions to duration records, which records the amount of + time a user spent doing something. Currently only has one property: + + - ``customization`` - how long a user spent customizing the browser. This is an array of + objects, where each object has a ``duration`` property indicating the time in milliseconds, + and a ``bucket`` property indicating a bucket in which the duration info falls. + + +.. _UITelemetry_countableEvents: + +``countableEvents`` +=================== + +Countable events are stored under the ``toolbars`` section. They count the number of times certain +events happen. No timing or other correlating information is stored - purely the number of times +things happen. + +``countableEvents`` contains a list of buckets as its properties. A bucket represents the state the browser was in when these events occurred, such as currently running an interactive tour. There are 3 types of buckets: + +- ``__DEFAULT__`` - No bucket, for times when the browser is not in any special state. +- ``bucket_<NAME>`` - Normal buckets, for when the browser is in a special state. The ``<NAME>`` in the bucket ID is the name associated with the bucket and may be further broken down into parts by the ``|`` character. +- ``bucket_<NAME>|<INTERVAL>`` - Expiring buckets, which are similar to a countdown timer. The ``<INTERVAL>`` in the bucket ID describes the time interval the recorded event happened in. The intervals are ``1m`` (one minute), ``3m`` (three minutes), ``10m`` (ten minutes), and ``1h`` (one hour). After one hour, the ``__DEFAULT__`` bucket is automatically used again. + +Each bucket is an object with the following properties: + +- ``click-builtin-item`` is an object tracking clicks on builtin customizable toolbar items, keyed + off the item IDs, with an object for each item with keys ``left``, ``middle`` and ``right`` each + storing a number indicating how often the respective type of click has happened. +- ``click-menu-button`` is the same, except the item ID is always 'button'. +- ``click-bookmarks-bar`` is the same, with the item IDs being replaced by either ``container`` for + clicks on bookmark or livemark folders, and ``item`` for individual bookmarks. +- ``click-menubar`` is similar, with the item IDs being replaced by one of ``menu``, ``menuitem`` + or ``other``, depending on the kind of item clicked. Note that this is not tracked on OS X, where + we can't listen for these events because of the global menubar. +- ``click-bookmarks-menu-button`` is also similar, with the item IDs being replaced by: + + - ``menu`` for clicks on the 'menu' part of the item; + - ``add`` for clicks that add a bookmark; + - ``edit`` for clicks that open the panel to edit an existing bookmark; + - ``in-panel`` for clicks when the button is in the menu panel, and clicking it does none of the + above; +- ``customize`` tracks different types of customization events without the ``left``, ``middle`` and + ``right`` distinctions. The different events are the following, with each storing a count of the + number of times they occurred: + + - ``start`` counts the number of times the user starts customizing; + - ``add`` counts the number of times an item is added somewhere from the palette; + - ``move`` counts the number of times an item is moved somewhere else (but not to the palette); + - ``remove`` counts the number of times an item is removed to the palette; + - ``reset`` counts the number of times the 'restore defaults' button is used; +- ``search`` is an object tracking searches of various types, keyed off the search + location, storing a number indicating how often the respective type of search + has happened. + + - There are also two special keys that mean slightly different things. + + - ``urlbar-keyword`` records searches that would have been an invalid-protocol + error, but are now keyword searches. They are also counted in the ``urlbar`` + keyword (along with all the other urlbar searches). + - ``selection`` searches records selections of search suggestions. They include + the source, the index of the selection, and the kind of selection (mouse or + enter key). Selection searches are also counted in their sources. + + + +``UITour`` +========== + +The UITour API provides ways for pages on trusted domains to safely interact with the browser UI and request it to perform actions such as opening menus and showing highlights over the browser chrome - for the purposes of interactive tours. We track some usage of this API via the ``UITour`` object in the UI Telemetry output. + +Each page is able to register itself with an identifier, a ``Page ID``. A list of Page IDs that have been seen over the last 8 weeks is available via ``seenPageIDs``. + +Page IDs are also used to identify buckets for :ref:`UITelemetry_countableEvents`, in the following circumstances: + +- The current tab is a tour page. This will be a normal bucket with the name ``UITour|<PAGEID>``, where ``<PAGEID>`` is the page's registered ID. This will result in bucket IDs such as ``bucket_UITour|australis-tour``. +- A tour tab is open but another tab is active. This will be an expiring bucket with the name ``UITour|<PAGEID>|inactive``. This will result in bucket IDs such as ``bucket_UITour|australis-tour|inactive|1m``. +- A tour tab has recently been open but has been closed. This will be an expiring bucket with the name ``UITour|<PAGEID>|closed``. This will result in bucket IDs such as ``bucket_UITour|australis-tour|closed|10m``. + + + +``contextmenu`` +=============== + +We track context menu interactions to figure out which ones are most often used and/or how +effective they are. In the ``contextmenu`` object, we first store things per-bucket. Next, we +divide the following different context menu situations: + +- ``selection`` if there is content on the page that's selected on which the user clicks; +- ``link`` if the user opened the context menu for a link +- ``image-link`` if the user opened the context menu on an image or canvas that's a link; +- ``image`` if the user opened the context menu on an image (that isn't a link); +- ``canvas`` if the user opened the context menu on a canvas (that isn't a link); +- ``media`` if the user opened the context menu on an HTML video or audio element; +- ``input`` if the user opened the context menu on a text input element; +- ``other`` for all other openings of the content menu; + +Each of these objects (if they exist) then gets a "withcustom" and/or a "withoutcustom" property +for context menus opened with custom page-created items and without them, and each of those +properties holds an object with IDs corresponding to a count of how often an item with that ID was +activated in the context menu. Only builtin context menu items are tracked, and besides those items +there are four special items which get counts: + +- ``close-without-interaction`` is incremented when the user closes the context menu without interacting with it; +- ``custom-page-item`` is incremented when the user clicks an item that was created by the page; +- ``unknown`` is incremented when an item without an ID was clicked; +- ``other-item`` is incremented when an add-on-provided menuitem is clicked. |