Adding upstream version 115.7.0esr.upstream/115.7.0esr

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

3 files changed, 217 insertions, 0 deletions

diff --git a/toolkit/components/telemetry/docs/start/adding-a-new-probe.rst b/toolkit/components/telemetry/docs/start/adding-a-new-probe.rst
new file mode 100644
index 0000000000..f1baeadf82
--- /dev/null
+++ b/toolkit/components/telemetry/docs/start/adding-a-new-probe.rst
@@ -0,0 +1,153 @@
+============================
+Adding a new Telemetry probe
+============================
+
+In Firefox, the Telemetry system collects various measures of Firefox performance, hardware, usage and customizations and submit it to Mozilla. This article provides an overview of what is needed to add any new Telemetry data collection.
+
+.. important::
+
+    Every new data collection in Firefox needs a `data collection review <https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval>`__ from a data collection peer. Just set the feedback? flag for one of the data peers. They try to reply within a business day.
+
+What is your goal?
+==================
+
+We have various :doc:`data collection tools <../collection/index>` available, each serving different needs. Before diving right into technical details, it is best to take a step back and consider what you need to achieve.
+
+Your goal could be to answer product questions like “how many people use feature X?” or “what is the error rate of service Y?”.
+You could also be focused more on answering engineering questions, say “which web features are most used?” or “how is the performance of engine Z?”.
+
+From there, questions you should ask are:
+
+- What is the minimum data that can answer your questions?
+- How many people do you need this data from?
+- Is data from the pre-release channels sufficient?
+
+This also informs the `data collection review <https://wiki.mozilla.org/Firefox/Data_Collection>`__, which requires a plan for how to use the data. Data collection review is required for all new data collection.
+
+Data collection levels
+======================
+
+Most of our data collection falls into one of two levels, *release* and *pre-release*.
+
+**Release data** is recorded by default on all channels, users need to explicitly opt out to disable it. This has `stricter constraints <https://wiki.mozilla.org/Firefox/Data_Collection#Requirements>`_ for what data we can collect. "Most" users submit this data.
+
+**Pre-release data** is not recorded on release, but is collected by default on our pre-release channels (Beta and Nightly), so it can be biased.
+
+These levels cover what is described in the `Firefox privacy notice <https://www.mozilla.org/en-US/privacy/firefox/>`_. For other needs, there might be custom mechanisms that clearly require user opt-in and show what data is collected.
+
+Rich data & aggregate data
+==========================
+
+For the recording and transmission of data, we have various data types available. We can divide these data types into two large groups.
+
+**Aggregate data** is aggregated on the client-side and cheap to send, process and analyze. This could e.g. be a simple count of tab opens or a histogram showing how long it takes to switch between tabs. This should be your default choice and is well supported in our analysis tools.
+
+**Rich data** is used when questions can not be answered from aggregate data. When we send more detailed data we can e.g. see when a specific UI interaction happened and in which context.
+
+As a general rule, you can inform the choice of data types from your goals like this:
+
++------------------------+-----------------+-----------------------+
+| Goals                  | Collection type | Implementation        |
++========================+=================+=======================+
+| On-going monitoring    | Aggregate data  | Histograms            |
+|                        |                 |                       |
+| Health tracking        |                 | Scalars               |
+|                        |                 |                       |
+| KPI impact             |                 | Environment data      |
++------------------------+-----------------+-----------------------+
+| Detailed user behavior | Rich data       | Event Telemetry       |
+|                        |                 |                       |
+| Funnel analysis        |                 | Detailed custom pings |
+|                        |                 |                       |
+| Diagnostics            |                 | Logs                  |
+|                        |                 |                       |
+|                        |                 | Crash data            |
++------------------------+-----------------+-----------------------+
+
+Aggregate data
+--------------
+
+Most of our data collection happens through :doc:`scalars <../collection/scalars>` and :doc:`histograms <../collection/histograms>`:
+
+- Scalars allow collection of simple values, like counts, booleans and strings.
+- Histograms allow collection of multiple different values, but aggregate them into a number of buckets. Each bucket has a value range and a count of how many values we recorded.
+
+Both scalars & histograms allow recording by keys. This allows for more flexible, two-level data collection.
+
+Other collections can build on top of scalars & histograms. An example is :doc:`use counters <../collection/use-counters>`, which submit web feature usage through histograms.
+
+We also collect :doc:`environment data <../data/environment>`. This consists of mostly scalar values that capture the “working environment” a Firefox session lives in, and includes e.g. data on hardware, OS, add-ons and some settings. Any data that is part of the "working environment", or needs to split :doc:`subsessions <../concepts/sessions>`, should go into it.
+
+Rich data
+---------
+
+Aggregate data can tell you that something happened, but is usually lacking details about what exactly. When more details are needed, we can collect them using other tools that submit less efficient data. This usually means that we can't enable the data collection for all users, for cost and performance concerns.
+
+There are multiple mechanisms to collect rich data:
+
+**Stack collection** helps with e.g. diagnosing hangs. Stack data is recorded into chrome hangs and threadhang stats. To diagnose where rarely used code is called from, you can use stack capturing.
+
+:doc:`Event Telemetry <../collection/events>` provides a way to record both when and what happened. This enables e.g. funnel analysis for usage.
+
+:doc:`Custom pings <../collection/custom-pings>` are used when other existing data collection does not cover your need. Submitting a custom ping enables you to submit your own JSON package that will be delivered to the Telemetry servers. However, this loses you access to existing tooling and makes it harder to join your data with other sources.
+
+Setup & building
+================
+
+Every build of Firefox has Telemetry enabled. Local developer builds with no custom build flags will record all Telemetry data, but not send it out.
+
+When adding any new scalar, histogram or event Firefox needs to be built. Artifact builds are currently not supported, even if code changes are limited to JavaScript.
+
+Usually you don't need to send out data to add new Telemetry. In the rare event you do, you need the following in your *.mozconfig*::
+
+   MOZ_TELEMETRY_REPORTING=1
+   MOZILLA_OFFICIAL=1
+
+Testing
+=======
+
+Local confirmation
+------------------
+
+Your first step should always be to confirm your new data collection locally.
+
+The *about:telemetry* page allows to view any data you submitted to Telemetry in the last 60 days, whether it is in existing pings or in new custom pings. You can choose which pings to display on the top-left.
+
+If you need to confirm when - or if - pings are getting sent, you can run an instance of the `gzipServer <https://github.com/mozilla/gzipServer>`_ locally. It emulates roughly how the official Telemetry servers respond, and saves all received pings to disk for inspection.
+
+Test coverage
+-------------
+
+Any data collection that you need to base decisions on needs to have test coverage. Using JS, you can access the recorded values for your data collection. You can use the following functions:
+
+- for scalars, `getSnapshotForScalars() <https://searchfox.org/mozilla-central/rev/f997bd6bbfc4773e774fdb6cd010142370d186f9/toolkit/components/telemetry/core/nsITelemetry.idl#90-102>`_
+  or `getSnapshotForKeyedScalars() <https://searchfox.org/mozilla-central/rev/f997bd6bbfc4773e774fdb6cd010142370d186f9/toolkit/components/telemetry/core/nsITelemetry.idl#104-116>`_
+- for histograms, `getSnapshotForHistograms() <https://searchfox.org/mozilla-central/rev/f997bd6bbfc4773e774fdb6cd010142370d186f9/toolkit/components/telemetry/core/nsITelemetry.idl#54-74>`_
+  or `getSnapshotForKeyedHistograms() <https://searchfox.org/mozilla-central/rev/f997bd6bbfc4773e774fdb6cd010142370d186f9/toolkit/components/telemetry/core/nsITelemetry.idl#76-88>`_
+
+  * Optionally, histogram objects have a `snapshot() <https://searchfox.org/mozilla-central/rev/f997bd6bbfc4773e774fdb6cd010142370d186f9/toolkit/components/telemetry/core/nsITelemetry.idl#285-287,313-315>`_ method.
+
+- for events, `snapshotEvents() <https://searchfox.org/mozilla-central/rev/f997bd6bbfc4773e774fdb6cd010142370d186f9/toolkit/components/telemetry/core/nsITelemetry.idl#542-558>`_
+
+If you need to test that pings were correctly passed to Telemetry, you can use `TelemetryArchiveTesting <https://searchfox.org/mozilla-central/search?q=TelemetryArchiveTesting&redirect=false>`_.
+
+Validation
+----------
+
+While it's important to confirm that the data collection works on your machine, the Firefox user population is very diverse. Before basing decisions on any new data, it should be validated. This could take various forms.
+
+For *new data collection* using existing Telemetry data types, the transport mechanism is already tested. It is sufficient to validate the incoming values. This could happen through `Redash <https://docs.telemetry.mozilla.org/tools/stmo.html>`_ or through `custom analysis <https://docs.telemetry.mozilla.org/tools/spark.html>`_.
+
+For *new custom pings*, you'll want to check schema validation results, as well as that the contents look valid.
+
+Getting help
+============
+
+You can find all important Telemetry resources listed on `telemetry.mozilla.org <https://telemetry.mozilla.org/>`_.
+
+The Telemetry team is there to help with any problems. You can reach us via:
+
+- Matrix in `#telemetry:mozilla.org <https://chat.mozilla.org/#/room/#telemetry:mozilla.org>`_
+- Slack in `#data-help <https://mozilla.slack.com/messages/data-help/>`_
+- the `fx-data-dev mailing list <https://mail.mozilla.org/listinfo/fx-data-dev>`_
+- flags for `one of the peers <https://wiki.mozilla.org/Modules/Toolkit#Telemetry>`_ on Bugzilla or send us an e-mail
diff --git a/toolkit/components/telemetry/docs/start/index.rst b/toolkit/components/telemetry/docs/start/index.rst
new file mode 100644
index 0000000000..9cc544c9a3
--- /dev/null
+++ b/toolkit/components/telemetry/docs/start/index.rst
@@ -0,0 +1,28 @@
+===============
+Getting started
+===============
+
+If you are interested in extending data collection by adding new probes have a look at
+
+.. toctree::
+   :maxdepth: 2
+   :titlesonly:
+   :glob:
+
+   adding-a-new-probe
+   report-gecko-telemetry-in-glean
+
+If you want to work with the telemetry code itself, for example to fix a bug, it is often helpful to start with these steps:
+
+1. Have a look at about:telemetry to see which data is being collected and sent.
+2. Increase the log level in about:config by setting toolkit.telemetry.log.level to Debug or Trace. This will show telemetry information in the browser console. To enable the browser console follow `these instructions <../../../../devtools-user/browser_console/index.html>`__.
+3. Run a local telemetry receiver, e.g. `this one <https://github.com/mozilla/gzipServer>`__ and set ``toolkit.telemetry.server`` to “localhost” (Like the next preference this needs a restart.)
+4. Set ``toolkit.telemetry.send.overrideOfficialCheck = true``, otherwise local debug builds will not send telemetry data. (Requires restart.)
+
+More information about the internals can be found `here <../internals/index.html>`__.
+
+Further Reading
+###############
+
+* `Telemetry Portal <https://telemetry.mozilla.org/>`_ - Discover all important resources for working with data
+* `Telemetry Data Documentation <https://docs.telemetry.mozilla.org/>`_ - Find what data is available & how to use it
diff --git a/toolkit/components/telemetry/docs/start/report-gecko-telemetry-in-glean.rst b/toolkit/components/telemetry/docs/start/report-gecko-telemetry-in-glean.rst
new file mode 100644
index 0000000000..1e7cb4aca5
--- /dev/null
+++ b/toolkit/components/telemetry/docs/start/report-gecko-telemetry-in-glean.rst
@@ -0,0 +1,36 @@
+=======================================================
+How to report Gecko Telemetry in engine-gecko via Glean
+=======================================================
+
+In Gecko, the `Telemetry <../index.html>`__ system collects various measures of Gecko performance, hardware, usage and customizations.
+When the Gecko engine is embedded in Android products that use
+`the Glean SDK <https://docs.telemetry.mozilla.org/concepts/glean/glean.html>`__ for data collection, then Gecko metrics can be reported in `Glean pings <https://mozilla.github.io/glean/book/user/pings/index.html>`__.
+This article provides an overview of what is needed to report any existing or new Telemetry data collection in Glean.
+
+.. important::
+
+    Every new or changed data collection in Firefox needs a `data collection review <https://wiki.mozilla.org/Data_Collection>`__ from a Data Steward.
+
+Using Glean
+===========
+In short, the way to collect data in all Mozilla products using Gecko is to use Glean.
+The Glean SDK is present in Gecko and all of Firefox Desktop via
+`Firefox on Glean (FOG) <../../glean/index.html>`__.
+If you instrument your Gecko data collection using Glean metrics,
+they will be reported by all products that use both Gecko and Glean so long as it is defined in a
+``metrics.yaml`` file in the ``gecko_metrics`` list of ``toolkit/components/glean/metrics_index.py``.
+
+See `these docs <../../glean/user/new_definitions_file.html>`__ for details.
+Additional relevant concepts are covered in
+`this guide for migrating Firefox Telemetry collections to Glean <../../glean/user/migration.html>`__.
+
+To make sure your data is also reported in Telemetry when your code is executed in Firefox Desktop,
+you must be sure to mirror the collection in Telemetry.
+If possible, you should use the
+`Glean Interface For Firefox Telemetry (GIFFT) <../../glean/user/gifft.html>`__
+and let FOG mirror the call for you.
+If that doesn't look like it'll work out for you,
+you can instead add your Glean API calls just next to your Telemetry API calls.
+
+As always, if you have any questions, you can find helpful people in
+`the #telemetry channel on Mozilla's Matrix <https://chat.mozilla.org/#/room/#telemetry:mozilla.org>`__.