summaryrefslogtreecommitdiffstats
path: root/toolkit/components/telemetry/docs/data/main-ping.rst
blob: 23f7d8c600c40d9a5977d663605f65d9c1158d3f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
"main" ping
===========

.. toctree::
   :maxdepth: 2

This is the "main" Telemetry ping type, whose payload contains most of the measurements that are used to track the performance and health of Firefox in the wild.
It includes histograms and other performance and diagnostic data.

This ping may be triggered for one of many reasons documented by the ``reason`` field:

* ``aborted-session`` - this ping is regularly saved to disk (every 5 minutes), overwriting itself, and deleted at shutdown. If a previous aborted session ping is found at startup, it gets sent to the server. The first aborted-session ping is generated as soon as Telemetry starts
* ``environment-change`` - the :doc:`environment` changed, so the session measurements got reset and a new subsession starts
* ``shutdown`` - triggered when the browser session ends. For the first browsing session, this ping is saved to disk and sent on the next browser restart. From the second browsing session on, this ping is sent immediately on shutdown using the :doc:`../internals/pingsender`, unless the OS is shutting down
* ``daily`` - a session split triggered in 24h hour intervals at local midnight. If an ``environment-change`` ping is generated by the time it should be sent, the daily ping is rescheduled for the next midnight
* ``saved-session`` - the *"classic"* Telemetry payload with measurements covering the whole browser session (only submitted on Android)

.. _sessionsplit:

Most reasons lead to a session split, initiating a new *subsession*. We reset important measurements for those subsessions.

After a new subsession split, the ``internal-telemetry-after-subsession-split`` topic is notified to all the observers. *This is an internal topic and is only meant for internal Telemetry usage.*

.. note::

  ``saved-session`` is sent with a different ping type (``saved-session``, not ``main``), but otherwise has the same format as discussed here. As of Firefox 61 this is sent on Android only.

Structure:

.. code-block:: js

    {
      version: 4,

      info: {
        reason: <string>, // what triggered this ping: "saved-session", "environment-change", "shutdown", ...
        revision: <string>, // the Histograms.json revision
        timezoneOffset: <integer>, // time-zone offset from UTC, in minutes, for the current locale
        previousBuildId: <string>, // null if this is the first run, or the previous build ID is unknown

        sessionId: <uuid>,  // random session id, shared by subsessions
        subsessionId: <uuid>,  // random subsession id
        previousSessionId: <uuid>, // session id of the previous session, null on first run.
        previousSubsessionId: <uuid>, // subsession id of the previous subsession (even if it was in a different session),
                                      // null on first run.

        subsessionCounter: <unsigned integer>, // the running no. of this subsession since the start of the browser session
        profileSubsessionCounter: <unsigned integer>, // the running no. of all subsessions for the whole profile life time

        sessionStartDate: <ISO date>, // hourly precision, ISO date in local time
        subsessionStartDate: <ISO date>, // hourly precision, ISO date in local time
        sessionLength: <integer>, // the session length until now in seconds, monotonic
        subsessionLength: <integer>, // the subsession length in seconds, monotonic

        addons: <string>, // obsolete, use ``environment.addons``
      },

      processes: {...},
      simpleMeasurements: {...},

      // The following properties may all be null if we fail to collect them.
      histograms: {...},
      keyedHistograms: {...},
      chromeHangs: {...}, // removed in firefox 62
      threadHangStats: [...], // obsolete in firefox 57, use the 'bhr' ping
      log: [...], // obsolete in firefox 61, use Event Telemetry or Scalars
      gc: {...},
      fileIOReports: {...},
      lateWrites: {...},
      addonDetails: {...},
      UIMeasurements: [...],  // Android only
      slowSQL: {...},
      slowSQLstartup: {...},
    }

info
----

sessionLength
~~~~~~~~~~~~~
The length of the current session so far in seconds.
This uses a monotonic clock, so this may mismatch with other measurements that
are not monotonic like calculations based on ``Date.now()``.

Note that this currently does not behave consistently over our supported platforms:

* On Windows this uses ``GetTickCount64()``, which does increase over sleep periods
* On macOS this uses ``mach_absolute_time()``, which does not increase over sleep periods
* On POSIX/Linux this uses ``clock_gettime(CLOCK_MONOTONIC, &ts)``, which should not increase over sleep time

See `bug 1204823 <https://bugzilla.mozilla.org/show_bug.cgi?id=1204823>`_ for details.

subsessionLength
~~~~~~~~~~~~~~~~
The length of this subsession in seconds.
This uses a monotonic clock, so this may mismatch with other measurements that are not monotonic (e.g. based on ``Date.now()``).

Also see the remarks for ``sessionLength`` on platform consistency.

processes
---------
This section contains per-process data.

Structure:

.. code-block:: js

    "processes" : {
      // ... other processes ...
      "parent": {
        scalars: {...},
        keyedScalars: {...},
        // parent process histograms and keyedHistograms are in main payload
      },
      "content": {
        scalars: {...},
        keyedScalars: {...},
        histograms: {...},
        keyedHistograms: {...},
      },
      "gpu": {
        // ...
      }
    }

histograms and keyedHistograms
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section contains histograms and keyed histograms accumulated on content processes. Histograms recorded on a content child process have different character than parent histograms. For instance, ``GC_MS`` will be much different in ``processes.content`` as it has to contend with web content, whereas the instance in ``payload.histograms`` has only to contend with browser JS. Also, some histograms may be absent if never recorded on a content child process (``EVENTLOOP_UI_ACTIVITY`` is parent-process-only).

This format was adopted in Firefox 51 via bug 1218576.

scalars and keyedScalars
~~~~~~~~~~~~~~~~~~~~~~~~
This section contains the :doc:`../collection/scalars` that are valid for the current platform. Scalars are only submitted if data was added to them, and are only reported with subsession pings. The recorded scalars are described in the `Scalars.yaml <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml>`_ file. The ``info.revision`` field indicates the revision of the file that describes the reported scalars.

simpleMeasurements
------------------
This section contains a list of simple measurements, or counters. In addition to the ones highlighted below, Telemetry timestamps (see `here <https://searchfox.org/mozilla-central/search?q=TelemetryTimestamps.add&redirect=false&case=true>`__ and `here <https://searchfox.org/mozilla-central/search?q=recordTimestamp&redirect=false&case=true>`__) can be reported.

totalTime
~~~~~~~~~
A non-monotonic integer representing the number of seconds the session has been alive.

addonManager
~~~~~~~~~~~~
Only available in the extended set of measures, it contains a set of counters related to Addons. See `here <https://searchfox.org/mozilla-central/search?q=AddonManagerPrivate.recordSimpleMeasure&redirect=false&case=true>`__ for a list of recorded measures.

UITelemetry
~~~~~~~~~~~
As of Firefox 61 this section is no longer present.

Only available in the extended set of measures. For more see :ref:`uitelemetry`.

startupInterrupted
~~~~~~~~~~~~~~~~~~
A boolean set to true if startup was interrupted by an interactive prompt.

js
~~
This section contains a series of counters from the JavaScript engine.

Structure:

.. code-block:: js

    "js" : {
      // ...
    }

As of Firefox 59 this section no longer contains any entries, as of Firefox 61 this section is removed.

maximalNumberOfConcurrentThreads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An integer representing the highest number of threads encountered so far during the session.

startupSessionRestoreReadBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes read by the main process up until the session store has finished restoring the windows.

startupSessionRestoreWriteBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes written by the main process up until the session store has finished restoring the windows.

startupWindowVisibleReadBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes read by the main process up until after a XUL window is made visible.

startupWindowVisibleWriteBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes written by the main process up until after a XUL window is made visible.

debuggerAttached
~~~~~~~~~~~~~~~~
A boolean set to true if a debugger is attached to the main process.

shutdownDuration
~~~~~~~~~~~~~~~~
The time, in milliseconds, it took to complete the last shutdown.

failedProfileLockCount
~~~~~~~~~~~~~~~~~~~~~~
The number of times the system failed to lock the user profile.

activeTicks
~~~~~~~~~~~
Integer count of the number of five-second intervals ('ticks') the user was considered 'active' (sending UI events to the window). An extra event is fired immediately when the user becomes active after being inactive. This is for some mouse and gamepad events, and all touch, keyboard, wheel, and pointer events (see `EventStateManager.cpp <https://searchfox.org/mozilla-central/source/dom/events/EventStateManager.cpp#504>`__).
This measure might be useful to give a trend of how much a user actually interacts with the browser when compared to overall session duration. It does not take into account whether or not the window has focus or is in the foreground. Just if it is receiving these interaction events.
Note that in ``main`` pings, this measure is reset on subsession splits, while in ``saved-session`` pings it covers the whole browser session.

histograms
----------
This section contains the histograms that are valid for the current platform. ``Flag`` histograms are always created and submitted with a default value of ``false`` if a value of ``true`` is not recorded during the time period. Other histogram types (see :ref:`choosing-histogram-type`) are not created nor submitted if no data was added to them. The type and format of the reported histograms is described by the ``Histograms.json`` file. Its most recent version is available `here <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json>`__. The ``info.revision`` field indicates the revision of the file that describes the reported histograms.

keyedHistograms
---------------
This section contains the keyed histograms available for the current platform.

As of Firefox 48, this section does not contain empty keyed histograms anymore.

threadHangStats
---------------
As of Firefox 57 this section is no longer present, and has been replaced with the :doc:`bhr ping <backgroundhangmonitor-ping>`.

Contains the statistics about the hangs in main and background threads. Note that hangs in this section capture the `label stack <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler#Native_stack_vs._label_stack>`_ and an incomplete JS stack, which is not 100% precise. For particularly egregious hangs, and on nightly, an unsymbolicated native stack is also captured. The amount of time that is considered "egregious" is different from thread to thread, and is set when the BackgroundHangMonitor is constructed for that thread. In general though, hangs from 5 - 10 seconds are generally considered egregious. Shorter hangs (1 - 2s) are considered egregious for other threads (the compositor thread, and the hang monitor that is only enabled during tab switch).

To avoid submitting overly large payloads, some limits are applied:

* Identical, adjacent "(chrome script)" or "(content script)" stack entries are collapsed together. If a stack is reduced, the "(reduced stack)" frame marker is added as the oldest frame.
* The depth of the reported label stacks is limited to 11 entries. This value represents the 99.9th percentile of the thread hangs stack depths reported by Telemetry.
* The native stacks are limited to a depth of 25 stack frames.

Structure:

.. code-block:: js

    "threadHangStats" : [
      {
        "name" : "Gecko",
        "activity" : {...}, // a time histogram of all task run times
        "nativeStacks": { // captured for all hangs on nightly, or egregious hangs on beta
          "memoryMap": [
            ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"],
            ["igd10iumd32.pdb", "D36DEBF2E78149B5BE1856B772F1C3991"],
            // ... other entries in the format ["module name", "breakpad identifier"] ...
          ],
          "stacks": [
            [
              [
                0, // the module index or -1 for invalid module indices
                190649 // the offset of this program counter in its module or an absolute pc
              ],
              [1, 2540075],
              // ... other frames ...
            ],
            // ... other stacks ...
          ]
        },
        "hangs" : [
          {
            "stack" : [
              "Startup::XRE_Main",
              "Timer::Fire",
              "(content script)",
              "IPDL::PPluginScriptableObject::SendGetChildProperty",
              ... up to 11 frames ...
            ],
            "nativeStack": 0, // index into nativeStacks.stacks array
            "histogram" : {...}, // the time histogram of the hang times
            "annotations" : [
              {
                "pluginName" : "Shockwave Flash",
                "pluginVersion" : "18.0.0.209"
              },
              ... other annotations ...
            ]
          },
        ],
      },
      ... other threads ...
    ]

.. _chromeHangs:

chromeHangs
-----------
As of Firefox 62, chromeHangs has been removed. Please look to the bhr ping for
similar functionality.

Contains the statistics about the hangs happening exclusively on the main thread of the parent process. Precise C++ stacks are reported. This is only available on Nightly Release on Windows, when building using "--enable-profiling" switch.

Some limits are applied:

* Reported chrome hang stacks are limited in depth to 50 entries.
* The maximum number of reported stacks is 50.

The module names can contain unicode characters.

Structure:

.. code-block:: js

    "chromeHangs" : {
      "memoryMap" : [
        ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"],
        ["igd10iumd32.pdb", "D36DEBF2E78149B5BE1856B772F1C3991"],
        ... other entries in the format ["module name", "breakpad identifier"] ...
       ],
      "stacks" : [
        [
          [
            0, // the module index or -1 for invalid module indices
            190649 // the offset of this program counter in its module or an absolute pc
          ],
          [1, 2540075],
          ... other frames, up to 50 ...
         ],
         ... other stacks, up to 50 ...
      ],
      "durations" : [8, ...], // the hang durations (in seconds)
      "systemUptime" : [692, ...], // the system uptime (in minutes) at the time of the hang
      "firefoxUptime" : [672, ...], // the Firefox uptime (in minutes) at the time of the hang
      "annotations" : [
        [
          [0, ...], // the indices of the related hangs
          {
            "pluginName" : "Shockwave Flash",
            "pluginVersion" : "18.0.0.209",
            ... other annotations as key:value pairs ...
          }
        ],
        ...
      ]
    },

log
---
As of Firefox 61 this section is no longer present, use :ref:`eventtelemetry` or :doc:`../collection/scalars`.

This section contains a log of important or unusual events reported through Telemetry.

Structure:

.. code-block:: js

    "log": [
      [
        "Event_ID",
        3785, // the timestamp (in milliseconds) for the log entry
        ... other data ...
      ],
      ...
    ]

At present there is one known users of this section: Telemetry Experiments.

Telemetry Experiments uses it to note when experiments are activated and terminated.

fileIOReports
-------------
Contains the statistics of main-thread I/O recorded during the execution. Only the I/O stats for the XRE and the profile directories are currently reported, neither of them disclosing the full local path.

Structure:

.. code-block:: js

    "fileIOReports": {
      "{xre}": [
        totalTime, // Accumulated duration of all operations
        creates, // Number of create/open operations
        reads, // Number of read operations
        writes, // Number of write operations
        fsyncs, // Number of fsync operations
        stats, // Number of stat operations
      ],
      "{profile}": [ ... ],
      ...
    }

lateWrites
----------
This sections reports writes to the file system that happen during shutdown. The reported data contains the stack and the file names of the loaded libraries at the time the writes happened.

The file names of the loaded libraries can contain unicode characters.

Structure:

.. code-block:: js

    "lateWrites" : {
      "memoryMap" : [
        ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"],
        ... other entries in the format ["module name", "breakpad identifier"] ...
       ],
      "stacks" : [
        [
          [
            0, // the module index or -1 for invalid module indices
            190649 // the offset of this program counter in its module or an absolute pc
          ],
          [1, 2540075],
          ... other frames ...
         ],
         ... other stacks ...
      ],
    },

addonDetails
------------
This section contains per add-on telemetry details, as reported by each add-on provider. The XPI provider is the only one reporting at the time of writing (`see Searchfox <https://searchfox.org/mozilla-central/search?q=setTelemetryDetails&case=true>`_). Telemetry does not manipulate or enforce a specific format for the supplied provider's data.

Structure:

.. code-block:: js

    "addonDetails": {
      "XPI": {
        "adbhelper@mozilla.org": {
          "location": "app-profile",
          "name": "ADB Helper",
          "creator": "Mozilla & Android Open Source Project",
        },
        ...
      },
      ...
    }

slowSQL
-------
This section contains the information about the slow SQL queries for both the main and other threads. The execution of an SQL statement is considered slow if it takes 50ms or more on the main thread or 100ms or more on other threads. Slow SQL statements will be automatically trimmed to 1000 characters. This limit doesn't include the ellipsis and database name, that are appended at the end of the stored statement.

Structure:

.. code-block:: js

    "slowSQL": {
      "mainThread": {
        "Sanitized SQL Statement": [
          1, // the number of times this statement was hit
          200  // the total time (in milliseconds) that was spent on this statement
        ],
        ...
      },
      "otherThreads": {
        "VACUUM /* places.sqlite */": [
          1,
          330
        ],
        ...
      }
    },

slowSQLStartup
--------------
This section contains the slow SQL statements gathered at startup (until the "sessionstore-windows-restored" event is fired). The structure of this section resembles the one for `slowSQL`_.

UIMeasurements
--------------
This section is Android-only and contains UI specific Telemetry measurements and events (`see here <https://searchfox.org/mozilla-central/search?q=UITelemetry.%28addEvent|startSession|stopSession%29&redirect=false&case=false&regexp=true>`_).

Structure:

.. code-block:: js

    "UIMeasurements": [
      {
        "type": "event", // either "session" or "event"
        "action": "action.1",
        "method": "menu",
        "sessions": [],
        "timestamp": 12345,
        "extras": "settings"
      },
      {
        "type": "session",
        "name": "awesomescreen.1",
        "reason": "commit",
        "start": 123,
        "end": 456
      }
      ...
    ],

Version History
---------------

- Firefox 88:

  - Stopped reporting ``flashVersion`` since Flash is no longer supported. (`bug 1682030 <https://bugzilla.mozilla.org/show_bug.cgi?id=1682030>`_)

- Firefox 61:

  - Stopped reporting ``childPayloads`` (`bug 1443599 <https://bugzilla.mozilla.org/show_bug.cgi?id=1443599>`_).
  - Stopped reporting ``saved-session`` pings on Firefox Desktop (`bug 1443603 <https://bugzilla.mozilla.org/show_bug.cgi?id=1443603>`_).
  - Stopped reporting ``simpleMeasurements.js`` (`bug 1278920 <https://bugzilla.mozilla.org/show_bug.cgi?id=1278920>`_).
  - Stopped reporting ``UITelemetry`` (`bug 1443605 <https://bugzilla.mozilla.org/show_bug.cgi?id=1443605>`_)

- Firefox 62:

  - ``events`` are now reported via the :doc:`../data/event-ping` (`bug 1460595 <https://bugzilla.mozilla.org/show_bug.cgi?id=1460595>`_).

- Firefox 80:

  - Stopped reporting ``GCTelemetry`` (`bug 1482089 <https://bugzilla.mozilla.org/show_bug.cgi?id=1482089>`_).