path: root/docs/contributing/debugging/understanding_crash_reports.rst

Understanding Crash Reports
===========================

+--------------------------------------------------------------------+
| This page is an import from MDN and the contents might be outdated |
+--------------------------------------------------------------------+

If a user experiences a crash they will be prompted to submit a raw
crash report, which is generated by Breakpad. The raw crash report is
received by `Socorro <https://github.com/mozilla/socorro>`__ which
`creates <https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py>`__
a processed crash report. The processed crash report is based on the raw
crash report but also has a signature, classifications, and a number of
improved fields (e.g. OS, product, version). Many of the fields in both
the raw crash report and the processed crash report are viewable and
searchable on `crash-stats <https://crash-stats.mozilla.org/>`__.
Although there are two distinct crash reports, the raw and the
processed, people typically talk about a single "crash report" because
crash-stats mostly presents them in a combined way.

Each crash report contains a wealth of data about the crash
circumstances. Despite this, many crash reports lack sufficient data for
a developer to understand why the crash occurred. As well as providing a
general overview, this page aims to highlight parts of a crash report
that may provide non-obvious insights.

Note that most crash report fields are visible, but a few
privacy-sensitive parts of it are only available to users who are logged
in and have "minidump access". A relatively small number of users have
minidump access, and they are required to follow certain rules. For
access, see the `Protected Data Access docs on Crash Stats
<https://crash-stats.mozilla.org/documentation/protected_data_access/>`__.

Each crash report has the following tabs: Details, Metadata, Modules,
Raw Dump, Extensions, and (optional) Correlations.

Details tab
-----------

The Details tab is the first place to look because it contains the most
important pieces of information.

Primary fields
~~~~~~~~~~~~~~

| The first part of the Details tab shows a table containing the most
  important crash report fields. It includes such things as when the
  crash occurred, in which product and version, the crash kind, and
  various details about the OS and configuration of the machine on which
  the crash occurred. The following screenshot shows some of these
  fields.
| |Example fields in the "Details" tab of a crash report|

All fields have a tool-tip. For many fields, the tool-tip describes its
meaning. For all fields, the tool-tip indicates the key to use when you
want to do searches involving this field. (The field name is usually but
not always similar to the search key. E.g. the field "Adapter Device ID"
has the search key "adapter_device_id".) These descriptions are shown in
the `SuperSearchFields
API <https://crash-stats.mozilla.org/api/SuperSearchFields/>`__ and can be
`modified in super_search_fields.py <https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py>`__
or by writing up a `bug in Socorro <https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro>`__.

The fields present in this tab vary depending on the crash kind. Not all
fields are always present.

The "Signature" field is the main identifier or label for a crash report.
Rather than considering each crash report in isolation, we want to put
crash reports into clusters so we can deal with groups of them at once.
An ideal clustering algorithm would put all crash reports with the same
root cause into a single cluster, and all crash reports with different
root causes into different clusters. The crash signature is our
imperfect but still useful attempt at such an algorithm. Most crash
signatures are based on the crashing stack trace, but some
special-purpose annotations are used to indicate particular kinds of
crashes.

-  ``Abort``: A controlled abort, e.g. via ``NS_RUNTIMEABORT``.
   (Controlled aborts that occur via ``MOZ_CRASH`` or
   ``MOZ_RELEASE_ASSERT`` currently don't get an ``Abort`` annotation,
   but they do get a "MOZ_CRASH Reason" field.)
-  ``OOM | <size>``, where ``<size>`` is one of ``large``, ``small``,
   ``unknown``: an out-of-memory (OOM) abort. The ``<size>`` annotation
   is determined by the "OOM Allocation Size" field; if that field is
   missing ``<size>`` will be ``unknown``.
-  ``hang``: a hang prior to shutdown.
-  ``shutdownhang``: a hang during shutdown.
-  ``IPCError-browser``: a problem involving IPC. If the parent Firefox
   process detects that the child process has sent broken or
   unprocessable IPDL data, or is not shutting down in a timely manner,
   it kills the child process with a crash report. These crashes will
   now have a signature that indicates why the process was killed,
   rather than the child stack at the moment.

When no special-purpose annotation is present and the signature begins
with a stack frame, it's usually a vanilla uncontrolled crash. The crash
cause can be determined from the "Crash Reason" field. Most commonly
it's a bad memory access. In that case, on Windows you can tell from the
reason field if the crash occurred while reading, writing or executing
memory (e.g. ``EXCEPTION_VIOLATION_ACCESS_READ`` indicates a bad memory
read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you
cannot tell from this field what kind of memory access it was.

See `this
file <https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst>`__
for a detailed explanation of the crash report signature generation
procedure, and for information on how modify this procedure.

There are no fields that uniquely identify the user that a crash report
came from, but if you want to know if multiple crashes come from a
single user the "Install Time" field is a good choice. Use it in
conjunction with other fields that don't change, such as those
describing the OS or graphics card, for additional confidence.

For bad memory accesses, the "Crash Address" field can give additional
indications what went wrong.

-  0x0 is probably a null pointer deference[*].
-  Small addresses like 0x8 can indicate an object access (e.g.
   ``this->mFoo``) via a null ``this`` pointer.
-  Addresses like 0xfffffffffd8 might be stack accesses, depending on
   the platform[*].
-  Addresses like 0x80cdefd3 might be heap accesses, depending on the
   platform.
-  Addresses may be poisoned: 0xe4 indicates the address comes from
   memory that has been allocated by jemalloc but not yet initialized;
   0xe5 indicates the address comes from memory freed by jemalloc. The
   JS engine also has multiple poison values defined in
   ``js/src/jsutil.h``.

[*] Note that due to the way addressing works on x86-64, if the crash
address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for
a Windows crash report, it's highly likely that the value is incorrect.
(There is a `bug
report <https://bugzilla.mozilla.org/show_bug.cgi?id=1493342>`__ open
for this problem.) You can sanity-check these crashes by looking at the
raw dump or minidump in the Raw Dump tab (see below).

Note that for non-release builds the "Version" field represents multiple
different builds since nightly and beta version numbers are reused for
builds created over a series of days until the version number is bumped.
(The "Build ID" field can disambiguate.) It's not currently possible to
`restrict searches to a given version or
later <https://bugzilla.mozilla.org/show_bug.cgi?id=1401517>`__ (using
>= with a build ID and a given release channel may work around this).

Some fields, such as "URL" and "Email Address", are privacy-sensitive
and are only visible to users with minidump access.

The Windows-only "Total Virtual Memory" field indicates if the Firefox
build and OS are 32-bit or 64-bit.

-  A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows.
-  A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows
   (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox.
-  A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit
   Firefox. (The "Build Architecture" field should be "amd64" in this
   case.)

Some crash reports might contain a memory report. This memory report will
have been made some time before the crash, at a time when available
memory was low. In this case, a number of key measurements from the
memory report are shown in the Details tab, each one having a field name
starting with "MR:", short for "memory report". The full memory report
can be obtained in the Raw Dump tab (see below).

Bug-related information
~~~~~~~~~~~~~~~~~~~~~~~

The second part of the Details tab shows bug-related information, as the
following screenshot shows.

|Information relating to bug reports in the "Details" tab of a crash
report|

The "Report this bug in" links can be used to easily file bug reports.
Each one links to a Bugzilla bug report creation page that has various
fields pre-filled, such as the crash signature.

The "Related Bugs" section shows related bug reports, as determined by
the crash signature.

Stack traces
~~~~~~~~~~~~

The third part of the Details tab shows the stack trace and thread
number of the crashing thread, as the following screenshot shows.

|Information relating to threads in the "Details" tab of a crash report|

Each stack frame has a link to the source code, when possible. If a
crash is new, the regressing changeset can often be identified by
looking for recent changes in the blame annotations for one or more of
the top stack frames. Blame annotations are also good for identifying
who might know about the code in question.

Sometimes the highlighted source code is puzzling, e.g. the identified
line may not touch memory even though the crash is memory-related. This
can be caused by compiler optimizations. It's often better to look at
the disassembly (e.g. in a minidump) to understand exactly what code is
being executed.

Stack frame entries take on a variety of forms.

-  The simplest are functions names, such as ``NS_InitXPCOM2``.
-  Name/address pairs such as ``nss3.dll@0x1eb720`` are within system
   libraries.
-  Names such as ``F1398665248_____________________________`` ('F'
   followed by many numbers then many underscores) are in Flash.
-  Addresses such as ``@0xe1a850ac`` may indicate an address that wasn't
   part of any legitimate code. If an address such as this occurs in the
   first stack frame, the crash may be
   `exploitable <https://developer.mozilla.org/en-US/docs/Mozilla/Security/Exploitable_crashes>`__.

Stack traces for other threads can be viewed by clicking on the small
"Show other threads" link.

If the crash report is for a hang, the crashing thread will be the
"watchdog" thread, which exists purely to detect hangs; its top stack
frame will be something
like\ :literal:`mozilla::`anonymous namespace'::RunWatchdog`. In that
case you should look at the other threads' stack traces to determine the
problem; many of them will be waiting on some kind of response, as shown
by a top stack frame containing a function like
``NtWaitForSingleObject`` or ``ZwWaitForMultipleObjects``.

Metadata tab
------------

The Metadata tab is similar to the first part of the Details tab,
containing a table with various fields. These are the fields from the
raw crash report, ordered alphabetically by field name, but with
privacy-sensitive fields shown only to users with minidump access. There
is some overlap with the fields shown in the Details tab.

Modules tab
-----------

The modules tab shows all the system libraries loaded at the time of the
crash, as the following screenshot shows.

|Table of modules in the "Modules" tab of a crash report|

On Windows these are mostly DLLs, on Mac they are mostly ``.dylib``
files, and on Linux they are mostly ``.so`` files.

This information is most useful for Windows crashes, because DLLs loaded
by antivirus software or malware often cause Firefox to crash.
Correlations between loaded modules and crash signatures can be seen in
the "Correlations" tab (see below).

`This page <https://support.mozilla.org/en-US/kb/helping-crashes>`__
says that files lacking version/debug identifier/debug filename are
likely to be malware.

Raw Dump tab
------------

The first part of the Raw Dump tab shows the raw crash report, in JSON
format. Once again, privacy-sensitive fields are shown only to users
with minidump access.

|JSON data in the "Raw Dump" tab of a crash report|

For users with minidump access, the second part of the Raw Dump tab has
some links, as the following screenshot shows.

|Links to downloadable files in the "Raw Dump" tab of a crash report|

These links are to the following items.

#. A minidump. Minidumps can be extremely useful in understanding a
   crash report; see :ref:`this page <Debugging A Minidump>` for an
   explanation how to use them.
#. The aforementioned JSON raw crash report.
#. The memory report contained within the crash report.
#. The unredacted crash report, which has additional information.

Extensions tab
--------------

The Extensions tab shows which extensions are installed and enabled.

|Table of extensions in the "Extensions" tab of a crash report|

Usually it just shows an ID rather than the proper extension name.

Note that several extensions ship by default with Firefox and so will be
present in almost all crash reports. (The exact set of default
extensions depends on the release channel.) The least obvious of these
has an Id of ``{972ce4c6-7e08-4474-a285-3208198ce6fd}``, which is the
default Firefox theme. Some (but not all) of the other extensions
shipped by default have the following Ids: ``webcompat@mozilla.org``,
``e10srollout@mozilla.org``, ``firefox@getpocket.com``,
``flyweb@mozilla.org``, ``loop@mozilla.org``.

If an extension only has a hexadecimal identifier, a Google search of
that identifier is usually enough to identify the extension's name.

This information is useful because some crashes are caused by
extensions. Correlations between extensions and crash signatures can be
seen in the "Correlations" tab (see below).

Correlations tab
----------------

This tab is only shown when crash-stats identifies correlations between
a crash and modules or extensions that are present, which happens
occasionally.

See also
--------

-  `A talk about understanding crash
   reports <https://air.mozilla.org/a-talk-about-understanding-crash-reports/>`__,
   by David Baron, from March 2016.
-  :ref:`A guide to searching crash reports`

.. |Example fields in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13579/Details1.png
.. |Information relating to bug reports in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13581/Details2.png
.. |Information relating to threads in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13583/Details3.png
.. |Table of modules in the "Modules" tab of a crash report| image:: https://mdn.mozillademos.org/files/13593/Modules1.png
.. |JSON data in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/13595/RawDump1.png
.. |Links to downloadable files in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/14047/raw-dump-links.png
.. |Table of extensions in the "Extensions" tab of a crash report| image:: https://mdn.mozillademos.org/files/13599/Extensions1.png