12 files changed, 358 insertions, 0 deletions
diff --git a/docs/crash-reporting/img/default-search-results.png b/docs/crash-reporting/img/default-search-results.png
new file mode 100644
index 0000000000..394f997554
--- /dev/null
+++ b/docs/crash-reporting/img/default-search-results.png
diff --git a/docs/crash-reporting/img/default-search-results2.png b/docs/crash-reporting/img/default-search-results2.png
new file mode 100644
index 0000000000..03fb33d8c3
--- /dev/null
+++ b/docs/crash-reporting/img/default-search-results2.png
diff --git a/docs/crash-reporting/img/facet-search-results.png b/docs/crash-reporting/img/facet-search-results.png
new file mode 100644
index 0000000000..a53db96b65
--- /dev/null
+++ b/docs/crash-reporting/img/facet-search-results.png
diff --git a/docs/crash-reporting/img/facet-search-results2.png b/docs/crash-reporting/img/facet-search-results2.png
new file mode 100644
index 0000000000..7166302974
--- /dev/null
+++ b/docs/crash-reporting/img/facet-search-results2.png
diff --git a/docs/crash-reporting/img/facet-search-results3.png b/docs/crash-reporting/img/facet-search-results3.png
new file mode 100644
index 0000000000..bc96d30ee9
--- /dev/null
+++ b/docs/crash-reporting/img/facet-search-results3.png
diff --git a/docs/crash-reporting/img/narrower-search-results.png b/docs/crash-reporting/img/narrower-search-results.png
new file mode 100644
index 0000000000..38b410b362
--- /dev/null
+++ b/docs/crash-reporting/img/narrower-search-results.png
diff --git a/docs/crash-reporting/img/super-search-form.png b/docs/crash-reporting/img/super-search-form.png
new file mode 100644
index 0000000000..63b35a23ad
--- /dev/null
+++ b/docs/crash-reporting/img/super-search-form.png
diff --git a/docs/crash-reporting/img/super-search-form2.png b/docs/crash-reporting/img/super-search-form2.png
new file mode 100644
index 0000000000..02dd3a541e
--- /dev/null
+++ b/docs/crash-reporting/img/super-search-form2.png
diff --git a/docs/crash-reporting/img/super-search-form3.png b/docs/crash-reporting/img/super-search-form3.png
new file mode 100644
index 0000000000..473706e548
--- /dev/null
+++ b/docs/crash-reporting/img/super-search-form3.png
diff --git a/docs/crash-reporting/index.rst b/docs/crash-reporting/index.rst
new file mode 100644
index 0000000000..c1273c6927
--- /dev/null
+++ b/docs/crash-reporting/index.rst
@@ -0,0 +1,52 @@
+Crash reporting
+===============
+
+Firefox ships with an open-source crash reporting system. This system is
+combination of projects:
+
+-  `Google
+   Breakpad <https://chromium.googlesource.com/breakpad/breakpad>`__
+   client and server libraries
+-  Mozilla-specific crash reporting user interface and bootstrap code
+-  `Socorro <https://github.com/mozilla-services/socorro>`__ Collection
+   and reporting server
+
+
+Where did my crash get submitted?
+---------------------------------
+
+Crash data submitted using the Mozilla Crash Reporter is located on
+`crash-stats <https://crash-stats.mozilla.org/>`__. If you want to find
+a specific crash that you submitted, you first need to find the Crash ID
+that the server has assigned your crash. Type ``about:crashes`` into
+your location bar to get a page listing both submitted and unsubmitted
+crash reports. For more information, see :ref:`How to get a stacktrace for a bug report`.
+
+
+Reports and queries
+-------------------
+
+crash-stats has built-in reports of "topcrashes" for each release
+grouped by signature. There is also a `custom query tool <https://crash-stats.mozilla.org/search/>`__
+which allows users to limit searches on more precise information.
+
+Finally, a set of Mozilla employees have access to directly query the
+underlying data in either SQL summary or using mapreduce on the storage
+cluster. If you are interested in obtaining this advanced access, read
+`Crash Stats Documentation: Protected Data Access <https://crash-stats.mozilla.org/documentation/protected_data_access/>`__
+
+
+See also
+--------
+
+-  :ref:`Understanding crash reports`
+-  :ref:`A guide to searching crash reports`
+-  `crash-stats <https://crash-stats.mozilla.org/>`__
+-  `Crash pings (Telemetry) and crash reports (Socorro/Crash
+   Stats) <https://bluesock.org/~willkg/blog/mozilla/crash_pings_crash_reports.html>`__
+-  :ref:`Building with Debug Symbols`
+-  :ref:`Environment variables affecting crash reporting <Crash Reporter#Environment variables affecting crash reporting>`
+-  :ref:`Uploading symbols to Mozilla's symbol server`
+-  :ref:`Crash reporter`
+-  :ref:`Crash manager`
+-  :ref:`Crash ping`
diff --git a/docs/crash-reporting/searching_crash_reports.rst b/docs/crash-reporting/searching_crash_reports.rst
new file mode 100644
index 0000000000..0668a03654
--- /dev/null
+++ b/docs/crash-reporting/searching_crash_reports.rst
@@ -0,0 +1,257 @@
+A guide to searching crash reports
+==================================
+
+.. note::
+
+   Please read the :ref:`documentation about individual crash
+   reports <Understanding crash reports>` before reading
+   this page.
+
+The Mozilla `crash-stats <https://crash-stats.mozilla.org/>`__ site
+provides facilities for investigating large numbers of Firefox `crash
+reports <Understanding crash reports>`__. This guide to
+searching through crash reports may help you locate the crash reports
+that will help you find and fix the Firefox bug you're working on.
+
+Specifically, crash-stats offers two basic functions:
+
+Searching
+   You can search the crash reports database by over 100 criteria: crash
+   signature, date, platform, product, version, etc.
+Grouping
+   You can cluster the results of each search into groups using the same
+   criteria.
+
+To achieve full power and flexibility requires a good understanding of
+both of these functions. Search is easy to understand, but the grouping
+capabilities are easy to overlook.
+
+Searching
+---------
+
+The search form
+~~~~~~~~~~~~~~~
+
+You can get to the `search
+page <https://crash-stats.mozilla.org/search/?product=Firefox&_dont_run=1>`__
+by clicking on the "Super Search" link near the toolbar at the top right
+of any page in crash-stats. This brings up a search form like the one in
+the following screenshot.
+
+|Search in crash-stats|
+
+Fields are provided for four common search criteria: product, version,
+platform, and process type. The product field is pre-populated with
+"Firefox" because that is a common case. As the fine print says, the
+default date range is the past week.
+
+The default search: Signature facet
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you click on the "Search" button, you will get
+`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature>`__
+like the ones in the following screenshot.
+
+|Results of a default search in crash-stats|
+
+By default, the "Signature facet" tab is selected. ("Facet" is a term
+that means "group".) In these results, the found crash reports are
+grouped according to crash signature and ranked by group size. The
+columns show each group's rank, signature, size (both a count and a
+proportion of matching crash reports), and finally a list of bugs that
+have been marked as relating to this signature.
+
+The numbers are large because this search matched all Firefox crash
+reports from the past seven days. The first group has over 100,000 crash
+reports, which accounts for 7.77% of all matching crashes. This
+indicates there are over 1.3 million crash reports matching this search.
+
+You can reorder the groups in various ways by clicking on the column
+headers. The links within the results do the following things.
+
+-  The first link in each "Signature" column cell links to a signature
+   report, which contains additional details about crash reports with
+   that signature.
+-  The "Add term" link in each "Signature" column cell lets you perform
+   a narrower subsequent search among crash reports with that signature.
+-  The links in each "Bugs" column cell link to bug reports in Bugzilla.
+
+The default search: Crash reports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you switch to the "Crash Reports" tab you will see
+`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports>`__
+like the ones in the following screenshot.
+
+|Results of a default search in crash-stats (crash reports tab)|
+
+This is a list of all the individual crash reports that match the search
+criteria. If the number of matches is large -- in this case it exceeds
+1.3 million, just as we saw in the "Signature facet" tab -- the results
+will be spread across multiple pages, which you can visit by clicking
+the links at the top right of the tab.
+
+The links within the results do the following things.
+
+-  The link in each "Crash ID" column cell links to an individual crash
+   report.
+-  The links in each "Signature" column cell have the same effect that
+   they did in the "Signature facet" tab.
+-  The links in the remaining column cells also let you perform a
+   narrower subsequent search with that link's value added to the search
+   criteria.
+
+A narrower search
+~~~~~~~~~~~~~~~~~
+
+You can add criteria to perform a narrower search. For example, to
+perform a search for all Mac crash reports that occurred while
+JavaScript garbage collection was running, do the following.
+
+-  Add "Mac OS X" to the "Platform" field.
+-  Select "New line", and then choose a field ("is garbage collecting")
+   and an operator ("is true"). The operators available for each field
+   depends on its type.
+
+With these criteria added the search form looks like the following
+screenshot.
+
+|crash-stats Super Search form with additional criteria|
+
+After clicking on "Search" we get
+`results <https://crash-stats.mozilla.org/search/?is_garbage_collecting=__true__&product=Firefox&platform=Mac%20OS%20X&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform>`__
+like those in the following screenshot.
+
+|Results of a narrower search in crash-stats|
+
+The number of crash reports matching this search is in the thousands,
+i.e. much smaller than the previous search.
+
+Proto signature
+~~~~~~~~~~~~~~~
+
+The "proto signature" field is just the raw unprocessed crash stack
+concatenated together.
+
+You can do things like:
+
+-  Search for crashes where the signature is Foo, and the proto
+   signature contains Bar. This is helpful if you have a fairly generic
+   signature and you want to see how many of them are a particular case
+   of it that you've come across. Or instead of a signature Foo, a moz
+   crash reason or something else.
+-  Use it as a facet. This lets you skim the full signatures of crashes
+   at a glance, bucketed together a bit. Note that because the proto
+   signature includes the entire signature, things aren't grouped all
+   that well.
+
+Grouping
+--------
+
+In the previous section we saw one example of grouping, in the
+"Signature facet" tab that is shown by default. But there are many other
+interesting ways to group searches.
+
+Facets in the search form
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To do a search with non-signature grouping first click on the "More
+options..." text, which reveals the additional fields shown in the
+following screenshot.
+
+|crash-stats Super Search form with different facets|
+
+(The "Show columns" and "Sort by" fields are straightforward. They apply
+to the "Crash reports" tab of any search results, and are not related to
+grouping.)
+
+The "Facet on" field is the one that controls grouping. By default, it
+contains the value "signature", which explains why we saw a "Signature
+facet" tab in the earlier search results. But we can change the values
+in this field and get different facet tabs in the search results.
+
+Grouping by platform
+~~~~~~~~~~~~~~~~~~~~
+
+For example, if we start with a default search for all Firefox crashes
+in the past week, but then replace the "signature" facet with "platform"
+and "moz crash reason", we get search results with two facet tabs. The
+first of these is a "Platform facet" tab, with
+`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=platform&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-platform>`__
+like those shown in the following screenshot.
+
+|Results of a faceted search in crash-stats|
+
+This has the same columns as the "Signature facet" tab we saw earlier,
+except for the "Bugs" column, because that is a special column that only
+applies to the signature facet. This tab shows the distribution of crash
+reports across the various platforms. Crash reports always include a
+platform field (though it may be empty if something has gone wrong) and
+so the percentages add up to 100.
+
+Grouping by "moz crash reason"
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The second facet tab is a "Moz crash reason facet" tab, with
+`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=platform&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-moz_crash_reason>`__
+like those shown in the following screenshot.
+
+|Results of a faceted search in crash-stats (moz crash reason tab)|
+
+This immediately shows which ``MOZ_CRASH`` calls are being hit
+frequently by users. Only a subset of crash reports have the "moz crash
+reason" field -- those that crashed due to hitting a ``MOZ_CRASH`` call
+-- so all crashes that lack that field are omitted from this tab. For
+that reason, the percentages do not add up to 100.
+
+An example of less useful grouping
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The usefulness of grouping varies from field to field. In particular,
+fields that can have many possible values (such as numeric fields) often
+don't group well. For example, if we do a default search grouped by
+uptime we get
+`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=uptime&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-uptime>`__
+like those in the following screenshot.
+
+|Results of a faceted search in crash-stats (uptime)|
+
+In this example the top 10 groups account for less than 12% of all
+crashes, and there is an extremely long tail. These results would be
+improved by using numeric ranges instead of individual values, but
+unfortunately that isn't supported.
+
+Advanced Usage
+--------------
+
+The combination of searching and grouping is powerful. Searches find
+crash reports that match particular criteria, and grouping organizes
+those crash reports into interesting groups.
+
+When a search is performed, the page's URL is updated to include the
+search parameters. This means that the results of any search can be
+easily shared by copying and pasting the page's URL.
+
+To become an expert at searching and grouping requires understanding the
+full range of the 100+ fields available for searching and grouping. One
+way to learn about them is to read lots of individual crash reports;
+note that all fields shown in the Details tab of an individual crash
+report have a tool-tip that indicates its key for search. Alternatively,
+you can browse the `complete
+list <https://crash-stats.mozilla.org/documentation/supersearch/api/#section-filters>`__.
+
+There is also an API through which searches can be performed
+programmatically. See the `API
+documentation <https://crash-stats.mozilla.org/documentation/supersearch/>`__
+for full details; note that it uses the term "aggregation" for
+grouping/faceting.
+
+.. |Search in crash-stats| image:: img/super-search-form.png
+.. |Results of a default search in crash-stats| image:: img/default-search-results.png
+.. |Results of a default search in crash-stats (crash reports tab)| image:: img/default-search-results2.png
+.. |crash-stats Super Search form with additional criteria| image:: img/super-search-form2.png
+.. |Results of a narrower search in crash-stats| image:: img/narrower-search-results.png
+.. |crash-stats Super Search form with different facets| image:: img/super-search-form3.png
+.. |Results of a faceted search in crash-stats| image:: img/facet-search-results.png
+.. |Results of a faceted search in crash-stats (moz crash reason tab)| image:: img/facet-search-results2.png
+.. |Results of a faceted search in crash-stats (uptime)| image:: img/facet-search-results3.png
diff --git a/docs/crash-reporting/uploading_symbol.rst b/docs/crash-reporting/uploading_symbol.rst
new file mode 100644
index 0000000000..1a7624ba07
--- /dev/null
+++ b/docs/crash-reporting/uploading_symbol.rst
@@ -0,0 +1,49 @@
+Uploading symbols to Mozilla's symbol server
+============================================
+
+As a third-party releasing your own builds of Firefox or B2G, you should
+consider uploading debug symbols from the builds to Mozilla's symbol
+server. If you have not disabled crash reporting in your builds, crash
+reports will be submitted to `Mozilla's crash reporting
+server <https://crash-stats.mozilla.org/>`__. Without the debug symbols
+that match your build the crash reports will not contain actionable
+information.
+
+Symbols can be uploaded either via a web browser or a web API.
+
+
+Building a Symbol Package
+-------------------------
+
+To upload symbols, you need to build a symbol package. This is a
+.zip file which contains the symbol files in a specific directory structure.
+
+If you are building Firefox,or a similar application using the Mozilla
+build system, you can build the symbol package using a make target:
+
+::
+
+   ./mach buildsymbols
+
+This will create a symbol package in ``dist/`` named something like
+``firefox-77.0a1.en-US.linux-x86_64.crashreporter-symbols.zip`` .
+
+This step requires the ``dump_syms`` tool which should have been automatically
+installed when you setup the Firefox build with ``./mach bootstrap``. If for
+some reason it's missing or outdated running the bootstrap step again will
+retrieve and install an up-to-date version of the tool.
+
+Uploading symbols
+-----------------
+
+Symbols are uploaded via your account on symbols.mozilla.org. Visit
+https://symbols.mozilla.org and log in. Then request upload
+permission by filing a bug in the Socorro component using `this
+template <https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&amp;bug_ignored=0&amp;bug_severity=--&amp;bug_status=NEW&amp;bug_type=task&amp;cc=gsvelto%40mozilla.com&amp;cc=willkg%40mozilla.com&amp;cf_fx_iteration=---&amp;cf_fx_points=---&amp;comment=What%20e-mail%20account%20are%20you%20requesting%20access%20for%3F%0D%0A...%0D%0A%0D%0AWhat%20symbols%20will%20you%20be%20uploading%20using%20this%20account%3F%0D%0A...%0D%0A%0D%0AIs%20there%20somebody%20at%20Mozilla%20who%20can%20vouch%20for%20you%3F%0D%0A...%0D%0A&amp;component=Upload&amp;contenttypemethod=list&amp;contenttypeselection=text%2Fplain&amp;defined_groups=1&amp;filed_via=standard_form&amp;flag_type-4=X&amp;flag_type-607=X&amp;flag_type-800=X&amp;flag_type-803=X&amp;flag_type-936=X&amp;form_name=enter_bug&amp;maketemplate=Remember%20values%20as%20bookmarkable%20template&amp;op_sys=Unspecified&amp;priority=--&amp;product=Tecken&amp;rep_platform=Unspecified&amp;short_desc=Symbol-upload%20permission%20for%20%3CPerson%3E&amp;target_milestone=---&amp;version=unspecified>`__.
+If you don't have an account yet use the template to request one.
+
+After symbol upload is turned on, you can upload the symbol archive
+directly using the web form at https://symbols.mozilla.org/uploads.
+It is also possible to upload via automated scripts: see the `symbol upload API
+docs <https://tecken.readthedocs.io/en/latest/>`__ for more
+details.