diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 17:32:43 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 17:32:43 +0000 |
commit | 6bf0a5cb5034a7e684dcc3500e841785237ce2dd (patch) | |
tree | a68f146d7fa01f0134297619fbe7e33db084e0aa /docs/crash-reporting | |
parent | Initial commit. (diff) | |
download | thunderbird-upstream.tar.xz thunderbird-upstream.zip |
Adding upstream version 1:115.7.0.upstream/1%115.7.0upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
-rw-r--r-- | docs/crash-reporting/img/default-search-results.png | bin | 0 -> 74498 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/default-search-results2.png | bin | 0 -> 97584 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/facet-search-results.png | bin | 0 -> 19042 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/facet-search-results2.png | bin | 0 -> 58106 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/facet-search-results3.png | bin | 0 -> 25830 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/narrower-search-results.png | bin | 0 -> 53573 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/super-search-form.png | bin | 0 -> 26165 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/super-search-form2.png | bin | 0 -> 28824 bytes | |||
-rw-r--r-- | docs/crash-reporting/img/super-search-form3.png | bin | 0 -> 40430 bytes | |||
-rw-r--r-- | docs/crash-reporting/index.rst | 52 | ||||
-rw-r--r-- | docs/crash-reporting/searching_crash_reports.rst | 257 | ||||
-rw-r--r-- | docs/crash-reporting/uploading_symbol.rst | 49 |
12 files changed, 358 insertions, 0 deletions
diff --git a/docs/crash-reporting/img/default-search-results.png b/docs/crash-reporting/img/default-search-results.png Binary files differnew file mode 100644 index 0000000000..394f997554 --- /dev/null +++ b/docs/crash-reporting/img/default-search-results.png diff --git a/docs/crash-reporting/img/default-search-results2.png b/docs/crash-reporting/img/default-search-results2.png Binary files differnew file mode 100644 index 0000000000..03fb33d8c3 --- /dev/null +++ b/docs/crash-reporting/img/default-search-results2.png diff --git a/docs/crash-reporting/img/facet-search-results.png b/docs/crash-reporting/img/facet-search-results.png Binary files differnew file mode 100644 index 0000000000..a53db96b65 --- /dev/null +++ b/docs/crash-reporting/img/facet-search-results.png diff --git a/docs/crash-reporting/img/facet-search-results2.png b/docs/crash-reporting/img/facet-search-results2.png Binary files differnew file mode 100644 index 0000000000..7166302974 --- /dev/null +++ b/docs/crash-reporting/img/facet-search-results2.png diff --git a/docs/crash-reporting/img/facet-search-results3.png b/docs/crash-reporting/img/facet-search-results3.png Binary files differnew file mode 100644 index 0000000000..bc96d30ee9 --- /dev/null +++ b/docs/crash-reporting/img/facet-search-results3.png diff --git a/docs/crash-reporting/img/narrower-search-results.png b/docs/crash-reporting/img/narrower-search-results.png Binary files differnew file mode 100644 index 0000000000..38b410b362 --- /dev/null +++ b/docs/crash-reporting/img/narrower-search-results.png diff --git a/docs/crash-reporting/img/super-search-form.png b/docs/crash-reporting/img/super-search-form.png Binary files differnew file mode 100644 index 0000000000..63b35a23ad --- /dev/null +++ b/docs/crash-reporting/img/super-search-form.png diff --git a/docs/crash-reporting/img/super-search-form2.png b/docs/crash-reporting/img/super-search-form2.png Binary files differnew file mode 100644 index 0000000000..02dd3a541e --- /dev/null +++ b/docs/crash-reporting/img/super-search-form2.png diff --git a/docs/crash-reporting/img/super-search-form3.png b/docs/crash-reporting/img/super-search-form3.png Binary files differnew file mode 100644 index 0000000000..473706e548 --- /dev/null +++ b/docs/crash-reporting/img/super-search-form3.png diff --git a/docs/crash-reporting/index.rst b/docs/crash-reporting/index.rst new file mode 100644 index 0000000000..c1273c6927 --- /dev/null +++ b/docs/crash-reporting/index.rst @@ -0,0 +1,52 @@ +Crash reporting +=============== + +Firefox ships with an open-source crash reporting system. This system is +combination of projects: + +- `Google + Breakpad <https://chromium.googlesource.com/breakpad/breakpad>`__ + client and server libraries +- Mozilla-specific crash reporting user interface and bootstrap code +- `Socorro <https://github.com/mozilla-services/socorro>`__ Collection + and reporting server + + +Where did my crash get submitted? +--------------------------------- + +Crash data submitted using the Mozilla Crash Reporter is located on +`crash-stats <https://crash-stats.mozilla.org/>`__. If you want to find +a specific crash that you submitted, you first need to find the Crash ID +that the server has assigned your crash. Type ``about:crashes`` into +your location bar to get a page listing both submitted and unsubmitted +crash reports. For more information, see :ref:`How to get a stacktrace for a bug report`. + + +Reports and queries +------------------- + +crash-stats has built-in reports of "topcrashes" for each release +grouped by signature. There is also a `custom query tool <https://crash-stats.mozilla.org/search/>`__ +which allows users to limit searches on more precise information. + +Finally, a set of Mozilla employees have access to directly query the +underlying data in either SQL summary or using mapreduce on the storage +cluster. If you are interested in obtaining this advanced access, read +`Crash Stats Documentation: Protected Data Access <https://crash-stats.mozilla.org/documentation/protected_data_access/>`__ + + +See also +-------- + +- :ref:`Understanding crash reports` +- :ref:`A guide to searching crash reports` +- `crash-stats <https://crash-stats.mozilla.org/>`__ +- `Crash pings (Telemetry) and crash reports (Socorro/Crash + Stats) <https://bluesock.org/~willkg/blog/mozilla/crash_pings_crash_reports.html>`__ +- :ref:`Building with Debug Symbols` +- :ref:`Environment variables affecting crash reporting <Crash Reporter#Environment variables affecting crash reporting>` +- :ref:`Uploading symbols to Mozilla's symbol server` +- :ref:`Crash reporter` +- :ref:`Crash manager` +- :ref:`Crash ping` diff --git a/docs/crash-reporting/searching_crash_reports.rst b/docs/crash-reporting/searching_crash_reports.rst new file mode 100644 index 0000000000..0668a03654 --- /dev/null +++ b/docs/crash-reporting/searching_crash_reports.rst @@ -0,0 +1,257 @@ +A guide to searching crash reports +================================== + +.. note:: + + Please read the :ref:`documentation about individual crash + reports <Understanding crash reports>` before reading + this page. + +The Mozilla `crash-stats <https://crash-stats.mozilla.org/>`__ site +provides facilities for investigating large numbers of Firefox `crash +reports <Understanding crash reports>`__. This guide to +searching through crash reports may help you locate the crash reports +that will help you find and fix the Firefox bug you're working on. + +Specifically, crash-stats offers two basic functions: + +Searching + You can search the crash reports database by over 100 criteria: crash + signature, date, platform, product, version, etc. +Grouping + You can cluster the results of each search into groups using the same + criteria. + +To achieve full power and flexibility requires a good understanding of +both of these functions. Search is easy to understand, but the grouping +capabilities are easy to overlook. + +Searching +--------- + +The search form +~~~~~~~~~~~~~~~ + +You can get to the `search +page <https://crash-stats.mozilla.org/search/?product=Firefox&_dont_run=1>`__ +by clicking on the "Super Search" link near the toolbar at the top right +of any page in crash-stats. This brings up a search form like the one in +the following screenshot. + +|Search in crash-stats| + +Fields are provided for four common search criteria: product, version, +platform, and process type. The product field is pre-populated with +"Firefox" because that is a common case. As the fine print says, the +default date range is the past week. + +The default search: Signature facet +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you click on the "Search" button, you will get +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature>`__ +like the ones in the following screenshot. + +|Results of a default search in crash-stats| + +By default, the "Signature facet" tab is selected. ("Facet" is a term +that means "group".) In these results, the found crash reports are +grouped according to crash signature and ranked by group size. The +columns show each group's rank, signature, size (both a count and a +proportion of matching crash reports), and finally a list of bugs that +have been marked as relating to this signature. + +The numbers are large because this search matched all Firefox crash +reports from the past seven days. The first group has over 100,000 crash +reports, which accounts for 7.77% of all matching crashes. This +indicates there are over 1.3 million crash reports matching this search. + +You can reorder the groups in various ways by clicking on the column +headers. The links within the results do the following things. + +- The first link in each "Signature" column cell links to a signature + report, which contains additional details about crash reports with + that signature. +- The "Add term" link in each "Signature" column cell lets you perform + a narrower subsequent search among crash reports with that signature. +- The links in each "Bugs" column cell link to bug reports in Bugzilla. + +The default search: Crash reports +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you switch to the "Crash Reports" tab you will see +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports>`__ +like the ones in the following screenshot. + +|Results of a default search in crash-stats (crash reports tab)| + +This is a list of all the individual crash reports that match the search +criteria. If the number of matches is large -- in this case it exceeds +1.3 million, just as we saw in the "Signature facet" tab -- the results +will be spread across multiple pages, which you can visit by clicking +the links at the top right of the tab. + +The links within the results do the following things. + +- The link in each "Crash ID" column cell links to an individual crash + report. +- The links in each "Signature" column cell have the same effect that + they did in the "Signature facet" tab. +- The links in the remaining column cells also let you perform a + narrower subsequent search with that link's value added to the search + criteria. + +A narrower search +~~~~~~~~~~~~~~~~~ + +You can add criteria to perform a narrower search. For example, to +perform a search for all Mac crash reports that occurred while +JavaScript garbage collection was running, do the following. + +- Add "Mac OS X" to the "Platform" field. +- Select "New line", and then choose a field ("is garbage collecting") + and an operator ("is true"). The operators available for each field + depends on its type. + +With these criteria added the search form looks like the following +screenshot. + +|crash-stats Super Search form with additional criteria| + +After clicking on "Search" we get +`results <https://crash-stats.mozilla.org/search/?is_garbage_collecting=__true__&product=Firefox&platform=Mac%20OS%20X&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform>`__ +like those in the following screenshot. + +|Results of a narrower search in crash-stats| + +The number of crash reports matching this search is in the thousands, +i.e. much smaller than the previous search. + +Proto signature +~~~~~~~~~~~~~~~ + +The "proto signature" field is just the raw unprocessed crash stack +concatenated together. + +You can do things like: + +- Search for crashes where the signature is Foo, and the proto + signature contains Bar. This is helpful if you have a fairly generic + signature and you want to see how many of them are a particular case + of it that you've come across. Or instead of a signature Foo, a moz + crash reason or something else. +- Use it as a facet. This lets you skim the full signatures of crashes + at a glance, bucketed together a bit. Note that because the proto + signature includes the entire signature, things aren't grouped all + that well. + +Grouping +-------- + +In the previous section we saw one example of grouping, in the +"Signature facet" tab that is shown by default. But there are many other +interesting ways to group searches. + +Facets in the search form +~~~~~~~~~~~~~~~~~~~~~~~~~ + +To do a search with non-signature grouping first click on the "More +options..." text, which reveals the additional fields shown in the +following screenshot. + +|crash-stats Super Search form with different facets| + +(The "Show columns" and "Sort by" fields are straightforward. They apply +to the "Crash reports" tab of any search results, and are not related to +grouping.) + +The "Facet on" field is the one that controls grouping. By default, it +contains the value "signature", which explains why we saw a "Signature +facet" tab in the earlier search results. But we can change the values +in this field and get different facet tabs in the search results. + +Grouping by platform +~~~~~~~~~~~~~~~~~~~~ + +For example, if we start with a default search for all Firefox crashes +in the past week, but then replace the "signature" facet with "platform" +and "moz crash reason", we get search results with two facet tabs. The +first of these is a "Platform facet" tab, with +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=platform&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-platform>`__ +like those shown in the following screenshot. + +|Results of a faceted search in crash-stats| + +This has the same columns as the "Signature facet" tab we saw earlier, +except for the "Bugs" column, because that is a special column that only +applies to the signature facet. This tab shows the distribution of crash +reports across the various platforms. Crash reports always include a +platform field (though it may be empty if something has gone wrong) and +so the percentages add up to 100. + +Grouping by "moz crash reason" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The second facet tab is a "Moz crash reason facet" tab, with +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=platform&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-moz_crash_reason>`__ +like those shown in the following screenshot. + +|Results of a faceted search in crash-stats (moz crash reason tab)| + +This immediately shows which ``MOZ_CRASH`` calls are being hit +frequently by users. Only a subset of crash reports have the "moz crash +reason" field -- those that crashed due to hitting a ``MOZ_CRASH`` call +-- so all crashes that lack that field are omitted from this tab. For +that reason, the percentages do not add up to 100. + +An example of less useful grouping +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The usefulness of grouping varies from field to field. In particular, +fields that can have many possible values (such as numeric fields) often +don't group well. For example, if we do a default search grouped by +uptime we get +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=uptime&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-uptime>`__ +like those in the following screenshot. + +|Results of a faceted search in crash-stats (uptime)| + +In this example the top 10 groups account for less than 12% of all +crashes, and there is an extremely long tail. These results would be +improved by using numeric ranges instead of individual values, but +unfortunately that isn't supported. + +Advanced Usage +-------------- + +The combination of searching and grouping is powerful. Searches find +crash reports that match particular criteria, and grouping organizes +those crash reports into interesting groups. + +When a search is performed, the page's URL is updated to include the +search parameters. This means that the results of any search can be +easily shared by copying and pasting the page's URL. + +To become an expert at searching and grouping requires understanding the +full range of the 100+ fields available for searching and grouping. One +way to learn about them is to read lots of individual crash reports; +note that all fields shown in the Details tab of an individual crash +report have a tool-tip that indicates its key for search. Alternatively, +you can browse the `complete +list <https://crash-stats.mozilla.org/documentation/supersearch/api/#section-filters>`__. + +There is also an API through which searches can be performed +programmatically. See the `API +documentation <https://crash-stats.mozilla.org/documentation/supersearch/>`__ +for full details; note that it uses the term "aggregation" for +grouping/faceting. + +.. |Search in crash-stats| image:: img/super-search-form.png +.. |Results of a default search in crash-stats| image:: img/default-search-results.png +.. |Results of a default search in crash-stats (crash reports tab)| image:: img/default-search-results2.png +.. |crash-stats Super Search form with additional criteria| image:: img/super-search-form2.png +.. |Results of a narrower search in crash-stats| image:: img/narrower-search-results.png +.. |crash-stats Super Search form with different facets| image:: img/super-search-form3.png +.. |Results of a faceted search in crash-stats| image:: img/facet-search-results.png +.. |Results of a faceted search in crash-stats (moz crash reason tab)| image:: img/facet-search-results2.png +.. |Results of a faceted search in crash-stats (uptime)| image:: img/facet-search-results3.png diff --git a/docs/crash-reporting/uploading_symbol.rst b/docs/crash-reporting/uploading_symbol.rst new file mode 100644 index 0000000000..1a7624ba07 --- /dev/null +++ b/docs/crash-reporting/uploading_symbol.rst @@ -0,0 +1,49 @@ +Uploading symbols to Mozilla's symbol server +============================================ + +As a third-party releasing your own builds of Firefox or B2G, you should +consider uploading debug symbols from the builds to Mozilla's symbol +server. If you have not disabled crash reporting in your builds, crash +reports will be submitted to `Mozilla's crash reporting +server <https://crash-stats.mozilla.org/>`__. Without the debug symbols +that match your build the crash reports will not contain actionable +information. + +Symbols can be uploaded either via a web browser or a web API. + + +Building a Symbol Package +------------------------- + +To upload symbols, you need to build a symbol package. This is a +.zip file which contains the symbol files in a specific directory structure. + +If you are building Firefox,or a similar application using the Mozilla +build system, you can build the symbol package using a make target: + +:: + + ./mach buildsymbols + +This will create a symbol package in ``dist/`` named something like +``firefox-77.0a1.en-US.linux-x86_64.crashreporter-symbols.zip`` . + +This step requires the ``dump_syms`` tool which should have been automatically +installed when you setup the Firefox build with ``./mach bootstrap``. If for +some reason it's missing or outdated running the bootstrap step again will +retrieve and install an up-to-date version of the tool. + +Uploading symbols +----------------- + +Symbols are uploaded via your account on symbols.mozilla.org. Visit +https://symbols.mozilla.org and log in. Then request upload +permission by filing a bug in the Socorro component using `this +template <https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=--&bug_status=NEW&bug_type=task&cc=gsvelto%40mozilla.com&cc=willkg%40mozilla.com&cf_fx_iteration=---&cf_fx_points=---&comment=What%20e-mail%20account%20are%20you%20requesting%20access%20for%3F%0D%0A...%0D%0A%0D%0AWhat%20symbols%20will%20you%20be%20uploading%20using%20this%20account%3F%0D%0A...%0D%0A%0D%0AIs%20there%20somebody%20at%20Mozilla%20who%20can%20vouch%20for%20you%3F%0D%0A...%0D%0A&component=Upload&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&filed_via=standard_form&flag_type-4=X&flag_type-607=X&flag_type-800=X&flag_type-803=X&flag_type-936=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Tecken&rep_platform=Unspecified&short_desc=Symbol-upload%20permission%20for%20%3CPerson%3E&target_milestone=---&version=unspecified>`__. +If you don't have an account yet use the template to request one. + +After symbol upload is turned on, you can upload the symbol archive +directly using the web form at https://symbols.mozilla.org/uploads. +It is also possible to upload via automated scripts: see the `symbol upload API +docs <https://tecken.readthedocs.io/en/latest/>`__ for more +details. |