diff options
Diffstat (limited to 'toolkit/crashreporter/docs')
| -rw-r--r-- | toolkit/crashreporter/docs/Using_the_Mozilla_symbol_server.rst | 167 | ||||
| -rw-r--r-- | toolkit/crashreporter/docs/img/snip_20170901070042.png | bin | 0 -> 15434 bytes | |||
| -rw-r--r-- | toolkit/crashreporter/docs/img/snip_20170901081816.png | bin | 0 -> 3821 bytes | |||
| -rw-r--r-- | toolkit/crashreporter/docs/img/symbol-server-windbg-menu.jpg | bin | 0 -> 17919 bytes | |||
| -rw-r--r-- | toolkit/crashreporter/docs/img/vs_tools_options.png | bin | 0 -> 12084 bytes | |||
| -rw-r--r-- | toolkit/crashreporter/docs/index.rst | 273 |
6 files changed, 440 insertions, 0 deletions
diff --git a/toolkit/crashreporter/docs/Using_the_Mozilla_symbol_server.rst b/toolkit/crashreporter/docs/Using_the_Mozilla_symbol_server.rst new file mode 100644 index 0000000000..c636e41994 --- /dev/null +++ b/toolkit/crashreporter/docs/Using_the_Mozilla_symbol_server.rst @@ -0,0 +1,167 @@ +Using the Mozilla symbol server +=============================== + +The Mozilla project runs a symbol server for trunk Firefox nightly and +release builds on Windows. Symbols are available for at least 30 +previous days worth of nightly builds, and Firefox releases from +2.0.0.4. This allows debugging of those builds without forcing all users +to download large debugging files. The server functions like +`Microsoft's symbol server <https://docs.microsoft.com/windows/win32/dxtecharts/debugging-with-symbols>`__ +so the documentation there can be useful. + +Note that because Mozilla release builds are heavily optimized, +debugging is not always easy. The debugger will not be able to show you +the content of all variables and the execution path can seem strange +because of inlining, tail calls, and other compiler optimizations. The +only workaround is to build an unoptimized local build. + +The official symbol server URL for Firefox is ``https://symbols.mozilla.org/``. +You cannot visit this URL directly: you must add it to the symbol path of your +debugging tool. In the examples below, a local cache directory is used to avoid +repeatedly fetching the PDB from the server. Replace +``C:\Users\myname\symbols`` with an appropriate cache directory on your +machine. + +Using the symbol server in Microsoft Visual C++ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +|The Tools menu of Visual C++ with the Options item selected| |The +Visual C++ 2017 Symbols pane of the Options dialog with the Mozilla +symbol server configured| + +Using the symbol server in Windbg +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Windbg symbol path is configured with a string value delimited with +asterisk characters. To use only the Mozilla symbol server, add the +following entry to your symbol path (**note:** you can replace ``C:\symcache\`` +with any writable directory on your computer, if you'd prefer a different +location for downloaded symbols): + +.. code:: + + SRV*c:\symcache\*https://symbols.mozilla.org/ + +Set this string as ``_NT_SYMBOL_PATH`` in the environment, using +the Windbg menus, or by typing the ``.sympath`` command. If you +would like to get symbols from Microsoft's symbol server as well, you +should list that first (**note:** you can replace ``C:\symcache\`` +with any writable directory on your computer, if you'd prefer a different +location for downloaded symbols): + +.. code:: + + SRV*c:\symcache\*http://msdl.microsoft.com/download/symbols;SRV*c:\symcache\*https://symbols.mozilla.org/ + +|Image:symbol-server-windbg-menu.jpg| |The WinDbg Symbol Search Path +dialog with the Mozilla symbol server configured| + +Downloading symbols using symchk.exe +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can also download the symbols using symchk.exe, part of Microsoft's +Debugging Tools for Windows. The command should look like this (again, +you can replace ``C:\symcache\`` with any writable directory on +your computer, if you'd prefer a different location for downloaded +symbols): + +:: + + "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\symchk.exe" /r "C:\Program Files\Mozilla Firefox/*" /s SRV*C:\symcache\*https://symbols.mozilla.org/ + +Note the ``\*`` after the Mozilla directory. The output of this command should +be similar to: + +:: + + SYMCHK: api-ms-win-core-file-l1-2-0.dll FAILED - api-ms-win-core-file-l1-2-0.pdb mismatched or not found + SYMCHK: api-ms-win-core-file-l2-1-0.dll FAILED - api-ms-win-core-file-l2-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-core-localization-l1-2-0.dll FAILED - api-ms-win-core-localization-l1-2-0.pdb mismatched or not found + SYMCHK: api-ms-win-core-processthreads-l1-1-1.dll FAILED - api-ms-win-core-processthreads-l1-1-1.pdb mismatched or not found + SYMCHK: api-ms-win-core-synch-l1-2-0.dll FAILED - api-ms-win-core-synch-l1-2-0.pdb mismatched or not found + SYMCHK: api-ms-win-core-timezone-l1-1-0.dll FAILED - api-ms-win-core-timezone-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-conio-l1-1-0.dll FAILED - api-ms-win-crt-conio-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-convert-l1-1-0.dll FAILED - api-ms-win-crt-convert-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-environment-l1-1-0.dll FAILED - api-ms-win-crt-environment-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-filesystem-l1-1-0.dll FAILED - api-ms-win-crt-filesystem-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-heap-l1-1-0.dll FAILED - api-ms-win-crt-heap-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-locale-l1-1-0.dll FAILED - api-ms-win-crt-locale-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-math-l1-1-0.dll FAILED - api-ms-win-crt-math-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-multibyte-l1-1-0.dll FAILED - api-ms-win-crt-multibyte-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-private-l1-1-0.dll FAILED - api-ms-win-crt-private-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-process-l1-1-0.dll FAILED - api-ms-win-crt-process-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-runtime-l1-1-0.dll FAILED - api-ms-win-crt-runtime-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-stdio-l1-1-0.dll FAILED - api-ms-win-crt-stdio-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-string-l1-1-0.dll FAILED - api-ms-win-crt-string-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-time-l1-1-0.dll FAILED - api-ms-win-crt-time-l1-1-0.pdb mismatched or not found + SYMCHK: api-ms-win-crt-utility-l1-1-0.dll FAILED - api-ms-win-crt-utility-l1-1-0.pdb mismatched or not found + SYMCHK: d3dcompiler_47.dll FAILED - D3DCompiler_47.pdb mismatched or not found + SYMCHK: maintenanceservice_installer.exe FAILED - Built without debugging information. + SYMCHK: msvcp140.dll FAILED - msvcp140.amd64.pdb mismatched or not found + SYMCHK: ucrtbase.dll FAILED - ucrtbase.pdb mismatched or not found + SYMCHK: vcruntime140.dll FAILED - vcruntime140.amd64.pdb mismatched or not found + SYMCHK: helper.exe FAILED - Built without debugging information. + + SYMCHK: FAILED files = 27 + SYMCHK: PASSED + IGNORED files = 60 + +Downloading symbols on Linux / Mac OS X +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are on Linux and running GDB 7.9 or newer, you can use `this GDB +Python script <https://gist.github.com/luser/193572147c401c8a965c>`__ to +automatically fetch symbols. You will need to source this script before +loading symbols (the part where it spends a few seconds loading each .so +when you attach gdb). If you want to reload symbols, you can try: + +:: + + nosharedlibrary + sharedlibrary lib + +On older GDB and Mac OS X there is a `Python script to download symbols +from the Mozilla symbol +server <http://hg.mozilla.org/users/jwatt_jwatt.org/fetch-symbols>`__ +for :ref:`gdb <Debugging Firefox with GDB>`, Shark and other software +that uses symbols. Note that the symbol file for the XUL library is very +large and takes some time to download. This may make it appear as if the +script has gotten stuck, but it will continue. + +Symbol indices are named like so: +symbols.mozilla.org/{lowercased:Name}/{lowercased:Name}-{Version}-{Platform}-{BuildID}-symbols.txt. +The Platform is either 'Darwin' (for Mac) or 'Linux'. The rest of values +are based on the contents of the application.ini file under the [App] +heading: For example, the Thunderbird 3.1b2 release with +Name=Thunderbird, Version=3.1b2, BuildID=20100430125415 would have a +filename of "thunderbird-3.1b2-Linux-20100430125415-symbols.txt" under +the thunderbird directory at symbols.mozilla.org. Its contents are a +list of paths to files, all relative to the directory the +BLAH-symbols.txt file is found in. + +The source server +~~~~~~~~~~~~~~~~~ + +In addition to symbols, Mozilla also has a +:ref:`source server <Using The Mozilla Source Server>`, letting you do +source-level debugging and inspection on demand. + +Troubleshooting: Symbols will not download +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If symbols will not download no matter what you do, the problem may be +that Internet Explorer has been set to the **Work Offline** mode. You +will not receive any warnings of this in Windbg, Visual C++ or Visual +Studio. Even using the command line with symchk.exe to download symbols +will fail. This is because Microsoft uses Internet Explorer's internet & +proxy settings to download the symbol files. Check the File menu of +Internet Explorer to ensure "Work Offline" is unchecked. + +.. |The Tools menu of Visual C++ with the Options item selected| image:: img/vs_tools_options.png + :class: internal +.. |The Visual C++ 2017 Symbols pane of the Options dialog with the Mozilla symbol server configured| image:: img/snip_20170901070042.png + :class: internal +.. |Image:symbol-server-windbg-menu.jpg| image:: img/=Symbol-server-windbg-menu.jpg + :class: internal +.. |The WinDbg Symbol Search Path dialog with the Mozilla symbol server configured| image:: img/snip_20170901081816.png + :class: internal + diff --git a/toolkit/crashreporter/docs/img/snip_20170901070042.png b/toolkit/crashreporter/docs/img/snip_20170901070042.png Binary files differnew file mode 100644 index 0000000000..94f266d91e --- /dev/null +++ b/toolkit/crashreporter/docs/img/snip_20170901070042.png diff --git a/toolkit/crashreporter/docs/img/snip_20170901081816.png b/toolkit/crashreporter/docs/img/snip_20170901081816.png Binary files differnew file mode 100644 index 0000000000..550f0d28ea --- /dev/null +++ b/toolkit/crashreporter/docs/img/snip_20170901081816.png diff --git a/toolkit/crashreporter/docs/img/symbol-server-windbg-menu.jpg b/toolkit/crashreporter/docs/img/symbol-server-windbg-menu.jpg Binary files differnew file mode 100644 index 0000000000..000bc761d7 --- /dev/null +++ b/toolkit/crashreporter/docs/img/symbol-server-windbg-menu.jpg diff --git a/toolkit/crashreporter/docs/img/vs_tools_options.png b/toolkit/crashreporter/docs/img/vs_tools_options.png Binary files differnew file mode 100644 index 0000000000..c16d2094a6 --- /dev/null +++ b/toolkit/crashreporter/docs/img/vs_tools_options.png diff --git a/toolkit/crashreporter/docs/index.rst b/toolkit/crashreporter/docs/index.rst new file mode 100644 index 0000000000..f41cd448b3 --- /dev/null +++ b/toolkit/crashreporter/docs/index.rst @@ -0,0 +1,273 @@ +============== +Crash Reporter +============== + +Overview +======== + +The **crash reporter** is a subsystem to record and manage application +crash data. + +While the subsystem is known as *crash reporter*, it helps to think of +it more as a *process dump manager*. This is because the heart of this +subsystem is really managing process dump files and these files are +created not only from process crashes but also from hangs and other +exceptional events. + +The crash reporter subsystem is composed of a number of pieces working +together. + +Breakpad + Breakpad is a library and set of tools to make collecting process + information (notably dumps from crashes) easy. Breakpad is a 3rd + party project (originally developed by Google) that is imported into + the tree. + +Dump files + Breakpad produces files called *dump files* that hold process data + (stacks, heap data, etc). + +Crash Reporter Client + The crash reporter client is a standalone executable that is launched + to handle dump files. This application optionally submits crashes to + Mozilla (or the configured server). + +Minidump Analyzer + The minidump analyzer is a standalone executable that is launched by the + crash reporter client or by the browser itself to extract stack traces from + the dump files generated during a crash. It appends the stack traces to the + .extra file associated with the crash dump. + +Ping Sender + The ping sender is a standalone executable that is launched by the crash + reporter client to deliver a crash ping to our telemetry servers. The ping + sender is used to speed up delivery of the crash ping which would otherwise + have to wait for Firefox to be restarted in order to be sent. + +How Main-Process Crash Handling Works +===================================== + +The crash handler is hooked up very early in the Gecko process lifetime. +It all starts in ``XREMain::XRE_mainInit()`` from ``nsAppRunner.cpp``. +Assuming crash reporting is enabled, this startup function registers an +exception handler for the process and tells the crash reporter subsystem +about basic metadata such as the application name and version. + +The registration of the crash reporter exception handler doubles as +initialization of the crash reporter itself. This happens in +``CrashReporter::SetExceptionHandler()`` from ``nsExceptionHandler.cpp``. +The crash reporter figures out what application to use for reporting +dumped crashes and where to store these dump files on disk. The Breakpad +exception handler (really just a mechanism for dumping process state) is +initialized as part of this function. The Breakpad exception handler is +a ``google_breakpad::ExceptionHandler`` instance and it's stored as +``gExceptionHandler``. + +As the application runs, various other systems may write *annotations* +or *notes* to the crash reporter to indicate state of the application, +help with possible reasons for a current or future crash, etc. These are +performed via ``CrashReporter::AnnotateCrashReport()`` and +``CrashReporter::AppendAppNotesToCrashReport()`` from +``nsExceptionHandler.h``. + +For well running applications, this is all that happens. However, if a +crash or similar exceptional event occurs (such as a hang), we need to +write a crash report. + +When an event worthy of writing a dump occurs, the Breakpad exception +handler is invoked and Breakpad does its thing. When Breakpad has +finished, it calls back into ``CrashReporter::MinidumpCallback()`` from +``nsExceptionHandler.cpp`` to tell the crash reporter about what was +written. + +``MinidumpCallback()`` performs a number of actions once a dump has been +written. It writes a file with the time of the crash so other systems can +easily determine the time of the last crash. It supplements the dump +file with an *extra* file containing Mozilla-specific metadata. This data +includes the annotations set via ``CrashReporter::AnnotateCrashReport()`` +as well as time since last crash, whether garbage collection was active at +the time of the crash, memory statistics, etc. + +If the *crash reporter client* is enabled, ``MinidumpCallback()`` invokes +it. It simply tries to create a new *crash reporter client* process (e.g. +*crashreporter.exe*) with the path to the written minidump file as an +argument. + +The *crash reporter client* performs a number of roles. There's a lot going +on, so you may want to look at ``main()`` in ``crashreporter.cpp``. First, +stack traces are extracted from the dump via the *minidump analyzer* tool. +The resulting traces are appended to the .extra file of the crash together with +the SHA256 hash of the minidump file. Once this +is done a crash ping is assembled holding the same information as the one +generated by the ```CrashManager``` and it's sent to the telemetry servers via +the *ping sender* program. The UUID of the ping is then stored in the extra +file; the ```CrashManager``` will later pick it up and generate a new ping +with the same UUID so that the telemetry server can deduplicate both pings. +Then, the +*crash reporter client* verifies that the dump data is sane. If it isn't +(e.g. required metadata is missing), the dump data is ignored. If dump data +looks sane, the dump data +is moved into the *pending* directory for the configured data directory +(defined via the ``MOZ_CRASHREPORTER_DATA_DIRECTORY`` environment variable +or from the UI). Once this is done, the main crash reporter UI is displayed +via ``UIShowCrashUI()``. The crash reporter UI is platform specific: there +are separate versions for Windows, OS X, and various \*NIX presentation +flavors (such as GTK). The basic gist is a dialog is displayed to the user +and the user has the opportunity to submit this dump data to a remote +server. + +If a dump is submitted via the crash reporter, the raw dump files are +removed from the *pending* directory and a file containing the +crash ID from the remote server for the submitted dump is created in the +*submitted* directory. + +If the user chooses not to submit a dump in the crash reporter UI, the dump +files are deleted. + +And that's pretty much what happens when a crash/dump is written! + +Plugin and Child Process Crashes +================================ + +Crashes in plugin and child processes are also managed by the crash +reporting subsystem. + +Child process crashes are handled by the ``mozilla::dom::CrashReporterParent`` +class defined in ``dom/ipc``. When a child process crashes, the toplevel IPDL +actor should check for it by calling TakeMinidump in its ``ActorDestroy`` +Method: see ``mozilla::plugins::PluginModuleParent::ActorDestroy`` and +``mozilla::plugins::PluginModuleParent::ProcessFirstMinidump``. That method +is responsible for calling +``mozilla::dom::CrashReporterParent::GenerateCrashReportForMinidump`` with +appropriate crash annotations specific to the crash. All child-process +crashes are annotated with a ``ProcessType`` annotation, such as "content" or +"plugin". + +Once the minidump file has been generated the +``mozilla::dom::CrashReporterHost`` is notified of the crash. It will first +try to extract the stack traces from the minidump file using the +*minidump analyzer*. Then the stack traces will be stored in the extra file +together with the rest of the crash annotations and finally the crash will be +recorded by calling ```CrashService.addCrash()```. This last step adds the +crash to the ```CrashManager``` database and automatically sends a crash ping +with information about the crash. + +Submission of child process crashes is handled by application code. This +code prompts the user to submit crashes in context-appropriate UI and then +submits the crashes using ``CrashSubmit.jsm``. + +Memory Reports +============== + +When a process detects that it is running low on memory, a memory report is +saved. If the process crashes, the memory report will be included with the crash +report. ``nsThread::SaveMemoryReportNearOOM()`` checks to see if the process is +low on memory every 30 seconds at most and saves a report every 3 minutes at +most. Since a child process cannot actually save to the hard drive, it instead +notifies its parent process, which saves the report for it. If a crash does +occur, the memory report is moved to the *pending* directory with the other dump +data and an annotation is added to indicate the presence of the report. This +happens in ``nsExceptionHandler.cpp``, but occurs in different functions +depending on what process crashed. When the main process crashes, this happens +in ``MinidumpCallback()``. When a child process crashes, it happens in +``OnChildProcessDumpRequested()``, with the annotation being added in +``WriteExtraData()``. + +Flash Process Crashes +===================== + +On Windows Vista+, the Adobe Flash plugin creates two extra processes in its +Firefox plugin to implement OS-level sandboxing. In order to catch crashes in +these processes, Firefox injects a crash report handler into the process using the code at ``InjectCrashReporter.cpp``. When these crashes occur, the +ProcessType=plugin annotation is present, and an additional annotation +FlashProcessDump has the value "Sandbox" or "Broker". + +Plugin Hangs +============ + +Plugin hangs are handled as crash reports. If a plugin doesn't respond to an +IPC message after 60 seconds, the plugin IPC code will take minidumps of all +of the processes involved and then kill the plugin. + +In this case, there will be only one .ini file with the crash report metadata, +but there will be multiple dump files: at least one for the browser process and +one for the plugin process, and perhaps also additional dumps for the Flash +sandbox and broker processes. All of these files are submitted together as a +unit. Before submission, the filenames of the files are linked: + +- **uuid.ini** - *annotations, includes an additional_minidumps field* +- **uuid.dmp** - *plugin process dump file* +- **uuid-<other>.dmp** - *other process dump file as listed in additional_minidumps* + +about:crashes +============= + +If the crash reporter subsystem is enabled, the *about:crashes* +page will be registered with the application. This page provides +information about previous and submitted crashes. + +It is also possible to submit crashes from *about:crashes*. + +Environment variables affecting crash reporting +=============================================== + +The exception handler and crash reporter client behavior can be altered by +setting certain environment variables, some of these variables are used for +testing but quite a few have only internal users. + +User-specified environment variables +------------------------------------ + +- ``MOZ_CRASHREPORTER`` - The opposite of MOZ_CRASHREPORTER_DISABLE, force + crash reporting on even if disabled in application.ini. You must use this to + enable crash reporting on debug builds. +- ``MOZ_CRASHREPORTER_DISABLE`` - Disable Breakpad crash reporting completely + in non-debug builds. You can use this if you would rather use the JIT + debugger on Windows with the symbol server, for example. +- ``MOZ_CRASHREPORTER_FULLDUMP`` - Store full application memory in the + minidump, so you can open it in a Microsoft debugger. Don't submit it to the + server. (Windows only.) +- ``MOZ_CRASHREPORTER_NO_DELETE_DUMP`` - Don't delete the crash report dump + file after submitting it to the server. Minidumps will still be moved to the + "Crash Reports/pending" directory. +- ``MOZ_CRASHREPORTER_NO_REPORT`` - Save the minidump file but don't launch the + crash reporting UI or send the report to the server. Minidumps will be stored + in the user's profile directory, in a subdirectory named "minidumps". +- ``MOZ_CRASHREPORTER_SHUTDOWN`` - Save the minidump and then force the + application to close. This is useful for content crashes that don't normally + close the chrome (main application) processes. This variable would cause the + application to close as well. +- ``MOZ_CRASHREPORTER_URL`` - Sets the URL that the crash reporter will submit + reports to. + +Environment variables used internally +------------------------------------- + +- ``MOZ_CRASHREPORTER_AUTO_SUBMIT`` - When set causes the crash reporter client + to skip the UI flow and submit the crash report directly. +- ``MOZ_CRASHREPORTER_DATA_DIRECTORY`` - Platform dependent data directory, the + pending crash reports will be stored in a subdirectory of this path. This + overrides the default one generated by the client's code. +- ``MOZ_CRASHREPORTER_DUMP_ALL_THREADS`` - When set to 1 stack traces for + all threads are generated and sent in the crash ping, when not set only the + trace for the crashing thread will be generated instead. +- ``MOZ_CRASHREPORTER_EVENTS_DIRECTORY`` - Path of the directory holding the + crash event files. +- ``MOZ_CRASHREPORTER_PING_DIRECTORY`` - Path of the directory holding the + pending crash ping files. +- ``MOZ_CRASHREPORTER_RESTART_ARG_<n>`` - Each of these variable specifies one + of the arguments that had been passed to the application, the crash reporter + client uses them for restarting it. +- ``MOZ_CRASHREPORTER_RESTART_XUL_APP_FILE`` - If a XUL app file was specified + when starting the app it has to be stored in this variable so that the crash + reporter client can restart the application. +- ``MOZ_CRASHREPORTER_STRINGS_OVERRIDE`` - Overrides the path used to load the + .ini file holding the strings used in the crash reporter client UI. + +Other topics +============ + +.. toctree:: + :titlesonly: + + Using_the_Mozilla_symbol_server |
