diff options
Diffstat (limited to 'docs/contributing/debugging')
32 files changed, 3718 insertions, 0 deletions
diff --git a/docs/contributing/debugging/capturing_minidump.rst b/docs/contributing/debugging/capturing_minidump.rst new file mode 100644 index 0000000000..4c8f7591f5 --- /dev/null +++ b/docs/contributing/debugging/capturing_minidump.rst @@ -0,0 +1,259 @@ +Capturing a minidump +==================== + +*Minidumps* are files created by various Windows tools which record the +complete state of a program as it's running, or as it was at the moment +of a crash. Small minidumps are created by the Breakpad :ref:`crash +reporting <Crash Reporter>` tool, but sometimes that's not +sufficient to diagnose a problem. For example, if the application is +hanging (not responding to input, but hasn't crashed) then Breakpad is +not triggered, and it can be difficult to determine where the problem +lies. Sometimes a more complete form of minidump is needed to see +additional details about a crash, in which case manual capture of a +minidump is desired. + +This page describes how to capture these minidumps on Windows, to permit +better debugging. + + +Privacy and minidumps +--------------------- + +.. warning:: + + **Warning!** Unlike the minidumps submitted by Breakpad, these + minidumps contain the **complete** contents of program memory. They + are therefore much more likely to contain private information, if + there is any in the browser. For this reason, you may prefer to + generate minidumps against a `clean + profile <http://support.mozilla.com/en-US/kb/Managing%20profiles>`__ + where possible. + + +Capturing a minidump: application crash +--------------------------------------- + +To capture a full minidump for an application crash, you can use a tool called +**Debugging Tools for Windows**, which is provided by Microsoft for free. + + +Install Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please follow `these instructions +<https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/>`__. +You can install the Debugging Tools for Windows alone, but you cannot +download it alone. To install it, download Windows Software Development Kit +(SDK) first, start the installer, and then select only **Debugging Tools for +Windows** in the list of features to install. + +.. image:: img/sdk-installer.png + +The latest installer installs the tools for all CPU architectures (X86, +X64, ARM, and ARM64). You need to choose a tool of the architecture +matching Firefox you want to capture a minidump from. For example, if +you want to capture a minidump from 32-bit Firefox on 64-bit Windows, +use the X86 version of tools, not X64 tools. + +The default install path of SDK for 64-bit Windows is +``%ProgramFiles(x86)%\Windows Kits\10``. The debugging tools can be found in +the folder named **Debuggers** under your install path of SDK. + +The Debugging Tools for Windows contains both graphical and command line +debugging tools that can be used to capture a minidump. If you prefer +a graphical tool, please follow `Capture a minidump in a graphical way +<#capture-a-minidump-in-a-graphical-way>`__. If you prefer a command +line tool, please follow `Capture a minidump from the command line +<#capture-a-minidump-from-the-command-line>`__. + + +Capture a minidump in a graphical way +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#. Launch the graphical debugger named WinDbg from the Start Menu. WinDbg + icons are registered under "Windows Kits" as below. + + |WinDbg in Start Menu| + +#. Connect Firefox to the debugger. + + a. If Firefox is not already running, open the **"File"** menu on WinDbg + and choose **"Open Executable..."**. In the file chooser window that + appears, open the firefox.exe executable. If you're not sure about where + it is, please see `How to find the location of firefox.exe + <#how-to-find-the-location-of-firefox-exe>`__. + + b. If Firefox is already running and you know which process you want to + capture a minidump from, open the **"File"** menu on WinDbg and choose + **"Attach to a Process..."**. In the "Attach to Process" dialog that + appears, select the process. To identify a process, please see + `Identify a process to attach a debugger to + <#identify-a-process-to-attach-a-debugger-to>`__. + +#. You should now see a "Command" text window with debug output at the + top and an input box at the bottom. From the menu, select + ``Debug → Go``, and Firefox should start. If the debugger spits out + some text right away and Firefox doesn't come up, select + ``Debug → Go`` again. + +#. When the program is about to crash, WinDbg will spit out more data, + and the prompt at the bottom will change from saying "``*BUSY*``" to + having a number in it. At this point, you should type + "``.dump /ma c:\temp\firefoxcrash.dmp``" -- without the quotes, but + don't forget the dot at the beginning. Once it completes, which can + take a fair while, you will have a very large file at + ``c:\temp\firefoxcrash.dmp`` that can be used to help debug your + problem. File size will depend on this size of Firefox running in + your environment, which could several GB. + +#. Ask in the relevant bug or thread how best to share this very large + file! + + +Capture a minidump from the command line +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox is not already running, open the Command Prompt and run the following +command. This command launches all Firefox processes under a debugger. This +technique is useful when you have a startup crash or when you're not sure about +which process will crash. + +To find out where firefox.exe is, please see `How to find the location +of firefox.exe <#how-to-find-the-location-of-firefox-exe>`__. + +.. code:: + + <path to debugger>\cdb.exe -g -G -o <path to firefox>\firefox.exe + + +For example, if both the debugging tools and Firefox are installed in the +default folder and you want to capture a minidump of 64-bit Firefox, +the command will be like this. Please note that you need to add double +quotes when a path contains one or more whitespaces. + +.. code:: + + "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o "C:\Program Files\Mozilla Firefox\firefox.exe" + + +If a Firefox process you want to capture a minidump from is already running, +attach the debugger to it with the following command. To identify a process, +please see `Identify a process to attach a debugger to +<#identify-a-process-to-attach-a-debugger-to>`__. + +.. code:: + + <path to debugger>\cdb.exe -g -G -p <PID> + +When the process crashes, the debugger tool captures it and waits for your +command. At this point, you should type ``.dump /ma c:\temp\firefoxcrash.dmp`` +-- don't forget the dot at the beginning. Once it completes, which can take +a fair while, you will have a very large file at ``c:\temp\firefoxcrash.dmp`` +that can be used to help debug your problem. File size will depend on this +size of Firefox running in your environment, which could several GB. + +After a minidump is generated, type ``q`` and press Enter to quit the debugger. + + +Capturing a minidump: application hang +-------------------------------------- + +On Windows Vista and Windows 7, you can follow `these +instructions <http://support.microsoft.com/kb/931673>`__ to capture a +dump file and locate it after it's been saved. + + +Identify a process to attach a debugger to +------------------------------------------ + +When you're running Firefox, even if you have only a single tab, you may have +noticed a bunch of firefox.exe instances in Task Manager. This means Firefox +consists of multiple processes. Since an application crash happens per process +and a minidump is generated per process, you need to identify which process will +crash before starting a debugger. + +Identify a process type +~~~~~~~~~~~~~~~~~~~~~~~ + +Each instance of firefox.exe has a type. To identify a process to attach +a debugger to, you need to know its process type first. + +When a crash happens, if all Firefox windows are suddenly gone and Mozilla +Crash Reporter window is opend, a process that crashed is the main process. + +.. image:: img/crashreporter.png + +When a crash happens, if you see the following page, a process that crashed +is a tab (content) process. + +.. image:: img/tabcrashed.png + +There are more process types, but there isn't an easy way to detect a crash in +a process of those types because the symptom varies. If you cannot be sure +about the type of a crashing process, terminate Firefox and launch a new +instance of Firefox under a debugger in the way described above. + +If a GPU process crashes, you may see a window is not rendered correctly as +below. Since the main process relaunches a GPU process, this symptom will be +transient and the window will be rendered correctly again. + +.. image:: img/crash-gpu.png + +If a GMP (= Gecko Media Plugin) process crashes, you will see an information +bar will be displayed below the address bar. + +.. image:: img/crash-gmp.png + +If an RDD (= Remote Data Decoder) process crashes, Firefox may stop playing +a video as below, but not limited to this symptom. + +.. image:: img/crash-rdd.png + + +Identify a process ID (PID) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you understand what type of process crashes, the next step is to get a +process ID (PID), which is a value to specify in the debugger command we +discussed above. We present two ways to get a PID here. + +The first way is to use Firefox itself. Open a new tab and go to the +**about:processes** page. This page shows the list of all processes and their +PIDs. In the example below, the PID of the main process is **6308** and the +PID of the tab process hosting a page of mozilla.org is **6748**. + +.. image:: img/about-processes.png + +The second way is to use `Process Explorer +<https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer>`__, +which is provided by Microsoft for free. You may need this technique to attach +a debugger to a hung process or when you cannot open a new tab in the existing +Firefox for some reason. + +Process Explorer is basically an advanced version of Task Manager. Since it +displays processes in a hierarchical tree view, you can easily locate the main +process, which is the parent of all child processes. In the example below, the +PID of the main process is **6308** and all processes, including +plugin-container.exe, under the main process is child processes. + +Another helpful feature of Process Explorer is that when you hover the mouse +cursor on a process, it displays a tooltip window showing the process's command +line string. For a child process of firefox.exe, the command line's last +keyword shows the process type, so you can tell which process is which process +type. In the example below, the tooltip window displays a command line string +of a GPU process. + +.. image:: img/process-explorer.png + + +How to find the location of firefox.exe +--------------------------------------- + +If you're not sure about the location of the executable file (firefox.exe) of +Firefox you run, you can find it in the **about:support** page. In the +"Application Basics" section, the path to firefox.exe is displayed in the row +of "Application Binary". + +.. image:: img/about-support.png + +.. |WinDbg in Start Menu| image:: img/windbg-in-startmenu.png + :width: 50% diff --git a/docs/contributing/debugging/debugging_a_hang_on_macos.rst b/docs/contributing/debugging/debugging_a_hang_on_macos.rst new file mode 100644 index 0000000000..ee2bed16b2 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_hang_on_macos.rst @@ -0,0 +1,10 @@ +Debugging A Hang On macOS +========================= + +See `How to Report a Hung +Firefox <https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox>`_. + +See also +~~~~~~~~ + +`Debugging on macOS <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_macOS>`__ diff --git a/docs/contributing/debugging/debugging_a_minidump.rst b/docs/contributing/debugging/debugging_a_minidump.rst new file mode 100644 index 0000000000..5638c29d85 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_minidump.rst @@ -0,0 +1,202 @@ +Debugging A Minidump +==================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +The +`minidump <http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx>`__ +file format contains data about a crash on Windows. It is used by +`rust-minidump <https://github.com/luser/rust-minidump>`__, +`Breakpad <https://wiki.mozilla.org/Breakpad>`__, and also by various +Windows debugging tools. Each minidump includes the following data. + +- Details about the exception which led to the crash. +- Information about each thread in the process: the address which was + executing and the register state at the time the process stopped. +- A list of shared libraries loaded into the process at the time of the + crash. +- The stack memory of each thread. +- The memory right around the crashing address. +- (Optional) Other memory regions, if requested by the application. +- (Optional) Other platform-specific data. + +Accessing minidumps from crash reports +-------------------------------------- + +Minidumps are not available to everyone. For details on how to gain +access and where to find minidump files for crash reports, consult the +:ref:`crash report documentation <Understanding Crash Reports>` + +Using rust-minidump's tooling +----------------------------------- + +Most of our crash-reporting infrastructure is based on rust-minidump. +The primary tool for this is the +`minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ +CLI application, which includes extensive user documentation. + +That documentation includes a +`dedicated section <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk#analyzing-firefox-minidumps>`__ +on locally analyzing Firefox crashreports and minidumps. + +If you're looking for minidump_dump, it's included as part of +minidump-stackwalk. + +Using the MS Visual Studio debugger +----------------------------------- + +#. Set up the debugger to :ref:`use the Mozilla symbol + server <Using The Mozilla Symbol Server>` and + :ref:`source server <Using The Mozilla Source Server>`. +#. Double-click on the minidump file to open it in the debugger. +#. When it loads, click the green icon in the visual studio debugger + toolbar that looks like a play button. + +For Firefox releases older than Firefox 41, you will also need to +install the relevant release of Firefox (for example from +`here <https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/>`__), +and add the directory it is in (e.g., "C:\Program Files\Mozilla +Firefox 3.6 Beta 1\") to the same dialog in which you set up the +symbol server (in case the binary location in the minidump is not the +same as the one on your machine). Note that you can install the +relevant release anywhere. Just make sure to configure the symbol +server to the directory where you installed it. For releases from 41 +onward, the binaries are available on the symbol server. + +If this doesn't work, downloading the exact build and crashreporter +symbols full files. These can be found in treeherder / build folder. +Load Visual Studio, and go to file -> open -> minidump location. Click +on "Run Native", and Visual Studio will ask for the corresponding symbol +files. For each .dll you wish to have symbols for, you must go to a +console and go to the corresponding directory. E.g. (xul.dll should go +to xul.pdf in the crashreporter symbols directory). Each directory will +have a .pd\_ file. In a command shell run: "expand /r foo.pd\_". Then +point Visual Studio to this directory. + +Then you'll be able to examine: + ++------------------+-------------------------------------------------------------------------+ +| stack trace | The debugger shows the stack trace. You can right-click on any frame | +| | in the stack, and then choose "go to disassembly" or "go to source". | +| | (Choosing "go to disassembly" from the source view might not get you | +| | to the right place due to optimizations.) When looking at the | +| | source, beware that the debugging information will associate all | +| | inlined functions as part of the line into which they were inlined, | +| | and compiler (with PGO) inlines *very* aggressively (including | +| | inlining virtual functions). You can often figure out where you | +| | *really* are by reading the disassembly. | ++------------------+-------------------------------------------------------------------------+ +| registers | In the Registers tab (Debug->Windows->Registers if you don't have | +| | it open), you can look at the registers associated with each stack | +| | frame, but only at the current state (i.e., the time of the crash). | +| | Registers that Visual Studio can't figure out will be grayed-out and | +| | have the value 00000000. | ++------------------+-------------------------------------------------------------------------+ +| stack memory | You open a window (Memory 1, etc.) that shows contiguous segments of | +| | memory using the Debug->Windows->Memory menu item. You can then | +| | enter the address of the stack pointer (ESP register) in this window | +| | and look at the memory on the stack. (The minidump doesn't have the | +| | memory on the heap.) It's a good idea to change the "width" dropdown | +| | in the top right corner of the window from its default "Auto" to | +| | either "8" or "16" so that the memory display is word-aligned. If | +| | you're interested in pointers, which is usually the case, you can | +| | right click in this window and change the display to show 4-byte | +| | words (so that you don't have to reverse the order due to | +| | little-endianness). This view, combined with the disassembly, can | +| | often be used to reconstruct information beyond what in shown the | +| | function parameters. | ++------------------+-------------------------------------------------------------------------+ +| local variables | In the Watch 1 (etc.) window (which, if you don't have open, you can | +| | iget from Debug->Windows->Watch), you can type an expression | +| | (e.g., the name of a local variable) and the debugger will show you | +| | its value (although it sometimes gets confused). If you're looking | +| | at a pointer to a variable that happens to be on the stack, you can | +| | even examine member variables by just typing expressions. If Visual | +| | Studio can't figure something out from the minidump, it might show | +| | you 00000000 (is this true?). | ++------------------+-------------------------------------------------------------------------+ + +Using minidump-2-core on Linux +------------------------------ + +The `Breakpad +source <https://chromium.googlesource.com/breakpad/breakpad/+/master/>`__ +contains a tool called +`minidump-2-core <https://chromium.googlesource.com/breakpad/breakpad/+/master/src/tools/linux/md2core/>`__, +which converts Linux minidumps into core files. If you checkout and +build Breakpad, the binary will be at +``src/tools/linux/md2core/minidump-2-core``. Running the binary with the +path to a Linux minidump will generate a core file on stdout which can +then be loaded in gdb as usual. You will need to manually download the +matching Firefox binaries, but then you can use the :ref:`GDB Python +script <Downloading symbols on Linux / Mac OS X>` to download symbols. + +The ``minidump-2-core`` source does not currently handle processing +minidumps from a different CPU architecture than the system it was +built for. If you want to use it on an ARM dump, for example, you may +need to build the tool for ARM and run it under QEMU. + +Using other tools to inspect minidump data +------------------------------------------ + +Ted has a few tools that can be built against an already-built copy of +Breakpad to do more targeted inspection. All of these tools assume you +have checked out their source in a directory next to the breakpad +checkout, and that you have built Breakpad in an objdir named +``obj-breakpad`` at the same level. + +- `stackwalk-http <https://hg.mozilla.org/users/tmielczarek_mozilla.com/stackwalk-http/>`__ + is a version of the breakpad's minidump_stackwalk that can fetch symbols + over HTTP, and also has the Mozilla symbol server URL baked in. If you + run it like ``stackwalk /path/to/dmp /tmp/syms`` it will print the stack + trace and save the symbols it downloaded in ``/tmp/syms``. Note that + symbols are only uploaded to the symbol server for nightly and + release builds, not per-change builds. +- `dump-lookup <https://hg.mozilla.org/users/tmielczarek_mozilla.com/dump-lookup/>`__ + takes a minidump and prints values on the stack that are potential + return addresses. This is useful when a stack trace looks truncated + or otherwise wrong. It needs symbol files to produce useful output, + so you will generally want to have run ``stackwalk-http`` to download + them first. +- `get-minidump-instructions <https://hg.mozilla.org/users/tmielczarek_mozilla.com/get-minidump-instructions/>`__ + retrieves and displays the memory range surrounding the faulting + instruction pointer from a minidump. You will almost always want to + run it with the ``--disassemble`` option, which will make it send the + bytes through ``objdump`` to display the disassembled instructions. + If you also give it a path to symbols (see ``stackwalk-http`` above) + it can download the matching source files from hg.mozilla.org and + display source interleaved with the disassembly. +- `minidump-modules <http://hg.mozilla.org/users/tmielczarek_mozilla.com/minidump-modules>`__ + takes a minidump and prints the list of modules from the crashed + process. It will print the full path to each module, whereas the + Socorro UI only prints the filename for each module for privacy + reasons. It also accepts a -v option to print the debug ID for each + module, and a -d option to print relative paths to the symbol files + that would be used instead of the module filenames. + +Getting a stack trace from a crashed B2G process +------------------------------------------------ + +#. Get the minidump file in the phone at + /data/b2g/mozilla/\*.default/minidump/. You can use `adb + pull <http://developer.android.com/tools/help/adb.html>`__ for that. +#. Build the debug symbols using the command ./build.sh buildsymbols + inside the B2G tree. The symbol files will be generated in + $OBJDIR/dist/crashreporter-symbols. +#. Build and install + `google-breakpad <https://code.google.com/p/google-breakpad/>`__. +#. Use the + `minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ + tool to get the stack trace. + +.. code:: bash + + Example: + + $ cd B2G + $ adb pull /data/b2g/mozilla/*.default/minidump/*.dmp . + $ls *.dmp + 71788789-197e-d769-67167423-4e7aef32.dmp + $ minidump-stackwalk 71788789-197e-d769-67167423-4e7aef32.dmp objdir-debug/dist/crashreporter-symbols/ diff --git a/docs/contributing/debugging/debugging_firefox_with_gdb.rst b/docs/contributing/debugging/debugging_firefox_with_gdb.rst new file mode 100644 index 0000000000..9572ce8e18 --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_gdb.rst @@ -0,0 +1,303 @@ +Debugging Firefox with GDB +========================== + +This page details how you can more easily debug Firefox with gdb. :ref:`rr +<Debugging Firefox with rr>` is most often a better choice to debug a problem, +but sometimes it isn't possible to use it, such as attempting to reproduce a +race condition or when performance is important to reproduce an issue. ``rr`` +chaos mode allows reproducing a lot of issues though, and should be tried. + +Where can I find general gdb documentation? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using GDB is beyond the scope of this document. Documentation is likely +available on your system if you have GDB installed, in the form of +**info,** **man** pages, or the gnome help browser. Additionally, you +can use a graphical front-end to GDB like +`ddd <https://www.gnu.org/software/ddd/>`__ or +`insight <https://sourceware.org/insight/>`__. For more information see +https://sourceware.org/gdb/current/onlinedocs/gdb/ + +How to debug Firefox with gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Firefox is a multiprocess application, with a parent process, and several child +processes, each specialized and sandboxed differently. + +.. code:: bash + + $ ./mach run --debugger=gdb + +allows running Firefox in gdb, and debug the parent process. You can substitute +``run`` with another action, such as ``test``, ``mochitest``, ``xpcshell-test``, +etc. + +Debugging child processes can be done by attaching the debugger. + +.. code:: bash + + $ gdb --pid <pid> + +There's a number of ways to find the PID of a process: hovering over a tab for a +content process, opening ``about:processes``, using ``ps ... | grep firefox`` on +the command line, etc. + +Sometimes, it is desirable to attach to a child process at startup, to diagnose +something very close to the start of the process. Setting the environment +variable ``MOZ_DEBUG_CHILD_PROCESS=10`` will make each new process print an few +informative lines, including the process type and its PID. The process with then +sleep for a number of seconds equal to the value of the environment variable, +allowing to attach a debugger. + +.. code:: bash + + $ MOZ_DEBUG_CHILD_PROCESS=10 ./mach run + ... + ... + ... + + CHILDCHILDCHILDCHILD (process type tab) + debug me @ 65230 + + ... + ... + ... + +Attaching gdb to Firefox might fail on Linux distributions that enable common +kernel hardening features such as the Yama security module. If you encounter the +following error when attaching: + +.. code:: bash + + $ gdb --pid <pid> + ... + ... + + Attaching to process <pid> + ptrace: Operation not permitted. + +Check the contents of `/proc/sys/kernel/yama/ptrace_scope`. If it set to `1` +you won't be able to attach to processes, set it to `0` from a root shell: + +.. code:: bash + + \# echo 0 > /proc/sys/kernel/yama/ptrace_scope + +If you still can't attach check your setup carefully. Do not, under any +circumstances, run gdb as root. Since gdb can execute arbitrary code and spawn +shells it can be extremely dangerous to use it with root permissions. + + +Advanced gdb configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The preferred method, is using the +:ref:`mach` command-line tool to run the +debugger, which can bypass several optional defaults. Use "mach help +run" to get more details. If inside the source directory, you would use +"./mach". Please note that +:ref:`mach is aware of mozconfigs <Configuring Build Options>`. + +.. code:: bash + + $ ./mach run --debug [arguments to pass to firefox] + +If you need to direct arguments to gdb, you can use '--debugger-args' +options via the command line parser, taking care to adhere to shell +splitting rules. For example, if you wanted to run the command 'show +args' when gdb starts, you would use: + +.. code:: bash + + $ ./mach run --debug --debugger-args "-ex 'show args'" + +Alternatively, you can run gdb directly against Firefox. However, you +won't get some of the more useful capabilities this way. For example, +mach sets an environment variable (see below) to stop the JS engine from +generating synthetic segfaults to support the slower script dialoging +mechanism. + +.. code:: + + (gdb) $OBJDIR/dist/bin/firefox + +How to debug a Firefox in the field (not compiled on the host) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you need to attach to a Firefox process live on a machine, and this Firefox +was built by Mozilla, or by certain Linux distros, it's possible to get symbols +and sources using the Mozilla symbol server, see :ref:`this section <Downloading +symbols on Linux / Mac OS X>` for setup instructions, it's just a matter of +sourcing a python script in ``.gdbinit``. + +Debugging then works as usual, except the build probably has a very high +optimization level. + +How do I pass arguments in prun? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the arguments in GDB before calling prun. Here's an example on how +to do that: + +.. code:: + + (gdb) set args https://www.mozilla.org + (gdb) prun + +Why breakpoints seem to not be hit? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most likely cause is that `gdb` hasn't been attached to the process in which +the code to diagnose is ran. Enabling the relevant `MOZ_LOG` modules can help, +since by default it prints the process type and pid of all logging statements. + +`break list` will display a list of breakpoints, and whether or not they're +enabled. C++ namespaces need to be specified entirely, and it's sometimes hard +to break in lambda. Breaking by line number is an alternative strategy that +often works in this case. + +How do I display an nsString? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + (gdb) p ToNewCString(string); + +This leaks a bit of memory but it doesn't really matter. + +How do I determine the concrete type of an object pointed to by an interface pointer? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can determine the concrete type of any object pointed to, by an +XPCOM interface pointer, by looking at the mangled name of the symbol +for the object's vtable: + +.. code:: + + (gdb) p aKidFrame + $1 = (nsIFrame *) 0x85058d4 + (gdb) x/wa *(void**)aKidFrame + 0x4210d380 <__vt_14nsRootBoxFrame>: 0x0 + (gdb) p *(nsRootBoxFrame*)aKidFrame + [ all the member variables of aKidFrame ] + +Or use the gdb command ``set print object on``. + +How can I debug JavaScript from gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have JavaScript Engine code on the stack, you'll probably want a +JS stack in addition to the C++ stack. + +.. code:: + + (gdb) call DumpJSStack() + +Please note that if `gdb` has been attached to a process, the stack might be +printed in the terminal window in which Firefox was started. + +See +`this MDN page +<https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_JavaScript>`__ +for more JS debugging tricks. + +How can I debug race conditions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Try :ref:`rr <Debugging Firefox with rr>` first. If this doesn't work, good +luck, maybe try :ref:`logging <Gecko Logging>` or sprinkling assertions. + +I keep getting a SIGSYS, or SIGSEGV in JS/JIT code under gdb even though there is no crash when gdb is not attached. How do I fix it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Allow gdb to read mozilla-central's .gdbinit, located at `build/.gdbinit`. In +your own `.gdbinit`, add the line: + + .. code:: + + add-auto-load-safe-path /path/to/mozilla-central + +How do I get useful stack traces inside system libraries? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many Linux distributions provide separate packages with debugging +information for system libraries, such as gdb, Valgrind, profiling +tools, etc., to give useful stack traces via system libraries. + +The modern way to do this is to enable ``debuginfod``. This can be done by adding: + + .. code:: + + set debuginfod enabled on + +in your ``.gdbinit``, but there might be distro-specific instructions. +Alternatively, you can install the packages that contain the debug symbols for +the libraries you want to debug. + +When using ``debuginfod``, the correct information will be downloaded +automatically when needed (and subsequently cached). + +If you're not sure what to use, there's a federated debuginfod server that +provides debug information for most mainstream distributions. You can use it +by adding the following line to your ``.gdbinit`` file: + + .. code:: + + set debuginfod urls "https://debuginfod.elfutils.org/" + +Keep in mind that it might take a while to download debug information the +very first time. This queries all the servers of multiple distributions +sequentially and debug information tends to be large. It will be cached for the +next run though. + +Fedora +^^^^^^ + +On Fedora, you need to enable the debuginfo repositories, as the +packages are in separate repositories. Enable them permanently, so when +you get updates you also get security updates for these packages. A way +to do this is edit ``/etc/yum.repos.d/fedora.repo`` and +``fedora-updates.repo`` to change the ``enabled=0`` line in the +debuginfo section to ``enabled=1``. This may then flag a conflict when +upgrading to a new distribution version. You would the need to perform +this edit again. + +You can finally install debuginfo packages with yum or other package +management tools. The best way is install the ``yum-utils`` package, and +then use the ``debuginfo-install`` command to install all the debuginfo: + +.. code:: bash + + $ yum install yum-utils + $ debuginfo-install firefox + +This can be done manually using: + +.. code:: bash + + $ yum install GConf2-debuginfo ORBit2-debuginfo atk-debuginfo \ + cairo-debuginfo dbus-debuginfo expat-debuginfo \ + fontconfig-debuginfo freetype-debuginfo gcc-debuginfo glib2-debuginfo \ + glibc-debuginfo gnome-vfs2-debuginfo gtk2-debuginfo gtk2-engines-debuginfo \ + hal-debuginfo libX11-debuginfo libXcursor-debuginfo libXext-debuginfo \ + libXfixes-debuginfo libXft-debuginfo libXi-debuginfo libXinerama-debuginfo \ + libXrender-debuginfo libbonobo-debuginfo libgnome-debuginfo \ + libselinux-debuginfo pango-debuginfo popt-debuginfo scim-bridge-debuginfo + +Disabling multiprocess +~~~~~~~~~~~~~~~~~~~~~~ + +``mach run`` and ``mach test`` both accept a ``--disable-e10s`` argument. Some +debuggers can't catch child-process crashes without it. This is sometimes a +viable alternative to attaching, but these days it changes enough thing that +it's not always a usable option. + +See also +~~~~~~~~~ + + +- `Mike Conley's blog post <https://mikeconley.ca/blog/2014/04/25/electrolysis-debugging-child-processes-of-content-for-make-benefit-glorious-browser-of-firefox>`__ +- `Performance tools <https://wiki.mozilla.org/Performance:Tools>`__ +- `Fun with + gdb <https://blog.mozilla.com/sfink/2011/02/22/fun-with-gdb/>`__ by + Steve Fink diff --git a/docs/contributing/debugging/debugging_firefox_with_lldb.rst b/docs/contributing/debugging/debugging_firefox_with_lldb.rst new file mode 100644 index 0000000000..88378bcc4a --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_lldb.rst @@ -0,0 +1,80 @@ +Debugging Firefox with LLDB +=========================== + +See http://lldb.llvm.org/index.html. + +Mozilla-specific lldb settings +------------------------------ + +There's an +``.lldbinit`` `file <https://searchfox.org/mozilla-central/source/.lldbinit>`_ +in the Mozilla source tree, which applies recommended settings and +includes a few type summaries and Mozilla-specific debugging commands +via the lldbutils module (see +`python/lldbutils/README.txt <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__). +For information about available features see the links above and the `Using +LLDB to debug Gecko <http://mcc.id.au/blog/2014/01/lldb-gecko>`__ blog +post. + +The in-tree ``.lldbinit`` should be loaded automatically in most cases +when running lldb from the command line (e.g. using +:ref:`mach`), but **not** +when using Xcode. See :ref:`Debugging on macOS` for information on setting up +Xcode. + +.. warning:: + + LLDB warning: Xcode 5 only comes with lldb (gdb is gone). The + introduction and use of UNIFIED_SOURCES in the source starting around + November 2013 has broken the default LLDB configuration so that it + will not manage to resolve breakpoints in files that are build using + UNIFIED_SOURCES (the breakpoints will be listed as "pending", and + lldb will not stop at them). To fix this add the following to your + $HOME/.lldbinit file: + + .. code:: + + # Mozilla's use of UNIFIED_SOURCES to include multiple source files into a + # single compiled file breaks lldb breakpoint setting. This works around that. + # See http://lldb.llvm.org/troubleshooting.html for more. + settings set target.inline-breakpoint-strategy always + + Restart Xcode/lldb and restart your debugging session. If that still + doesn't fix things then try closing Xcode/lldb, doing a clobber + build, reopening Xcode/lldb, and restarting your debugging session. + +Starting a debugging session +---------------------------- + +Attaching to an existing process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can attach to Firefox with following command: + +.. code:: + + (lldb) process attach --name firefox + +Some versions of lldb causes crashes after attaching to Firefox. + +Running a new process +~~~~~~~~~~~~~~~~~~~~~ + +To start Firefox under the debugger, run ``lldb`` followed by "--", +followed by the command line you'd like to run, like this: + +.. code:: bash + + $ lldb -- obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox -no-remote -profile /path/to/profile + +Then set breakpoints you need and start the process: + +.. code:: + + (lldb) breakpoint set --name nsInProcessTabChildGlobal::InitTabChildGlobal + Breakpoint created: 1: name = 'nsInProcessTabChildGlobal::InitTabChildGlobal', locations = 0 (pending) + WARNING: Unable to resolve breakpoint to any actual locations. + + (lldb) r + Process 7602 launched: '/.../obj-ff-opt/dist/Nightly.app/Contents/MacOS/firefox' (x86_64) + 1 location added to breakpoint 1 diff --git a/docs/contributing/debugging/debugging_firefox_with_rr.rst b/docs/contributing/debugging/debugging_firefox_with_rr.rst new file mode 100644 index 0000000000..ef68d9c6ea --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_rr.rst @@ -0,0 +1,98 @@ +Debugging Firefox with rr +========================= + +This page is intended to help Firefox/Gecko developers get started using rr to debug Firefox. + +Prerequisites +------------- + +You must have Linux installed with a recent kernel. If you're not running Linux already, an option is set up a virtual machine in which to record Firefox. Be forewarned though that + + * rr requires a VM hypervisor that virtualizes CPU performance counters. VMWare Workstation supports that. + * there's a 20% or so performance hit from running in a VM; generally speaking recorder overhead increases from ~1.2x to ~1.4x. (It's a feather in the cap of the hypervisor authors that the hit is that small, though!) + * Some features (reverse execution) `may not work well in VMWare <https://robert.ocallahan.org/2014/09/vmware-cpuid-conditional-branch.html>`__ due to a VMWare optimization that can be disabled `this way <http://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html>`__. + +When using VSCode, consider adding the `Midas <https://github.com/farre/midas>`__ addon to debug rr traces from the editor. You can use Midas to install / build rr and to configure the :code:`auto-load safe path` via the setup commands. + +Ensure that you've `installed <http://rr-project.org/>`__ or `built <https://github.com/mozilla/rr/wiki/Building-And-Installing>`__ rr and have `used it successfully <https://github.com/mozilla/rr/wiki/Usage>`__. Check that you have the latest version. + +You likely need to configure your :code:`auto-load safe path` for rr / gdb to work correctly. If you are on Linux and your builds are in ~/moz, then add the following to ~/.gdbinit + +.. code:: + + add-auto-load-safe-path ~/moz + +Firefox developers are strongly encouraged to build rr from source. If your Firefox patch triggers a bug in rr, rr developers will fix that bug with high priority. You might be able to pull a fix within a few hours or days instead of waiting for the next release. + +Recording Firefox +----------------- + + +To record Firefox running normally, simply launch it under rr as you would if running it under valgrind or gdb + +.. code:: bash + + $ rr $ff-objdir/dist/bin/firefox ... + +or use mach + +.. code:: bash + + $ ./mach run --debugger=rr + +This will save a trace to your working directory as described in the `usage instructions <https://github.com/mozilla/rr/wiki/Usage>`__. Please refer to `those instructions <https://github.com/mozilla/rr/wiki/Usage>`__ for details on how to debug the recording, which isn't covered in this document. + +Sandboxing reduces recording performance because of the SIGSYS signals and extra syscall. Disabling it can help. + +The background hang monitor might also be making things worse by causing a lot of extra syscalls. It can be disabled by setting +toolkit.content-background-hang-monitor.disabled=true. + +SIGSYS +------ + +When recording and replaying Firefox running with the Linux sandbox, you will get SIGSYS signals frequently. This is expected behavior caused by the sandbox. In gdb, use handle SIGSYS noprint nostop to suppress the signals. + +Recording test suites +--------------------- + +You can use the test runners' --debugger feature to punch rr down through the layers of python script to where Firefox is launched. This is used in the same way you would use --debugger to run valgrind or gdb, for example: + +.. code:: bash + + $ ./mach mochitest --debugger=rr ... + +The test harnesses disable the slow-script timeout when the --debugger argument is passed. That's usually sensible, because you don't want those warnings being generated while Firefox is stopped in gdb. However, this has been `observed to change Gecko behavior <https://bugzilla.mozilla.org/show_bug.cgi?id=986673>`__. rr doesn't need to have the slow-script timeout disabled, so to avoid those kinds of pitfalls, pass the --slowscript argument to the test harness. + +To run rr in chaos mode: + +.. code:: bash + + $ ./mach mochitest --debugger=rr --debugger-args="record --chaos" + +You can also run the entire test harness in rr: + +.. code:: bash + + $ rr ./mach mochitest ... + +The trace will contain many processes, so to debug the correct one, you'll want to use rr ps or rr replay -p firefox etc. + +Working with multiple processes +------------------------------ + +rr should work out of the box with multi-process Firefox. Once you have a recording you can use rr ps to show all the process that were recorded and rr replay -p <pid> to attach to a particular process. + +If you want to debug a particular part of code, you can use the :code:`MOZ_DBG` macro and :code:`getpid()` function to write the process id to stderr. `MOZ_LOG <https://firefox-source-docs.mozilla.org/xpcom/logging.html>`__ will include the pid in log messages by default. + +You can combine that with the -M and -g flags to jump to a particular point in a particular process's lifetime. + +Get help! +--------- + +If you encounter a problem with rr, please `file an issue <https://github.com/mozilla/rr/issues>`__. Firefox bugs are high priority, so usually your issue can be fixed very quickly. + +If you want to chat with rr developers, because you need more help or want to contribute or want to complain, we hang out in the `#rr channel <https://chat.mozilla.org/#/room/#rr:mozilla.org>`__. There is also a channel for `#midas <https://chat.mozilla.org/#/room/#midas:mozilla.org>`__. + +You also may find `these debugging protips <https://github.com/mozilla/rr/wiki/Debugging-protips>`__ helpful, though many are for rr developers, not users. + +Happy debugging! diff --git a/docs/contributing/debugging/debugging_firefox_with_valgrind.rst b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst new file mode 100644 index 0000000000..33aab3638a --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst @@ -0,0 +1,177 @@ +Debugging Firefox with Valgrind +=============================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This page describes how to use Valgrind (specifically, its Memcheck +tool) to find memory errors. + +Supported platforms +------------------- + +Valgrind runs desktop Firefox fine on Linux, especially on x86 and +x86-64. Firefox for Android and Firefox OS on ARMv7 should also run, +though perhaps not as smoothly. The other architectures supported by +Valgrind on Linux (AARCH64, PPC{32,64}, MIPS{32,64}, S390X) should also +work, in theory. + +MacOS X 10.10 (Yosemite), 64-bit only, works, although it can be a bit +of a rough ride. + +- Expect lower performance and a somewhat higher false positive error + rate than on Linux. +- Valgrind's handling of malloc zones on Yosemite is imperfect. Regard + leak reports with caution. +- Valgrind has been known to cause kernel panics, for unknown reasons. + +Where to get Valgrind +--------------------- + +Linux: Download `Valgrind <https://valgrind.org/>`__ directly, or use +your distribution's package manager (if it has a recent enough version). + +MacOSX: `Get Valgrind trunk from +SVN <https://valgrind.org/downloads/repository.html>`__ and build it. +Don't use 3.10.x or any other tarball. + +Make sure you have Valgrind 3.14 or later, version 3.16.1 is known to work, +3.13.0 did not. Newer versions tend to have better compatibility with both +Firefox's JITs and newer toolchain components (compiler, libc and linker +versions). + +Basics +------ + +Build +~~~~~ + +Build Firefox with the following options, which maximize speed and +accuracy. + +.. code:: + + ac_add_options --disable-jemalloc + ac_add_options --disable-strip + ac_add_options --enable-valgrind + ac_add_options --enable-optimize="-g -O2" + ac_add_options --disable-sandbox + +Run +~~~ + +Note that programs run *much* more slowly under Valgrind than they do +natively. Slow-downs of 20x or 30x aren't unexpected, and it's slower on +Mac than on Linux. Don't try this on an underpowered machine. + +Linux +^^^^^ + +On Linux, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --read-inline-info=yes + +The ``--smc-check`` and ``--vex-iropt-register-updates`` options are +necessary to avoid crashes in JIT-generated code. + +The ``--show-mismatched-frees`` option is necessary due to inconsistent +inlining of ``new`` and ``delete`` -- i.e. one gets inlined but the +other doesn't -- which lead to false-positive mismatched-free errors. + +The ``--read-inline-info`` option improves stack trace readability in +the presence of inlining. + +Also, run with the following environment variable set. + +.. code:: + + G_SLICE=always-malloc + +This is necessary to get the Gnome system libraries to use plain +``malloc`` instead of pool allocators. + +Mac +^^^ + +On Mac, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --dsymutil=yes + +The ``--dsymutil`` option ensures line number information is present in +stack traces. + +Advanced usage +-------------- + +Shared suppression files +~~~~~~~~~~~~~~~~~~~~~~~~ + +`/build/valgrind/ <https://searchfox.org/mozilla-central/source/build/valgrind/>`__ +contains the suppression files used by the periodic Valgrind jobs on +Treeherder. Some of these files are platform-specific. + +Running mochitests under Valgrind? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To run a mochitest under Valgrind, use the following command. + +.. code:: bash + + $ ./mach mochitest-plain --debugger="valgrind" --debugger-args="$VALGRIND_OPTIONS" relative/path/to/tests + +Where ``$VALGRIND_OPTIONS`` are the options described +:ref:`above <Debugging Firefox With Valgrind>`. You might also +need ``--trace-children=yes`` to trace into child processes. + +As of December 2014 it is possible to do a complete run of +mochitests-plain on Valgrind in about 8 CPU hours on a Core i4910 +(Haswell) machine. Maximum process size is 5.4G, of which about 80% is +in memory. Runs of small subsets of mochitests take far less memory. + +Bits and pieces +~~~~~~~~~~~~~~~ + +For un-released Linux distros (Fedora Rawhide, etc.) you'll need to use +a version of Valgrind trunk build, because fixes for the latest gcc and +glibc versions appear there first. Without them you'll be flooded with +false errors from Memcheck, and have debuginfo reading problems. + +On Linux, code compiled by LLVM at high optimisation levels can cause +Memcheck to report false uninitialised value errors. See +`here <https://bugs.kde.org/show_bug.cgi?id=242137#c3>`__ for an easy +workaround. On Mac, Valgrind has this workaround built in. + +You can make stack traces easier to read by asking for source file names +to be given relative to the root of your source tree. Do this by using +``--fullpath-after=`` to specify the rightmost part of the absolute path +that you don't want to see. For example, if your source tree is rooted +at ``/home/sewardj/MC-20-12-2014``, use ``--fullpath-after=2014/`` to +get path names relative to the source directory. + +The ``--track-origins=yes`` slows down Valgrind greatly, so don't use it +unless you are hunting down a specific uninitialised value error. But if +you are hunting down such an error, it's extremely helpful and worth +waiting for. + +Additional help +--------------- + +The `Valgrind Quick Start +Guide <http://www.valgrind.org/docs/manual/quick-start.html>`__ is short +and worth reading. The `User +Manual <https://valgrind.org/docs/manual/manual.html>`__ is also useful. + +If Valgrind asserts, crashes, doesn't do what you expect, or otherwise +acts up, first of all read this page and make sure you have both Firefox +and Valgrind correctly configured. If that's all OK, try using the +`Valgrind trunk from +SVN <http://www.valgrind.org/downloads/repository.html>`__. Oftentimes +bugs are fixed in the trunk before most users fall across them. If that +doesn't help, consider `filing a bug +report <http://www.valgrind.org/support/bug_reports.html>`__, and/or +mailing Julian Seward or Nick Nethercote. diff --git a/docs/contributing/debugging/debugging_on_macos.rst b/docs/contributing/debugging/debugging_on_macos.rst new file mode 100644 index 0000000000..036d5df550 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_macos.rst @@ -0,0 +1,356 @@ +Debugging On macOS +================== + +This document explains how to debug Gecko-based applications such as +Firefox, Thunderbird, and SeaMonkey on macOS using Xcode. If you want to +debug from the terminal see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. For specific +information on a way to debug hangs, see :ref:`Debugging a hang on macOS <Debugging A Hang On macOS>`. + +Creating a debuggable build +--------------------------- + +First, you need to build the application you're going to debug using +this in your .mozconfig + +.. code:: + + ac_add_options --disable-optimize + ac_add_options --enable-debug-symbols + +you can also add this flag if you want assertions etc. compiled in + +.. code:: + + ac_add_options --enable-debug + +See :ref:`Building Firefox for macOS <Building Firefox On MacOS>` +if you need help creating your own build. + +Debugging Firefox on macOS 10.14+ +--------------------------------- + +macOS 10.14 introduced Notarization and Hardened Runtime features for +improved application security. macOS 10.15 went further, requiring +applications to be Notarized with Hardened Runtime enabled in order to +launch (ignoring workarounds). When run on earlier macOS versions, +Notarization and Hardened Runtime settings have no effect. + +Official Builds +~~~~~~~~~~~~~~~ + +At this time, official builds of Firefox 69 and later are Notarized. +**As a result, it is not possible to attach a debugger to these official +Firefox releases on macOS 10.14+ without disabling System Integrity +Protection (SIP).** This is due to Notarization requiring Hardened +Runtime to be enabled with the ``com.apple.security.get-task-allow`` +entitlement disallowed. **Rather than disabling SIP (which has security +implications), it is recommended to debug with try builds or local +builds. The differences are explained below.** + +try Server Builds +~~~~~~~~~~~~~~~~~ + +In most cases, developers needing to debug a build as close as possible +to the production environment should use a :ref:`try +build <Pushing to Try>`. These +builds enable Hardened Runtime and only differ from production builds in +that they are not Notarized which should not otherwise affect +functionality, (other than the ability to easily launch the browser on +macOS 10.15+ -- see quarantine note below). At this time, developers can +obtain a Hardened Runtime build with the +``com.apple.security.get-task-allow`` entitlement allowed by submitting +a try build and downloading the dmg generated by the "Rpk" shippable +build job. A debugger can be attached to Firefox processes of these +builds. try builds use the developer entitlement files from the +source tree (allowing debugger attach) while production builds use +the production versions (which must meet notarization requirements). +**On macOS 10.15+, downloaded try builds will not launch by default +because Notarization is required. To workaround this problem, remove the +quarantine extended attribute from the downloaded Nightly:** + + ``$ xattr -r -d com.apple.quarantine /Path/to/Nightly.app`` + +Local Builds +~~~~~~~~~~~~ + +Local builds of mozilla-central do not enable Hardened Runtime and hence +do not have debugging restrictions. As a result, some functionality will +be permitted on local builds, but blocked on production builds which +have Hardened Runtime enabled. `Bug +1522409 <https://bugzilla.mozilla.org/show_bug.cgi?id=1522409>`__ was +filed to automate codesigning local builds to enable Hardened Runtime by +default and eliminate this discrepancy. + +Disabling System Integrity Protection (SIP) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If debugging a production build is required, follow Apple's documented +steps for disabling System Integrity Protection (SIP). Note that +disabling SIP bypasses Hardened Runtime restrictions which can mask some +bugs that only occur with Hardened Runtime so it is recommended to test +fixes with SIP enabled. **Disabling SIP has system security implications +that should be understood before taking this step.** + +Creating an Xcode project +------------------------- + +If you try to create a new Xcode project in an existing directory +then Xcode will delete its existing contents (Xcode will warn you +beforehand). To work around that, the steps below have you initialize +the project outside the Mozilla source tree, close the project, copy +the .xcodeproj project "file" into the source tree, and then reopen +the project to finish setting it up. + +Note also that since Xcode 7.3.1 it doesn't seem to be possible to +have the Xcode project live outside the source tree. If you try to do +that then Xcode will simply **copy** the source files under the +project directory rather than link to them which breaks debugging and the +possibility to modify-rebuild-relaunch from inside Xcode. + +These steps were last updated for Xcode 15: + +#. Open Xcode, and create a new Project with File > New > Project. + Select the "Other" tab then select the "Empty" project type, then + click Next. Name the project and click Next again. Xcode will refuse + to create a new project in a non-empty directory, so create/select + a temporary directory to contain the project files, untick "Create + Git repository" and then click Create. +#. Before going any further, close the project (File > Close Project). + Now open Finder, find the \*.xcodejproj directory in the temporary + directory, move it into the root directory of your Mozilla source + tree, then double-click on it to reopen it. (You can now delete the + temporary directory.) +#. In the left-hand pane in Xcode you should see a tree item where the + root item has the project name. Now, right click on the root item, + select 'Add files to "<project-name>"', select all the files and + directories in your source directory, untick "Copy items if needed", + then click Add. (These will then be progressively added under the + root item <project-name> in the left-hand pane. Note that + subdirectories may initially appear to be empty, but they too will + progressively be populated as Xcode processes the sourse files. + Once done, you should be able to open any file quickly by hitting + Cmd-Shift-O and typing in the name of a file.) +#. In the Product menu, select Scheme > New Scheme and name your scheme + (for example, "Debug"). After you click OK, Xcode should open the + settings window for the new scheme. (If not, then open its settings + from the Product > Edit Scheme menu.) +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on "None" and + selecting "Other...". A new dialog titled "Choose an executable to + launch" will pop up. Browse to the ``.app`` file that you want to + debug (``Firefox.app``, ``Nightly``\ ``Debug.app`` etc). The ``.app`` + file is typically found inside the ``dist`` folder in your build + directory. +#. If you are debugging Firefox, Thunderbird, or some other application + that supports multiple profiles, using a separate profile for + debugging purposes is recommended. See "Having a profile for + debugging purposes" below. Select the "Arguments" tab in the scheme + editor, and click the '+' below the "Arguments passed on launch" + field. Add "-P *profilename*", where *profilename* is the name of a + profile you created previously. Repeat that to also add the argument + "-no-remote". +#. Also in the "Arguments" panel, you may want to add an environment + variable MOZ_DEBUG_CHILD_PROCESS set to the value 1 to help with + debugging e10s. +#. Select "Build" from the left of the scheme editor window, and check + that there is nothing listed under Targets (otherwise it may cause + problems when you try to run the executable for debugging since you + will get build errors). +#. Click "Close" to close the scheme editor. + +At this point you can run and debug the Mozilla application from Xcode +(if the application doesn't appear to open, check whether it opened in +the background). When it hits breakpoints, Xcode should open the +correct source file at the correct line. + +Setting up lldb +--------------- + +``lldb`` is the debugger Xcode provides/uses. + +.. warning:: + + One important issue that the Mozilla .lldbinit file fixes is that by + default some breakpoints will be listed as "pending", and Xcode will + not stop at them. If you don't include the Mozilla's .lldbinit, you + must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Firefox with + lldb <Debugging Firefox with LLDB>`. + +The +`.lldbinit <http://searchfox.org/mozilla-central/source/.lldbinit>`__ +file in the source tree imports many useful `Mozilla specific lldb +settings, commands and +formatters <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__ +into ``lldb``, but you may need to take one of the following steps to +make sure this file is used. + +If you are using ``lldb`` on the command line (independently of Xcode) +and you will always run it from either the top source directory, the +object directory or else the dist/bin subdirectory of the object +directory, then adding the following setting to your ``$HOME/.lldbinit`` +is sufficient: + +:: + + settings set target.load-cwd-lldbinit true + +*However*, if you will run lldb from a different directory, or if you +will be running it indirectly by debugging in Xcode (Xcode always runs +lldb from "/"), then this setting will not help you. Instead, add the +following to your ``$HOME/.lldbinit``: + +:: + + # This automatically sources the Mozilla project's .lldbinit as soon as lldb + # starts or attaches to a Mozilla app (that's in an object directory). + # + # This is mainly a workaround for Xcode not providing a way to specify that + # lldb should be run from a given directory. (Xcode always runs lldb from "/", + # regardless of what directory Xcode was started from, and regardless of the + # value of the "Custom working directory" field in the Scheme's Run options. + # Therefore setting `settings set target.load-cwd-lldbinit true` can't help us + # without Xcode providing that functionality.) + # + # The following works by setting a one-shot breakpoint to break on a function + # that we know will both run early (which we want when we start first start the + # app) and run frequently (which we want so that it will trigger ASAP if we + # attach to an already running app). The breakpoint runs some commands to + # figure out the object directory path from the attached target and then + # sources the .lldbinit from there. + # + # NOTE: This scripts actions take a few seconds to complete, so the custom + # formatters, commands etc. that are added may not be immediately available. + # + breakpoint set --name nsThread::ProcessNextEvent --thread-index 1 --auto-continue true --one-shot true + breakpoint command add -s python + # This script that we run does not work if we try to use the global 'lldb' + # object, since it is out of date at the time that the script runs (for + # example, `lldb.target.executable.fullpath` is empty). Therefore we must + # get the following objects from the 'frame' object. + target = frame.GetThread().GetProcess().GetTarget() + debugger = target.GetDebugger() + + # Delete our breakpoint (not actually necessary with `--one-shot true`): + target.BreakpointDelete(bp_loc.GetBreakpoint().GetID()) + + # For completeness, find and delete the dummy breakpoint (the breakpoint + # lldb creates when it can't initially find the method to set the + # breakpoint on): + # BUG WORKAROUND! GetID() on the *dummy* breakpoint appears to be returning + # the breakpoint index instead of its ID. We have to add 1 to correct for + # that! :-( + dummy_bp_list = lldb.SBBreakpointList(target) + debugger.GetDummyTarget().FindBreakpointsByName("nsThread::ProcessNextEvent", dummy_bp_list) + dummy_bp_id = dummy_bp_list.GetBreakpointAtIndex(0).GetID() + 1 + debugger.GetDummyTarget().BreakpointDelete(dummy_bp_id) + + # "source" the Mozilla project .lldbinit: + os.chdir(target.executable.fullpath.split("/dist/")[0]) + debugger.HandleCommand("command source -s true " + os.path.join(os.getcwd(), ".lldbinit")) + DONE + +see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. for more information. + +Having a profile for debugging purposes +--------------------------------------- + +It is recommended to create a separate profile to debug with, whatever +your task, so that you don't lose precious data like Bookmarks, saved +passwords, etc. So that you're not bothered with the profile manager +every time you start to debug, expand the "Executables" branch of the +"Groups & Files" list and double click on the Executable you added for +Mozilla. Click the plus icon under the "Arguments" list and type "-P +<profile name>" (e.g. "-P MozillaDebug"). Close the window when you're +done. + +Running a debug session +----------------------- + +Make sure breakpoints are active (which implies running under the +debugger) by opening the Product menu and selecting "Debug / Activate +Breakpoints" (also shown by the "Breakpoints" button in the top right +section of the main window). Then click the "Run" button or select "Run" +from the Product menu. + +Setting breakpoints +~~~~~~~~~~~~~~~~~~~ + +Setting a breakpoint is easy. Just open the source file you want to +debug in Xcode, and click in the margin to the left of the line of code +where you want to break. + +During the debugging session, each time that line is executed, the +debugger will break there, and you will be able to debug it. + +.. warning:: + + Note that with the default configuration, some breakpoints will be + listed as "pending", and Xcode will not stop at them. If you don't + include the Mozilla's .lldbinit, you must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with + lldb <Debugging Firefox with LLDB>`. + +Using Firefox-specific lldb commands +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you included the .lldbinit when `Setting up +lldb <#setting-up-lldb>`__, you can use Mozilla-specific lldb commands +in the console, located in the Debug area of Xcode. For example, type +``js`` to see the JavaScript stack. For more information, see :ref:`Debugging +Mozilla with lldb <Debugging Firefox with LLDB>`. + +Debugging e10s child processes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using Xcode to debug child processes created by an e10s-enabled browser +is a little trickier than debugging a single-process browser, but it can +be done. These directions were written using Xcode 6.3.1 + +#. Complete all the steps above under "Creating the Project" +#. From the "Product" menu, ensure the scheme you created is selected + under "Scheme", then choose "Scheme > Edit Scheme" +#. In the resulting popup, click "Duplicate Scheme" +#. Give the resulting scheme a more descriptive name than "Copy of + Scheme" +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on the + "Executable" drop-down, and selecting the ``plugin-container.app`` + that is inside the app bundle of the copy of Firefox you want to + debug. +#. On the same tab, under "Launch" select "Wait for executable to be + launched" +#. On the "Arguments" tab, remove all arguments passed on launch. + +Now you're ready to start debugging: + +#. From the "Product" menu, ensure the scheme you created above is + selected under "Scheme" +#. Click the "Run" button. The information area at the top of the window + will show "Waiting for plugin-container to launch" +#. From a command line, run your build of Firefox. When that launches a + child process (for example, when you start to load a webpage), Xcode + will notice and attach to that child process. You can then debug the + child process like you would any other process. +#. When you are done debugging, click the "Stop" button and quit the + instance of Firefox that you were debugging in the normal way. + +For some help on using lldb see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. + +Other resources +--------------- + +Apple has an extensive list of `debugging tips and +techniques <https://developer.apple.com/library/mac/#technotes/tn2124/_index.html>`__. + +Questions? Problems? +~~~~~~~~~~~~~~~~~~~~ + +Try asking in our Element channels +`#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ or +`#macdev <https://chat.mozilla.org/#/room/#macdev:mozilla.org>`__. diff --git a/docs/contributing/debugging/debugging_on_windows.rst b/docs/contributing/debugging/debugging_on_windows.rst new file mode 100644 index 0000000000..3213299199 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_windows.rst @@ -0,0 +1,330 @@ +Debugging On Windows +==================== + +This document explains how to debug Gecko based applications such as +Firefox, Thunderbird, and SeaMonkey on Windows using the Visual Studio IDE. + +If VS and your Gecko application hang shortly after you launch the +application under the debugger, see `Problems Loading Debug +Symbols <#problems-loading-debug-symbols>`__. + +Ways to start the debugger +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First of all, it's necessary to install a Visual Studio extension to be +able to follow child processes as they are created. Firefox, in general, +and even in non-e10s mode, does not start the main process directly, it +starts it via a Launcher Process. This means that Visual Studio will +only attach to the first process it finds, and will not hit any +break-point (and even notifies you that it cannot find their location). +`Microsoft Child Process Debugging Power +Tool <https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool>`__ +allows automatically attaching to child processes, such as Web Content +process, GPU process, etc. Enable it by going its configuration menu in +"Debug > Other debugging targets > Child process debugging settings", +and ticking the box. + +If you have followed the steps in :ref:`Building Firefox for +Windows <Building Firefox On Windows>` +and have a local debug build, you can **execute this command from same command line.** + +.. code:: + + ./mach run --debug + +It would open Visual Studio with Firefox's +run options configured. You can **click "Start" button** to run Firefox +then, already attached in the debugger. + +Alternatively, if you have generated the Visual Studio solution, via +``./mach build-backend -b VisualStudio``, opening this solution allows +you to run ``firefox.exe`` directly in the debugger. To make it the +startup project, right click on the project and select ``Set As Startup +Project``. It appears bold when it's the case. Breakpoints are kept +across runs, this can be a good way to debug startup issues. + +**Run the program until you hit an assertion.** You will get a dialog +box asking if you would like to debug. Hit "Cancel". The MSDEV IDE will +launch and load the file where the assertion happened. This will also +create a Visual Studio Mozilla project in the directory of the executable +by default. + +**Attach the debugger to an existing Mozilla process**. In the Visual +Studio, select Debug > Attach to Process. If you want to debug a content +process, you can **hover on the tab** of page you want to debug, which +would show the pid. You can then select the process from dialog opened +from "Attach to Process". You can open ``about:processes`` to see the pid +for all subprocesses, including tabs but also GPU, networking etc. +For more information, see `Attach to Running Processes with the Visual Studio +Debugger <http://msdn.microsoft.com/en-us/library/vstudio/3s68z0b3.aspx>`__. + +**Starting an MSIX installed Firefox with the debugger**. In Visual +Studio, select Debug -> Other Debug Targets -> Debug Installed App Package. +In the dialog, select the installed Firefox package you wish to debug +and click "Start". + +Debugging Release and Nightly Builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the steps to :ref:`use the Mozilla symbol +server <Using The Mozilla Symbol Server>` and :ref:`source +server <Using The Mozilla Source Server>` + +Creating a Visual Studio project for Firefox +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please refer to :ref:`this <Visual Studio Projects>`. + +Changing/setting the executable to debug +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To change or set the executable to debug, go to Project > Properties > +Debugging > Command. (As of Visual Studio 2022.) + +It should show the executable you are debugging. If it is empty or +incorrect, manually add the correct path to the executable. + +Command line parameters and environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To change or set the command line options, go to Project > Properties > +Debugging > Command Arguments. + +Some common options would be the URL of the file you want the browser to +open as soon as it starts, starting the Profile Manager, or selecting a +profile. You can also redirect the console output to a file (by adding +"``> filename.txt``" for example, without the quotes). + +Customizing the debugger's variable value view +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can customize how Visual Studio displays classes in the variable view. +By default VS displays "{...}" and you need to click the small + icon +to expand the members. You can change this behaviour, and make Visual +Studio display whatever data member you want in whatever order, formatted +however you like instead of just "{...}". + +You need to locate a file called "gecko.natvis" under toolkit/library. +The file contains a list of types and how they should be displayed in +the debugger. It is XML and after a little practice you should be well +on your way. + +To understand the file in detail refer to `Create custom views of C++ +objects in the debugger using the Natvis framework +<https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects>`__ + +The file already comes with a number of entries that will make your life +easier, like support for several string types. If you need to add a custom +type, or want to change an existing entry for debugging purposes, you can +easily edit the file. For your convenience it is included in all generated +Visual Studio projects, and if you edit and save it within Visual Studio, it +will pick up the changes immediately. + +Handling multiple processes in Visual Studio +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Turn off "Break all processes when one process breaks" to single step a single +process. + +Turning off "Break all processes when one process breaks" adds "Step Into +Current Process", "Step Over Current Process" and "Step Out Current Process" to +the "Debug" menu. + +To single step a single process with the other processes paused: + +- Turn on "Break all processes when one process breaks" +- Hit a breakpoint which stops all processes +- Turn off "Break all processes when one process breaks" +- Now using "Step Into Current Process" will leave the other processes stopped + and just advance the current one. + +Obtaining ``stdout`` and other ``FILE`` handles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Running the following command in the Command Window in Visual Studio +returns the value of ``stdout``, which can be used with various +debugging methods (such as ``nsGenericElement::List``) that take a +``FILE*`` param: + +.. code:: + + Debug.EvaluateStatement {,,msvcr80d}(&__iob_func()[1]) + +(Alternatively you can evaluate ``{,,msvcr80d}(&__iob_func()[1])`` in +the Immediate window) + +Similarly, you can open a file on the disk using ``fopen``: + +.. code:: + + >Debug.EvaluateStatement {,,msvcr80d}fopen("c:\\123", "w") + 0x10311dc0 { ..snip.. } + >Debug.EvaluateStatement ((nsGenericElement*)0x03f0e710)->List((FILE*)0x10311dc0, 1) + <void> + >Debug.EvaluateStatement {,,msvcr80d}fclose((FILE*)0x10311dc0) + 0x00000000 + +Note that you may not see the debugging output until you flush or close +the file handle. + +Disabling ASSERTIONS +~~~~~~~~~~~~~~~~~~~~ + +There are basically two ways to disable assertions. One requires setting +an environment variable, while the other affects only the currently +running program instance in memory. + +Environment variable +^^^^^^^^^^^^^^^^^^^^ + +There is an environment variable that can disable breaking for +assertions. This is how you would normally set it: + +.. code:: + + set XPCOM_DEBUG_BREAK=warn + +The environment variable takes also other values besides ``warn``, see +``XPCOM_DEBUG_BREAK`` for more details. + +Note that unlike Unix, the default for Windows is not warn, it's to pop +up a dialog. To set the environment variable for Visual Studio, use +Project > Properties > Debugging > Environment and click the little box. +Then use + +.. code:: + + XPCOM_DEBUG_BREAK=warn + +Changing running code +^^^^^^^^^^^^^^^^^^^^^ + +You normally shouldn't need to do this (just quit the application, set +the environment variable described above, and run it again). And this +can be **dangerous** (like **trashing your hard disc and corrupting your +system**). So unless you feel comfortable with this, don't do it. **You +have been warned!** + +It is possible to change the interrupt code in memory (which causes you +to break into debugger) to be a NOP (no operation). + +You do this by running the program in the debugger until you hit an +assertion. You should see some assembly code. One assembly code +instruction reads "int 3". Check the memory address for that line. Now +open memory view. Type/copy/drag the memory address of "int 3" into the +memory view to get it to update on that part of the memory. Change the +value of the memory to "90", close the memory view and hit "F5" to +continue. + +Automatically handling ASSERTIONS without a debugger attached +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When an assertion happens and there is not a debugger attached, a small +helper application +(```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__) +is run. That application can automatically select a response to the "Do +you want to debug" dialog instead of prompting if you configure it, for +more info, see +```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__. + +Debugging optimized builds +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To effectively debug optimized builds, you should enable debugging +information which effectively leaves the debug symbols in optimized code +so you can still set breakpoints etc. Because the code is optimized, +stepping through the code may occasionally provide small surprises when +the debugger jumps over something. + +You need to make sure this configure parameter is set: + +.. code:: + + ac_add_options --enable-debug + +You can also choose to include or exclude specific modules. + +Console debugging +~~~~~~~~~~~~~~~~~ + +When printing to STDOUT from a content process, the console message will +not appear on Windows. One way to view it is simply to disable e10s +(``./mach run --disable-e10s``) but in order to debug with e10s enabled +one can run + +:: + + ./mach run ... 2>&1 | tee + +It may also be necessary to disable the content sandbox +(``MOZ_DISABLE_CONTENT_SANDBOX=1 ./mach run ...``). + +Running two instances of Mozilla simultaneously +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can run two instances of Mozilla (e.g. debug and optimized) +simultaneously by setting the environment variable ``MOZ_NO_REMOTE``: + +.. code:: + + set MOZ_NO_REMOTE=1 + +Or, starting with Firefox 2 and other Gecko 1.8.1-based applications, +you can use the ``-no-remote`` command-line switch instead (implemented +in +`bug 325509 <https://bugzilla.mozilla.org/show_bug.cgi?id=325509>`__). + +You can also specify the profile to use with the ``-P profile_name`` +command-line argument. + +Debugging JavaScript +~~~~~~~~~~~~~~~~~~~~ + +You can use helper functions from +`nsXPConnect.cpp <https://searchfox.org/mozilla-central/source/js/xpconnect/src/nsXPConnect.cpp>`__ +to inspect and modify the state of JavaScript code from the MSVS +debugger. + +For example, to print current JavaScript stack to stdout, evaluate this +in Immediate window: + +.. code:: + + {,,xul}DumpJSStack() + +Visual Studio will show you something in the quick watch window, but +not the stack, you have to look in the OS console for the output. + +Also this magical command only works when you have JS on the VS stack. + +Debugging minidumps +~~~~~~~~~~~~~~~~~~~ + +See :ref:`debugging a minidump <Debugging A Minidump>`. + +Problems post-mortem debugging on Windows 7 SP1 x64? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you attempt to use ``NS_DebugBreak`` etc to perform post-mortem +debugging on a 64bit Windows 7, but as soon as you try and continue +debugging the program crashes with an Access Violation, you may be +hitting a Windows bug relating to AVX support. For more details, +including a work-around see `this blog +post <http://www.os2museum.com/wp/?p=960>`__ or `this social.msdn +thread <http://social.msdn.microsoft.com/Forums/vstudio/en-US/392ca62c-e502-42d9-adbc-b4e22d5da0c3/jit-debugging-32bit-app-crashing-with-access-violation>`__. +(And just in-case those links die, the work-around is to execute + +:: + + bcdedit /set xsavedisable 1 + +from an elevated command-prompt to disable AVX support.) + +Got a tip? +~~~~~~~~~~ + +If you think you know a cool Mozilla debugging trick, feel free to +discuss it with `#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ and +then post it here. + +.. |Screenshot of disabling assertions| image:: https://developer.mozilla.org/@api/deki/files/420/=Win32-debug-nop.png + :class: internal diff --git a/docs/contributing/debugging/debugging_webrtc_calls.rst b/docs/contributing/debugging/debugging_webrtc_calls.rst new file mode 100644 index 0000000000..a08ab05999 --- /dev/null +++ b/docs/contributing/debugging/debugging_webrtc_calls.rst @@ -0,0 +1,1060 @@ +Debugging WebRTC Calls +====================== + +.. _reporting_webrtc_call_issues: + +Reporting WebRTC Call Issues +---------------------------- + +The best way to report an issue is through Bugzilla using this +`link <https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=WebRTC>`__. +Describe the issue you've run into, include a URL, along with the +details of the call setup. See the :ref:`adding-call-setup-information` +section for helpful templates. Here are some common examples of descriptive +WebRTC bug summaries: + +* caller is experiencing frozen video, screen capture, or desktop capture +* caller does not hear audio +* caller's voice sounds distorted or robotic +* video resolution is lower than expected +* caller's video appears rotated +* there is significant delay between a caller's video and audio +* camera, microphone, or screens are not appearing in the Firefox device access + permission prompts, etc. +* caller's video is garbled, partially missing, or the colors are incorrect +* caller can not share external display, but can share integrated display + +.. note:: + Not all web conferencing software makes extensive use of WebRTC. + + +For simple issues, the first place to look is to check the +`web developer console </devtools-user/web_console>`__ for error messages +related to media format issues. If you see messages here related to WebRTC, +getUserMedia, or getDisplayMedia, please add this information to your bug. + +.. _adding-call-setup-information: + +Adding Call Setup Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following template can help one provide all the call details needed to +diagnose many common issues. + +.. code:: md + + * Does this problem occur in Firefox for Desktop or Android? + + * Is this problem reproducible in Firefox Nightly? + + * Has this worked previously? + + * Have you tried using `about:profiles` to reproduce the problem in a + clean profile? + + * How many participants were in the call? + + * In which web conferencing or web calling services does the problem occur? + + * Does the problem present itself immediately upon starting the call? + + * If not how long does it take to occur? + + * If this is a problem with audio or video capture, what camera or microphone + are you using? (adding about:support text may be helpful) + + * If this is problem with screen capture, which screen was being captured, + and are there other screens attached to the same machine? + + * Would you be willing to help us find a regression range? + +If the issue is of specification compliance the template provided below may be +more useful. If one is unsure if this is a compliance issue, one can refer to +the :ref:`standards-documentation` section for links. + +.. code:: md + + * What unexpected behavior is being observed? What are the steps to reproduce + this issue? + + * What is the expected behavior? + + * Where is this behavior specified? + + * Is this problem reproducible in Nightly? + + * Have you tried using `about:profiles` to reproduce the problem in a clean + profile? + + * Has this worked previously? + + * If so, would you be willing to help us find a regression range? + +.. _share_your_aboutsupport_text: + +Adding about:support Text +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In your Bugzilla report, please include support information about the +current device on which you are experiencing an issue. + +#. Open a tab and visit about:support +#. Click 'Copy Text to Clipboard' +#. Paste this text in your Bugzilla bug comment and post. + +.. note:: + + To open :code:`about:*` links in Firefox one must do the following: + #. Right-click the link and select `Open Link in New Tab` + #. Select the new tab + #. Click inside the address bar which should contain the about URL + #. Press Enter + +.. _share_your_aboutwebrtc_contents: + +Adding about:webrtc RTCPeerConnection Stats +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Open about:webrtc. +#. Expand the RTCPeerConnection section. +#. Locate and expand the subsection RTCPeerConnection that one wishes to copy + stats from. +#. Press the :code:`Copy Stats Report` +#. In the Bugzilla bug, press the :code:`Attach New File` button. +#. Click inside the large text box labeled :code:`File`, and paste the copied + stats report. +#. Add a descriptive label to the the :code:`Description:`, e.g. + "PeerConnection statistics sample taken during period of frame loss". +#. In the drop down box, next to the :code:`select from list` radio option, + select "JSON source (application/json)". +#. If needed, add a descriptive comment to the :code:`comment` field. +#. Press the :code:`Submit` button. + +.. note:: + Hovering the mouse over some headings will reveal a clipboard icon. Clicking + this icon will copy the contents under that section to the clipboard as a + JSON document. This can be useful if one wants to submit a portion of the + available stats. Of particular note are the :code:`RTP Stats` heading whose + button will copy the latests RTP stats, and the :code:`SDP` section whose + button will copy the SDP offer; answer; and roles. + +Adding Your about:webrtc Contents +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For issues with call quality, please share web conferencing related +performance information by providing your about:webrtc information. Note +this information should be collected while the call in question is still +active. + +#. While your call is still ongoing, open an additional tab and visit + about:webrtc. +#. Click "Clear History" to clear the stats from other recent calls + which are no longer ongoing. +#. At the bottom of the page click 'Save Page', and save this file. +#. Add this file as an attachment to your bug. + +This data contains statistics about your call, the signalling that was +used to setup your call, and information about the network transports. + +.. _diagnosing_call_quality_issues: + +Diagnosing Call Quality Issues +------------------------------ + +.. _about_webrtc_overview: + +about:webrtc Overview +~~~~~~~~~~~~~~~~~~~~~ + +about:webrtc is an in-browser resource for debugging WebRTC calls. The +primary audience for about:webrtc is browser developers but it can also be +of use to anyone that needs to troubleshoot a WebRTC call. When there is no +call data to display, about:webrtc will appear as follows: + +.. figure:: img/about_webrtc_default_view.png + :alt: about:webrtc before any calls have been made + + about:webrtc before any calls have been made + +Note that there are several sections. During a call each section will contain +information pertinent to different aspects of the WebRTC browser implementation. + +RTCPeerConnection Statistics +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This section presents information useful for diagnosing active calls. It +contains RTCPeerConnection creation parameters, connection information, +negotiation details, RTP stream statistics, bandwidth statistics, and +output frame statistics. + +Connection Log +^^^^^^^^^^^^^^ + +When one needs to diagnose issues with the underlying transports, logs +can be found under Connection Log. + +User Modified WebRTC Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +This section will display any user modified preferences which effect the +performance or behavior of browser components which can impact WebRTC calls. +When hovering over a preference path displayed in this section a clipboard +icon will appear. Clicking that icon will copy the path to the clipboard. It +can then be pasted into `about:config <about:config>`__, to change or reset the +value to its default. + +.. warning:: + Unexpected values in this section may be caused by installed extensions. It + is best to test issues in a clean profile when possible using + `about:profiles <about:profiles>`__. + +Media Context +^^^^^^^^^^^^^ + +Information that is gathered to determine Codec availability and capability is +recorded under Media Context. + + +Bottom Control Bar +^^^^^^^^^^^^^^^^^^ + +At the bottom of about:webrtc is a row of buttons which allow the user to +perform miscellaneous actions. + +The `Save Page` button expands all the sections and presents a dialog to save +the contents of the page. This will produce an HTML file suitable for attaching +to bug reports. + +In the event of a WebRTC issue, the :code:`Enable WebRTC Log Preset` button is a very +quick way to start logging. Pressing that button will open a new tab with +`about:logging <about:logging>`__ with the :code:`webrtc` preset selected. This +preset contains all :ref:`standard-logging-modules`. Logging +will begin immediately. If one needs to change other log settings on that page +one can customize them, and then press :code:`Start Logging`. This may be +necessary if one wishes to log to the profiler. + +If experiencing echo cancellation issues, one may be asked to submit echo +cancellation logs. These logs are gathered by pressing the +:code:`Start AEC Logging` button. One should press the button to activate +the logging while actively experiencing an echo cancellation failure. + +.. note:: + + Producing echo cancellation logs is incompatible with the content sandbox. + The user will be prompted with further instructions if the sandbox is active + and the :code:`Start AEC Logging` button is pressed. + +.. _audiovideo_delay: + +Audio/Video Delay +^^^^^^^^^^^^^^^^^ + +Delayed media is commonly caused by long physical paths between +endpoints, though anything that slows down inter-hop delivery of packets +can be at fault. Note that this is different than the bandwidth of the +network path, and a high latency will not be fixed by reducing the video +resolution or audio sample rate. Round trip time, or RTT, is the time it +takes for a packet to get from the sender to the receiver and then for a +packet to get from the receiver back to the sender. If the path is +symmetric between the two endpoints one can assume that the one way +delay is half the delay of the round trip. + +The second common cause of A/V delay is jitter, the magnitude of +variability in packet inter-arrival times. In order to smoothly play out +of the incoming stream a receiver experiencing jitter will have to +buffer (delay) incoming packets. + +**Using** `about:webrtc <about:webrtc>`__ **to Diagnose Delay** + +The key metrics in `about:webrtc <about:webrtc>`__ are RTT (round-trip-time) and +jitter. They can be found in the RTP stats section of the +PeerConnection. The PeerConnection informational blocks start out in a +minimized state, and one will need to expand a block to find the RTP +stats section: + +.. figure:: img/about_webrtc_reveal_PeerConnection_stats.png + :alt: How to reveal the full statistics of a PeerConnection in about:webrtc + :width: 800px + + How to reveal the full statistics of a PeerConnection in about:webrtc + +Once expanded one can locate the RTP Stats section and find the key +diagnostic stats: + +.. figure:: img/about_webrtc_Jitter_and_RTT.png + :alt: Location in about:webrtc of jitter and RTT stats + :width: 800px + + Location in about:webrtc of jitter and RTT stats + +In this image we can see that there are 0 milliseconds of jitter, and 32 +milliseconds of round trip delay. This call should not be experiencing any +noticeable delay. See the `Delay Calculation <#delay-calculation>`__ +appendix section below for some more detail. + +If the perceived delay is larger than the estimated delay that could +indicate a problem within Firefox that requires debugging. Under these +circumstances it would be helpful to grab a JSON copy of the current +stats by pressing the "Copy Report" button, pasting those stats into +your Bugzilla bug report. + +.. figure:: img/about_webrtc_copy_report.png + :alt: Location in about:webrtc of Copy Report button + :width: 800px + + Location in about:webrtc of Copy Report button + +.. _performance-profiling-and-logging: + +Performance Profiling and Logging +--------------------------------- + +.. _capturing-a-firefox-performance-profile: + +Capturing a Firefox Performance Profile +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For basic performance issues, a performance profile can help engineers +diagnose issues with video formats, performance, and rendering. + +#. Visit https://profiler.firefox.com/ and enable the Profiler toolbar + button. +#. Click the toolbar button down arrow and select 'Media' in the + Settings drop down. +#. Open a tab and visit the page with the affected media content. +#. Click the Profiler toolbar main button to start recording. +#. Play media until the issue you are seeing manifests. +#. Click the Profiler toolbar button again to stop recording. +#. When a new Profile tab opens, click the upload profile button on the + upper right. +#. Copy the resulting profile URL and post this to your Bugzilla report. + +Additionally, detailed logging can be collected within performance +profiles to help aid in debugging complicated issues. To enable the +collection of a profile with low level debugging - + +#. Visit https://profiler.firefox.com/ and enable the Profiler toolbar + button. +#. In a new tab, visit about:webrtc. Click the 'Enable WebRTC Log + Preset' button, which will open a tab for about:logging with + pre-populated information. +#. In about:logging, click the "Start Logging" button. (You are now + recording a profile, the profiler toolbar toggle button should be + selected automatically.) +#. Open a new tab for testing and view the media you are having an issue + with. (After reproducing, DO NOT close this test tab yet.) +#. Switch to the about:logging tab, click 'Stop logging', and then close + the test tab. +#. Wait approximately 10 - 20 seconds for a new tab to automatically + open containing the generated performance profile. +#. Within the upper-right side of the profiler tab click the 'upload + local profile' button to initiate profile upload. Once the upload is + complete, a drop down text field will open displaying the URL of the + profile. Select this text and copy it. +#. Share the URL of the profile for analysis with the engineer you are + working with. + +Alternatively one can set the following environment variable: + +.. code:: sh + + MOZ_LOG="jsep:5,sdp:5,signaling:5,mtransport:5,RTCRtpReceiver:5,RTCRtpSender:5,RTCDMTFSender:5,VideoFrameConverter:5,WebrtcTCPSocket:5,CamerasChild:5,CamerasParent:5,VideoEngine:5,ShmemPool:5,TabShare:5,MediaChild:5,MediaParent:5,MediaManager:5,MediaTrackGraph:5,cubeb:5,MediaStream:5,MediaStreamTrack:5,DriftCompensator:5,ForwardInputTrack:5,MediaRecorder:5,MediaEncoder:5,TrackEncoder:5,VP8TrackEncoder:5,Muxer:5,GetUserMedia:5,MediaPipeline:5,PeerConnectionImpl:5,WebAudioAPI:5,webrtc_trace:5,RTCRtpTransceiver:5,ForwardedInputTrack:5,HTMLMediaElement:5,HTMLMediaElementEvents:5" + +Note that webrtc_trace will not be active until "Enable WebRTC Log +Preset" is pressed. + +.. _standard-logging-modules: + +Standard Logging Modules +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Standard Logging Modules + :header-rows: 1 + + * - Module + - Component + - Function + - Notes + * - jsep + - signalling + - JSEP state machine + - + * - sdp + - signalling + - SDP parsing + - + * - mtransport + - networking + - Network transports + - + * - RTCRtpReceiver + - JS API + - JS API related to receiving media and media control packets + - + * - RTCRtpSender + - JS API + - JS API related to sending media and media control packets + - + * - RTCDMTFSender + - JS API + - JS API related to sending DTMF messages + - + * - VideoFrameConverter + - + - + - + * - WebrtcTCPSocket + - networking + - + - + * - CamerasChild + - media capture + - Content process end of IPC channel for receiving frames from media + capture devices + - + * - CamerasParent + - media capture + - Parent process end of IPC channel for sending frames from media capture devices + - + * - VideoEngine + - media capture + - Orchestrates capture of frames from media capture devices in the parent process + - + * - ShmemPool + - media capture + - Object pool of shared memory frame buffers for transferring media capture frames from parent to child process + - + * - TabShare + - media capture + - Captures tab content for sharing + - + * - MediaChild + - media + - + - + * - MediaParent + - media + - + - + * - MediaManager + - media + - + - + * - MediaTrackGraph + - media + - + - + * - cubeb + - media + - + - + * - MediaStream + - media + - + - + * - MediaStreamTrack + - media + - + - + * - DriftCompensator + - media + - + - + * - ForwardInputTrack + - media + - + - + * - MediaRecorder + - media + - + - + * - MediaEncoder + - media + - + - + * - TrackEncoder + - media + - + - + * - VP8TrackEncoder + - media + - + - + * - Muxer + - media + - + - + * - MediaPipeline + - network + - Glue code between transport, media, and libwebrtc components + - + * - PeerConnectionImpl + - JS API + - implements the RTCPeerConnection object + - + * - WebAudioAPI + - + - + - + * - webrtc_trace + - webrtc + - libwebrtc logging + - Prior to Firefox v123 it must be enabled from + `about:webrtc <about:webrtc>`__ at runtime, or it must be set in the + :code:`MOZ_LOG` environment variable at launch. + * - RTCRtpTransceiver + - JS API + - implements the RTCRtpTransceiver object + - + * - HTMLMediaElement + - + - + - + * - ForwardedInputTrack + - + - + - + * - HTMLMediaElementEvents + - + - + - + +.. _non-standard-loggin-modules: + +Non-standard Logging Modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Standard Logging Modules + :header-rows: 1 + + * - Module + - Component + - Function + - Notes + * - RTPLogger + - network + - Logs RTP and RTCP packet fragments + - See `Debugging Encrypted Packets <#debugging-encrypted-packets>`__ + +.. _examining-call-performance-issues: + +Examining Call Performance Issues +--------------------------------- + +.. _enabling-call-stats-history: + +Enabling Call Stats History +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Call stats history is enabled by default in Nightly. To enable in +release builds open `about:config <about:config>`__, and change +"media.aboutwebrtc.hist.enabled" to true. This will keep a history +window of stats for a number of recent calls, allowing for inspection +in `about:webrtc <about:webrtc>`__ after a call has completed. + +.. _dumping-call-stats: + +Dumping Call Stats +~~~~~~~~~~~~~~~~~~ + +One can dump a JSON blob of call stats for an active call, or a recent +call if call stats history is enabled. There are two buttons in +`about:webrtc <about:webrtc>`__ to do this, "Copy Report" and "Copy Report +History". The former will create a copy of the most recent stats for the +PeerConnection. The later will copy all the history of stats reports +that `about:webrtc <about:webrtc>`__ has accumulated for that PeerConnection, this +can be up to several minutes of stats. + +.. _debugging-encrypted-packets: + +Debugging Encrypted Packets +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. warning:: + There is a `blog + post <https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-used-to-be/>`__ + covering dumping unencrypted partial RTP and RTCP packets in the logs. + While the information presented in that post is still relevant, + the command to extract the packet data in the blog is out of date. A + working method is presented below; + +Using the gecko logging system, unencrypted, mangled, partial, RTP-packets can +be written out. This may be a good avenue of investigation for packet loss and +recovery, simulcast, and feedback. Because the entirety of the packet is not +guaranteed to be logged, this is less suitable for debugging issues with +encoded media. These logged packets can be converted to PCAP files, which can +then be explored in `Wireshark <https://www.wireshark.org/>`__. The logs +produced by this module can be quite large, making it easy to identify by file +size which child process log files contain packet dumps. + +To start RTP logging, one must enable the :code:`RtpLogger` log module. The :code:`sync` +option should also be used, as it prevents undesirable interleaving of log +messages. Here are the minimal log settings needed: + +.. code:: sh + + MOZ_LOG='sync,RtpLogger:5' + +In order to interpret the packet contents, one needs to refer to the SDP. +Wireshark is unaware of the negotiated details, so it can not directly decode +the media, nor can it decode the header extensions. The SDP can also be logged, +and so the following is a more useful set of log settings: + +.. code:: sh + + MOZ_LOG='sync,RtpLogger:5,jsep:5' + +.. note:: + + On macOS it is simple to install Wireshark and text2pcap with Homebrew: + + .. code:: sh + + # Use only one of the following: + # ============================== + + # To install the Wireshark GUI application and the command line utilities: + brew install --cask wireshark + + # To install only the command line utilities: + brew install wireshark + +One can use :code:`tee` to capture log output +from a copy of Firefox launched from the command line, e.g. through +:code:`mach`. Alternatively, one can set a log file through the environment +variable :code:`MOZ_LOG_FILE` or through about:logging. + +.. warning:: + + If log files are not being created by child processes, this is likely due + to sandboxing of content processes. To work around this one must either + select a location within the sandbox, disable the content sandbox, or launch + Firefox from the command line, e.g. from a Firefox dev environment: + + .. code:: + + MOZ_LOG=sync,RtpLogger:5,jsep:5 MOZ_LOG_FILE= ./mach run 2>&1 | tee your.log + + +To produce a PCAP file one needs to filter the logs to include only the +RtpLogger log lines, reduce them down to the expected ingestion format for +text2pcap, and finally to invoke text2pcap. + +.. code:: sh + + cat your.log | rg 'RtpLogger.*RTC?P_PACKET|>>\s(?P<packet>.+$)' --only-matching --replace '$packet' | text2pcap -D -n -l 1 -i 17 -u 1234,1235 -t '%H:%M:%S.' - your.output.pcap + +.. note:: + + If :code:`rg`, a.k.a ripgrep, is not already available, one can install it via one of the following methods: + + .. code:: sh + + # Install through cargo on macOS, Linux, or Windows + cargo install ripgrep + + # Install via Homebrew on macOS + brew install ripgrep + + # ripgrep packages may be available through the package manager for your + # Linux distro + + +The resulting PCAP file can be explored with Wireshark. Currently, one must refer to the SDP in order to interpret the RTP packets. + +.. code:: sh + + # On most Linux distros + wireshark -d 'udp.port==1234,rtp' your.output.pcap + + # On macOS when installed via Homebrew + open /Applications/Wireshark.app --args -d 'udp.port==1234,rtp' your.output.pcap + +.. _examining-codec_availability-and-capabilities: + +Examining Codec Availability and Capabilities +--------------------------------------------- + +When codec negotiation doesn't happen as expected there are several helpful +locations where one can find information. The SDP offer and answer contain the +list of codecs that were in the initial offer, and the subset of those codecs +that were selected in the answer. + +The easiest way to get this information on a live call is through +about:webrtc. Each RTCPeerConnection has its own subsection, that when +expanded contains an SDP section. There are buttons to display the offer +and the answer. Depending on which party was the offerer and which was +the answerer one may have a local offer and a remote answer, or a remote offer +and a local answer. + +Firefox chooses which codecs to offer based on availability. Some codecs, +like Opus or VP8, are always available. Some codecs are available in software +and some codecs on some platforms are available in hardware. H264 support is +provided by a third-party, and is automatically downloaded the first time its +use is requested. This is a process which can take a variable amount of time +depending on network circumstances. + +.. note:: + A list of media codecs with playback support are available in the Media + section of `about:support#media <about:support#media>`__ . Not all media + codecs present and available to Firefox for playback are supported in WebRTC + calls. + +To check the current factors, including preferences, that are being used to +calculate availability beyond codec presence, one can check the `Media Context` +section of about:webrtc. + +.. figure:: img/about_webrtc_media_context.png + :alt: example about:webrtc media context values + +.. _running-webrtc-tests: + +For an in-depth reference covering the codecs available through WebRTC please +see the MDN Page: `Codecs Used by WebRTC <https://developer.mozilla.org/en-US/docs/Web/Media/Formats/WebRTC_codecs>`__. + +Running WebRTC Tests +-------------------- + +There are a variety of tests providing coverage over WebRTC related code. The +Web Platform Suite provides conformance tests for browsers. The +:code:`gtest` suite is composed of unit tests. Crashtests are a type of +regression test which are written to induce crashes. There are fuzzing tests +which exercise APIs in ways that the authors did not foresee. All of the WebRTC +tests can be run locally with :code:`mach` or in CI on `Try`. There is a +detailed overview of all available test types including those not exercised by +WebRTC code `here </testing/automated-testing/index.html#functional-testing>`__. + +.. note:: + Running :code:`./mach <verb> --help` is an indispensable tool for discovering + options that can streamline your testing process. + +.. note:: + A test suite on `Try` maybe an aggregate of multiple logical test suites. + For example, the `mochitest-media` suite on try includes both the WebRTC and + playback mochitests. + +.. warning:: + WebRTC calls make use of a number of internal timers. Amongst the + behaviors these timers control are transport selection, bandwidth estimation, + packet loss determination, media adaptation, lip sync, connection timeout, + and more. There are `Try` targets which are too slow to reliably run a + number of the tests. Before running a specific test on `Try` for the first + time, it may be best to check the relevant test suite manifest. This can be + done easily with Searchfox.org by searching for and viewing a test file. If + that test has been disabled on one or more platforms the details will appear + as shown below: + + .. figure:: img/searchfox_test_disabled_warning.png + :alt: Searchfox.org warning that the displayed test file has been disabled on Android + +.. _test-atlas: + +Test Atlas +---------- + +.. list-table:: WebRTC Test Locations + :widths: 10 10 20 10 10 + :header-rows: 1 + + * - Component + - Test type + - Test file location + - Try suite + - Treeherder Abbreviations + * - WebRTC + - Mochitest + - dom/media/webrtc/mochitests + - mochitest-media + - :code:`mda`, :code:`M(mda)` + * - + - Web Platform Test + - testing/web-platform/tests/webrtc + - wpt + - :code:`wpt`, :code:`W(wpt)` + * - + - Crashtest + - dom/media/webrtc/tests/crashtests + - crash + - :code:`R(C)` + * - WebRTC Signalling + - GTest + - media/webrtc/signaling/gtest + - gtest + - :code:`gtest` + * - WebRTC (gUM / gDDM) + - Browser Chrome Test (mochitest) + - browser/base/content/test/webrtc + - browser-chrome + - :code:`bc`, :code:`M(bc)` + * - WebRTC Transport + - CPPUnit + - dom/media/webrtc/transport/test + - cppunit + - + * - + - fuzztest + - dom/media/webrtc/transport/fuzztest + - fuzzing + - + * - SDP parser + - Fuzzing + - dom/media/webrtc/tests/fuzztests + - fuzzing + - + +.. _webrtc-web-platform-tests: + +Web Platform Tests +~~~~~~~~~~~~~~~~~~ + +The WPT suite comprises conformance tests for various W3C specs such as: CSS, +JS APIs, and HTML. WebRTC is a JS API and as such its tests are of the `testharness.js <https://web-platform-tests.org/writing-tests/testharness.html>`__ type. +There is detailed :code:`WPT` documentation available `here </web-platform/index.html#web-platform-tests>`__ +Web Platform Tests can be run locally from + +.. code:: bash + + # Run the entire WebRTC WPT test suite + ./mach wpt testing/web-platform/tests/webrtc + + # Run a single test, e.g. RTCPeerConnection-createAnswer.html + ./mach wpt testing/web-platform/tests/webrtc/RTCPeerConnection-createAnswer.html + + # Run all of the PeerConnection tests, i.e. RTCPeerConnection-*.html + # NOTE that the `mach` verb in use is `test` not `wpt` + ./mach test testing/web-platform/tests/webrtc/RTCPeerConnection + +.. warning:: + Running the :code:`WPT` tests locally can be very disruptive to one's working + desktop environment, as windows will frequently appear and grab focus. Using + :code:`mach`'s :code:`--headless` flag will prevent this, and can be a great + way to run them if one's problem can be evaluated from command line output. + +These tests are synced from the main `Web Platform Test +repository <https://github.com/web-platform-tests/wpt>`__, and likewise +our changes are synced from our `in-tree +copy <https://searchfox.org/mozilla-central/search?q=&path=testing%2Fweb-platform%2Ftests%2Fwebrtc&case=false®exp=false>`__ +back to that repository. + +.. warning:: + Running the WebRTC mochitests in `Try` is done using the entire Web Platform Test suite, :code:`wpt`. + As such, this can be slow. + +.. code:: bash + + ./mach try fuzzy --query 'wpt' + +One can `run those same tests in +Chromium <https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/testing/web_platform_tests_wptrunner.md>`__, +Safari, or Servo if one needs to compare behavior between browsers. This can be +done directly through :code:`mach`, see `running tests in other browsers </web-platform/index.html#running-tests-in-other-browsers>`__ +for more details. + +.. _mochitests: + +Mochitests +~~~~~~~~~~ + +The WebRTC mochitests are integration tests, regression tests, and sanity +tests. The needs of these tests did not align with specification conformance +testing in the WPT, Web Platform Test, suite. Before writing a new mochitest, +one should consider if a test would be better expressed as a WPT, which all browsers +can test against. + +Locally running the WebRTC mochitests should be done in a Firefox dev +environment using :code:`mach` as follows: + +.. code:: bash + + # Run the whole suite + ./mach mochitest dom/media/webrtc/tests/mochitests + + # Run a single test, e.g. test_peerConnection_basicAudioVideo.html + ./mach mochitest dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioVideo.html + # Or + ./mach mochitest test_peerConnection_basicAudioVideo.html + + # Run all of the PeerConnection tests, i.e. test_peerConnection_*.html + ./mach mochitest test_peerConnection + + +On :code:`try`, WebRTC mochitests are part of the larger media test suite. +This suite can be easily selected with the following fuzzy query: + +.. code:: bash + + # Run the media mochitest suite on all regular platforms + ./mach try fuzzy --query 'mochitest-media' + + # Run the media mochitest suite only on Linux which will resolve far faster + ./mach try fuzzy --query 'linux mochitest-media' + + +.. _gtests: + +GTests +~~~~~~ + +The gtests are all compiled into a single library target: :code:`xul-test`. +This makes running gtests from :code:`mach` slightly different than the other +test types. + +.. code:: bash + + # Run a single test by using Prefix.TestName, e.g. JsepSessionTest.FullCall + # https://searchfox.org/mozilla-central/rev/4d6a5b97428760d15bfcad13f8fc81439370a7ec/media/webrtc/signaling/gtest/jsep_session_unittest.cpp#1551 + ./mach gtest 'JsepSessionTest.FullCall' + + # Run all the tests in a single Prefix, e.g. JsepSessionTest + ./mach gtest 'JsepSessiontTest.*' + + # Run all tests which have a Prefix.TestName containing the substring 'Jsep' + # See the table of selectors below + ./mach gtest '*Jsep*' + + # Run all the gtests for Firefox + ./mach gtest + +Here is a list of helpful substring selectors for executing specific WebRTC gtests: + +.. list-table:: WebRTC GTest Selectors + :header-rows: 1 + + * - Selector + - Description + - Files + * - :code:`*Jsep*` + - JSEP (signalling) tests + - `jsep_session_unittest.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/jsep_session_unittest.cpp>`__ + `jsep_trak_unittest.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/jsep_track_unittest.cpp>`__ + * - :code:`*Sdp*` + - SDP parsing tests + - `sdp_unittests.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/sdp_unittests.cpp>`__ + * - :code:`*MediaPipeline*` + - MediaPipline and MediaPipeline filter tests for RTP media handling + - `mediapipeline_unittest.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/mediapipeline_unittest.cpp>`__ + * - :code:`*AudioConduit*` + - AudioConduit tests for libwebrtc glue for RTP audio media + - `audioconduit_unittests.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/audioconduit_unittests.cpp>`__ + * - :code:`*VideoConduit*` + - VideoConduit tests for libwebrtc glue for RTP video media + - `videoconduit_unittests.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/videoconduit_unittests.cpp>`__ + +For more general information about gtests see the documentation `here </gtest/index.html>`__. + +Fuzz Testing +~~~~~~~~~~~~ + +It is not common to need to run fuzz testing as it is run on a semi-continuous +fashion in CI. It is more likely that one will need to respond to a bug filed +by a fuzzing bot. If one is interested in fuzzing one should consult the +excellent documentation available `here </tools/fuzzing/index.html>`__. + +.. _code-atlas: + +Code Atlas +---------- + +There are a number of components that work together to create a successful +WebRTC call. When debugging a call it can be difficult to see the larger puzzle +for all the pieces. A listing of the WebRTC related source code directories is +provided below to help one navigate. + +.. list-table:: WebRTC Code Atlas + :header-rows: 1 + + * - Directory + - Component + - Description + - Notes + * - `dom/media/webrtc <https://searchfox.org/mozilla-central/source/dom/media/webrtc>`__ + - WebRTC + - This is the primary directory for Firefox WebRTC code + - + * - `dom/media/webrtc/common <https://searchfox.org/mozilla-central/source/dom/media/webrtc/common>`__ + - WebRTC + - This contains WebRTC related utility code + - + * - `dom/media/webrtc/jsapi <https://searchfox.org/mozilla-central/source/dom/media/webrtc/jsapi>`__ + - JS API + - This contains the C++ implementations of the JavaScript WebRTC interfaces + - + * - `dom/media/webrtc/jsep <https://searchfox.org/mozilla-central/source/dom/media/webrtc/jsep>`__ + - Signalling + - This is the JSEP state engine implementation + - + * - `media/webrtc/libwebrtcglue <https://searchfox.org/mozilla-central/source/dom/media/webrtc/libwebrtcglue>`__ + - WebRTC (various) + - This is the glue code between libwebrtc and Firefox + - + * - `dom/media/webrtc/sdp <https://searchfox.org/mozilla-central/source/dom/media/webrtc/sdp>`__ + - Signalling + - This contains the SDP parsing interface + - + * - `dom/media/webrtc/tests <https://searchfox.org/mozilla-central/source/dom/media/webrtc/tests>`__ + - Tests + - This contains `some` of the WebRTC related tests + - + * - `dom/media/webrtc/third_party_build <https://searchfox.org/mozilla-central/source/dom/media/webrtc/third_party_build>`__ + - Build + - The scripting and configuration for vendoring new versions of libwebrtc are here + - This is unlikely to be of concern for debugging + * - `dom/media/webrtc/transport <https://searchfox.org/mozilla-central/source/dom/media/webrtc/transport>`__ + - Network + - This contains the ICE implementation, the MDNS implementation, and transport code + - + * - `dom/media/webrtc/transportbridge <https://searchfox.org/mozilla-central/source/dom/media/webrtc/transportbridge>`__ + - WebRTC + - This contains the MediaPipeline and MediaPipeline filter code which is glue between transport and the libwebrtc RTP stack + - + * - `third_party/libsrtp <https://searchfox.org/mozilla-central/source/third_party/libsrtp>`__ + - Network + - This is the SRTP implementation used by Firefox + - + * - `third_party/libwebrtc <https://searchfox.org/mozilla-central/source/third_party/libwebrtc>`__ + - WebRTC (various) + - libwebrtc handles many aspects of WebRTC calls above the transport layer and below the presentation layer + - + * - `third_party/rust/webrtc-sdp <https://searchfox.org/mozilla-central/source/third_party/rust/webrtc-sdp>`__ + - Signalling + - webrtc-sdp is a Rust implementation of a WebRTC-only SDP parser + - + * - `third_party/sipcc <https://searchfox.org/mozilla-central/source/third_party/sipcc>`__ + - Signalling + - sipcc is a C implementation of a general SDP parser + - this carries many local modifications + * - `dom/media <https://searchfox.org/mozilla-central/source/dom/media>`__ + - Media Capture + - GetUserMedia and related classes are here + - There are many other unrelated media source files here + * - `dom/webidl <https://searchfox.org/mozilla-central/source/dom/webidl>`__ + - WebIDL (JS API) + - This contains the WebIDL definitions for the WebRTC JS API amongst many other WebIDL definitions + - :code:`RTC*.webidl` + +.. _standards-documentation: + +Standards Documentation +----------------------- + +When debugging API behavior it may be necessary to consult the specifications +for WebRTC. The ECMAScript API is defined in several W3C standards, +`webrtc-pc <https://www.w3.org/TR/webrtc/>`__, and +`webrtc-stats <https://www.w3.org/TR/webrtc-stats/>`__. The number of IETF +standards that are incorporated into WebRTC are too numerous to list here. One +can find these standards in the +`Normative References <https://www.w3.org/TR/webrtc/#normative-references>`__ +section of the :code:`webrtc-pc` spec. + +.. _delay-calculation: + +Appendix: Delay Calculation +--------------------------- + +For all intents and purposes jitter and RTT are +additive in nature. If there was 25ms of jitter reported and a RTT of +272ms, one could estimate the expected delay from transmission at the +send side to play out on receive side to be + +:: + + 25ms + (272ms / 2) = 161ms diff --git a/docs/contributing/debugging/img/about-processes.png b/docs/contributing/debugging/img/about-processes.png Binary files differnew file mode 100644 index 0000000000..6f1563bcd1 --- /dev/null +++ b/docs/contributing/debugging/img/about-processes.png diff --git a/docs/contributing/debugging/img/about-support.png b/docs/contributing/debugging/img/about-support.png Binary files differnew file mode 100644 index 0000000000..2b12d1027d --- /dev/null +++ b/docs/contributing/debugging/img/about-support.png diff --git a/docs/contributing/debugging/img/about_webrtc_Jitter_and_RTT.png b/docs/contributing/debugging/img/about_webrtc_Jitter_and_RTT.png Binary files differnew file mode 100644 index 0000000000..0cb6ad329a --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_Jitter_and_RTT.png diff --git a/docs/contributing/debugging/img/about_webrtc_copy_report.png b/docs/contributing/debugging/img/about_webrtc_copy_report.png Binary files differnew file mode 100644 index 0000000000..79d8dcf9c3 --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_copy_report.png diff --git a/docs/contributing/debugging/img/about_webrtc_default_view.png b/docs/contributing/debugging/img/about_webrtc_default_view.png Binary files differnew file mode 100644 index 0000000000..4b65a666cc --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_default_view.png diff --git a/docs/contributing/debugging/img/about_webrtc_media_context.png b/docs/contributing/debugging/img/about_webrtc_media_context.png Binary files differnew file mode 100644 index 0000000000..a4a6ee11d4 --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_media_context.png diff --git a/docs/contributing/debugging/img/about_webrtc_reveal_PeerConnection_stats.png b/docs/contributing/debugging/img/about_webrtc_reveal_PeerConnection_stats.png Binary files differnew file mode 100644 index 0000000000..f877703b71 --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_reveal_PeerConnection_stats.png diff --git a/docs/contributing/debugging/img/crash-gmp.png b/docs/contributing/debugging/img/crash-gmp.png Binary files differnew file mode 100644 index 0000000000..595e439307 --- /dev/null +++ b/docs/contributing/debugging/img/crash-gmp.png diff --git a/docs/contributing/debugging/img/crash-gpu.png b/docs/contributing/debugging/img/crash-gpu.png Binary files differnew file mode 100644 index 0000000000..9e40b63917 --- /dev/null +++ b/docs/contributing/debugging/img/crash-gpu.png diff --git a/docs/contributing/debugging/img/crash-rdd.png b/docs/contributing/debugging/img/crash-rdd.png Binary files differnew file mode 100644 index 0000000000..5548672d27 --- /dev/null +++ b/docs/contributing/debugging/img/crash-rdd.png diff --git a/docs/contributing/debugging/img/crashlist.jpg b/docs/contributing/debugging/img/crashlist.jpg Binary files differnew file mode 100644 index 0000000000..4cb376d0f2 --- /dev/null +++ b/docs/contributing/debugging/img/crashlist.jpg diff --git a/docs/contributing/debugging/img/crashreporter.png b/docs/contributing/debugging/img/crashreporter.png Binary files differnew file mode 100644 index 0000000000..3137c71c78 --- /dev/null +++ b/docs/contributing/debugging/img/crashreporter.png diff --git a/docs/contributing/debugging/img/process-explorer.png b/docs/contributing/debugging/img/process-explorer.png Binary files differnew file mode 100644 index 0000000000..26e7837a8d --- /dev/null +++ b/docs/contributing/debugging/img/process-explorer.png diff --git a/docs/contributing/debugging/img/sdk-installer.png b/docs/contributing/debugging/img/sdk-installer.png Binary files differnew file mode 100644 index 0000000000..c7dcc1f1a4 --- /dev/null +++ b/docs/contributing/debugging/img/sdk-installer.png diff --git a/docs/contributing/debugging/img/searchfox_test_disabled_warning.png b/docs/contributing/debugging/img/searchfox_test_disabled_warning.png Binary files differnew file mode 100644 index 0000000000..a1815262b8 --- /dev/null +++ b/docs/contributing/debugging/img/searchfox_test_disabled_warning.png diff --git a/docs/contributing/debugging/img/tabcrashed.png b/docs/contributing/debugging/img/tabcrashed.png Binary files differnew file mode 100644 index 0000000000..9982e527ed --- /dev/null +++ b/docs/contributing/debugging/img/tabcrashed.png diff --git a/docs/contributing/debugging/img/windbg-in-startmenu.png b/docs/contributing/debugging/img/windbg-in-startmenu.png Binary files differnew file mode 100644 index 0000000000..0819d4b424 --- /dev/null +++ b/docs/contributing/debugging/img/windbg-in-startmenu.png diff --git a/docs/contributing/debugging/local_symbols.rst b/docs/contributing/debugging/local_symbols.rst new file mode 100644 index 0000000000..875717e1a1 --- /dev/null +++ b/docs/contributing/debugging/local_symbols.rst @@ -0,0 +1,64 @@ +Symbolicating TreeHerder stacks locally +======================================= + +When using tools like the :ref:`Dark Matter Detector (DMD)` or +:ref:`refcount logging<Refcount Tracing and Balancing>` to +investigate issues occurring on TreeHerder that you can't reproduce locally, you +can often end up with unsymbolicated stacks. Fortunately, there is a way to +symbolicate these stacks on your own machine. + +These instructions are for a Linux TreeHerder build for MacOS, so they might +require some modifications for other combinations of platforms. + +Download ``target.tar.bz2`` and ``target.crashreporter-symbols.zip`` from the +Build job. **Note that these files are very large so you'll want to delete +them and the extracted files when you are done.** + +These files each contain a large number of files, so I'd recommend creating +a directory for each of them. Call these ``<TARGET_DIR>`` and ``<SYMB_DIR>``, +and move the prior two files into these two directories, respectively. + +Go to ``<TARGET_DIR>`` and run something like + +.. code-block:: shell + + tar xf target.tar.bz2 + +then go to ``<SYMB_DIR>`` and run something like + +.. code-block:: shell + + unzip target.crashreporter-symbols.zip + +You should be able to delete the two original files now. + +Next we need to ensure that the locations of binaries are rewritten from +where they are on TreeHerder to where we have them locally. We'll do this by +editing ``fix_stacks.py``. This file is located in the ``tools/rb/`` directory of +the Firefox source directory. You need to add these two lines to the function +``fixSymbols``, after ``line_str`` is defined and before it is written to +``fix_stacks.stdin``. I've done this right before the definition of +``is_missing_newline``. + +.. code-block:: python + + line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox", + "<TARGET_DIR>/firefox/firefox-bin") + line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so", + "<TARGET_DIR>/firefox/libxul.so") + +The initial locations should appear verbatim in the stack you are trying to +symbolicate, so double check that they match. Also, ``<TARGET_DIR>`` of course +needs to be replaced with the actual local directories where those files are +located. Note that the ``firefox`` executable is changed to ``firefox-bin``. +I don't know why that is necessary, but only the latter existed for me. + +Finally, we need to make it so that the stack fixer can find the location of +the breakpad symbols we downloaded. If you are running ``fix_stacks.py`` via +``dmd.py`` or directly (in a recent version), you can do this by running with the +environment variable ``BREAKPAD_SYMBOLS_PATH`` set to the ``<SYMB_DIR>`` from above. +If that doesn't work, you'll have to edit ``initFixStacks`` in ``fix_stacks.py`` to +set ``breakpadSymsDir`` to ``<SYMB_DIR>``. + +With all of that done, you should now be able to run ``dmd.py`` or ``fix_stacks.py`` +to fix the stacks. Note that the stack fixing process can take a minute or two. diff --git a/docs/contributing/debugging/process_dump_task_manager.rst b/docs/contributing/debugging/process_dump_task_manager.rst new file mode 100644 index 0000000000..d345171856 --- /dev/null +++ b/docs/contributing/debugging/process_dump_task_manager.rst @@ -0,0 +1,69 @@ +How to get a process dump with Windows Task Manager +=================================================== + +Introduction +------------ + +When tracking down the causes of process hangs, it is often helpful to +obtain a process dump while the process is experiencing a hang. This +article describes how to get a process dump with Task Manager on +Windows. (To get a process dump for Thunderbird or some other product, +substitute the product name where ever you see Firefox in these +instructions.) + + +Caution +------- + +The memory dump that will be created through this process is a complete +snapshot of the state of Firefox when you create the file, so it +contains URLs of active tabs, history information, and possibly even +passwords depending on what you are doing when the snapshot is taken. It +is advisable to create a new, blank profile to use when reproducing the +hang and capturing the memory dump. Please ask for help doing this! + + +Requirements +------------ + +Windows + To get a process dump, you need to be using Windows Vista or above. +A Firefox nightly or release + You need a Firefox version for which symbols are available from the + :ref:`symbol server <Using The Mozilla Symbol Server>`. You + can use any `official nightly + build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released + version of Firefox from Mozilla. You can find the latest trunk + nightly builds under + `http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/>`__. + + +Creating the Dump File +---------------------- + +Ensure that Firefox is not already running. + + +Run Firefox, reproduce the hang +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Start Firefox and perform whatever steps are necessary to cause Firefox +to hang. Once the browser hangs, continue with the steps below. + + +After the hang +~~~~~~~~~~~~~~ + +#. Open Windows Task Manager (CTRL+SHIFT+ESC). +#. Find Firefox.exe among the list of processes. +#. Right-click Firefox.exe and select "Create dump file". Task manager + should indicate where the dump file was written to. + + +See also +-------- + +- :ref:`How to get a stacktrace for a bug report <How to get a stacktrace for a bug report>` +- `How to create a user-mode process dump file in Windows Vista and in + Windows 7 + (MSDN) <https://docs.microsoft.com/en-us/windows/client-management/generate-kernel-or-complete-crash-dump#manually-generate-a-memory-dump-file>`__ diff --git a/docs/contributing/debugging/stacktrace_report.rst b/docs/contributing/debugging/stacktrace_report.rst new file mode 100644 index 0000000000..08f4bded00 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_report.rst @@ -0,0 +1,153 @@ +How to get a stacktrace for a bug report +======================================== + +If you file a bug report in Bugzilla about a crash you should include a +stacktrace (call stack) in your report. A stacktrace will tell Mozilla +developers what crashed and provide a starting point for investigating +its cause. This article describes how to use the Mozilla Crash Reporter +(Breakpad) to get a crash ID, which our engineers can use to get a +stacktrace, and alternative ways to get a stacktrace if you can't get a +crash ID. + +Requirements +------------ + +You need a binary build of Firefox from +`Mozilla.org <https://www.mozilla.org/firefox/>`__. SeaMonkey and +Thunderbird also support crash reporting. + +Mozilla's crash report server currently only has debug information for +Mozilla builds and thus the crash reporter cannot work if you use a +build from a Linux distribution or if you compile from source code. In +these cases you will need to use one of the :ref:`alternative +methods <Alternative ways to get a stacktrace>` listed below. + +.. note:: + + **Note:** When filing a crash report, it is important to know whether + the crash occurs with `Firefox safe + mode <http://support.mozilla.com/kb/Safe+Mode>`__. This helps + engineers determine whether a particular + `extension <http://support.mozilla.com/kb/Troubleshooting+extensions+and+themes>`__ + or + `plugin <http://support.mozilla.com/kb/Troubleshooting+plugins>`__ + is the cause of the crash. + + +How to get a crash ID with the Mozilla Crash Reporter +----------------------------------------------------- + +1 - Crash and submit a report to the system. + +.. image:: img/crashreporter.png + +The Mozilla Crash Reporter window should automatically come up after Firefox crashes. +If you have any additional information about the crash, such as additional detail on +what you were doing at the time that may have triggered the crash, please enter it +into the comments box. Be sure that you **check the "Tell Mozilla about this crash"** +checkbox and click the restart button. The crash reporter should now submit the +crash report and Firefox should open again. + +.. note:: + + The "Details" button gives additional data about the incident, + however this is not useful in a bug report. + + +2 - Tell us the ID of the report you submitted. + +.. image:: img/crashlist.jpg + +To access all of your submitted reports type "about:crashes" into the Firefox address bar +and press enter. Firefox should open a list of IDs for your submitted crash reports. +Copy two or three of the IDs for the appropriate crashes and paste them into your +Bugzilla report. Please check the listed times to avoid copying the ID of an unrelated +crash report. + +.. note:: + + You can prefix a "bp-" to the beginning of an ID to make Bugzilla turn it + into a link: bp-a70759c6-1295-4160-aa30-bc4772090918 + + +How to get the crash ID if Firefox crashes on startup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox crashes on startup you can still access your submitted crash +reports. Crash reports are accessible from all Firefox profiles, so if a +`new +profile <https://support.mozilla.org/kb/profile-manager-create-remove-switch-firefox-profiles>`__ +does not crash you can use it to access them through "about:crashes" as above. + + +Accessing crash report IDs outside of Firefox +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you cannot load Firefox at all you can find the crash report files at +this location depending on your operating system: + +* Windows : ``%APPDATA%\Mozilla\Firefox\Crash Reports\submitted\`` +* macOS : ``~/Library/Application Support/Firefox/Crash Reports/submitted/`` +* Linux : ``~/.mozilla/firefox/Crash Reports/submitted/`` + +Each file in this folder contains one submitted crash report ID. You can +check the modified or creation time for each file to discern which crash +reports are relevant to your bug report. + +.. _Alternative ways to get a stacktrace: + +Alternative ways to get a stacktrace +------------------------------------ + +If the Mozilla crash reporter doesn't come up or isn't available you +will need to obtain a stacktrace manually: + + +Windows +~~~~~~~ + +See the article :ref:`Create a stacktrace with Windbg <How to get a stacktrace with WinDbg>` for information +on how to do this. + +For a full process dump, see :ref:`How to get a process dump with Windows +Task Manager`. + + +macOS +~~~~~ + +Run /Applications/Utilities/Console.app. Expand "~/Library/Logs" and +"CrashReporter", then look for logs for "firefox". + + +Linux +~~~~~ + +Note that for most distros, the package you need to get symbols for will +be something like "xulrunner", not "firefox". + + +Crash reports files on your computer +------------------------------------ + +When Breakpad initially catches a crash it first writes crash report +files (e.g. .dump and .extra files) into the 'pending' subdirectory of +its 'Crash Reports' directory. + +If Breakpad successfully sends the crash report to the reporting server +then, by default, the files added to the 'pending' subdirectory for the +crash are removed, and a .txt file is placed in the 'submitted' +directory containing the crash ID created by the reporting server. + +If you want Breakpad to leave the .dump and .extra files on your +computer so that you can examine them locally, then set the +MOZ_CRASHREPORTER_NO_DELETE_DUMP environment variable to 1. + +- Ubuntu: `Instructions from the Ubuntu + Team <https://wiki.ubuntu.com/MozillaTeam/Bugs#Obtain%20a%20backtrace%20from%20an%20apport%20crash%20report%20(using%20gdb)>`__ +- openSUSE: `General instructions from + openSUSE <https://en.opensuse.org/openSUSE:Bugreport_application_crashed>`__ +- Fedora: `Capturing Stack + Traces <https://fedoraproject.org/wiki/StackTraces>`__ +- Gentoo: `Debugging using + GDB <https://wiki.gentoo.org/wiki/Debugging_with_GDB>`__ diff --git a/docs/contributing/debugging/stacktrace_windbg.rst b/docs/contributing/debugging/stacktrace_windbg.rst new file mode 100644 index 0000000000..a0d2abfc25 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_windbg.rst @@ -0,0 +1,232 @@ +How to get a stacktrace with WinDbg +=================================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Introduction +------------ + +Sometimes you need to get a stacktrace (call stack) for a crash or hang +but `Breakpad <http://kb.mozillazine.org/Breakpad>`__ fails because it's +a special crash or a hang. This article describes how to get a +stacktrace in those cases with WinDbg on Windows. (To get a stacktrace +for Thunderbird or some other product, substitute the product name where +ever you see Firefox in this instructions.) + +Requirements +------------ + +To get such a stacktrace you need to install the following software: + +Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Microsoft distributes the Debugging Tools for Windows for free, those +include WinDbg which you will need here. Download it from `Install +Debugging Tools for +Windows <https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk>`__. +(*You'll want the 32-bit version*, even if you are using a 64-bit +version of Windows) Then install it, the standard settings in the +installation process are fine. + +A Firefox nightly or release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You need a Firefox version for which symbols are availables from the +:ref:`symbol server <Using The Mozilla Symbol Server>` to use +with WinDbg. You can use any `official nightly +build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released +version of Firefox from Mozilla. You can find the latest trunk nightly +builds under +`http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/>`__. + + +Debugging +--------- + +To begin debugging, ensure that Firefox is not already running and open +WinDbg from the Start menu. (Start->All Programs->Debugging Tools for +Windows->WinDbg) Next, open the **"File"** menu and choose **"Open +Executable..."**. In the file chooser window that appears, open the +firefox.exe executable in your Firefox program folder (C:\Program +Files\Mozilla Firefox). + +You should now see a "Command" text window with debug output at the top +and an input box at the bottom. Before debugging can start, several +commands must be entered into the one-line input box at the bottom of +the Command window. + +.. note:: + + Tip: All commands must be entered exactly as written, one line at a + time, into the bottom of the Command box. + + - Copying and pasting each line is the easiest method to avoid + mistakes + - Some commands start with a period (.) or a pipe character (|), + which is required. (The keystroke for a pipe character on US + keyboards is SHIFT+\) + - Submit the log file on a bug or via the support site, even if + nothing seems to happen during the debug process + + +Start debugging +~~~~~~~~~~~~~~~ + +Now that Firefox is opened in the debugger, you need to configure your +WinDbg to download symbols from the Mozilla symbol server. To load the +symbols, enter the three commands below, pressing enter after each one. +(More details are available at :ref:`symbol server <Using The Mozilla Symbol Server>`.) + +:: + + .sympath SRV*c:\symbols*http://symbols.mozilla.org/firefox;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols + .symfix+ c:\symbols + .reload /f + +Now wait for the symbols to download. This may take some time depending +on your connection speed; the total size of the Mozilla and Microsoft +symbols download is around 1.4GB. WinDbg will show "Busy" at the bottom +of the application window until the download is complete. + +Once the download is complete, you need to configure WinDbg to examine +child processes, ignore a specific event caused by Flash Player, and +record a log of loaded modules. You will also want to open a log file to +save data you collect. To do this, enter these four commands, pressing +enter after each one. + +:: + + .logopen /t c:\temp\firefox-debug.log + .childdbg 1 + .tlist + sxn gp + lm + +If you see firefox.exe listed in the output from .tlist more than once, +then you are already running the application and need to close the +running instance first before you start debugging, otherwise you won't +get useful results. + +Now run Firefox by opening the **Debug** menu and clicking **Go**. +**While Firefox is running, you will not be able to type any commands +into the debugger.** After it starts, try to reproduce the crash or +hanging issue that you are seeing. + +.. note:: + + If Firefox fails to start, and you see lines of text followed by a + command prompt in the debugger, a "breakpoint" may have been + triggered. If you are prompted for a command but don't see an error + about a crash, go back to the **Debug** menu and press **Go**. + +Once the browser crashes, you will see an error (such as "Access +violation") in the WinDbg Command window. If Firefox hangs and there is +no command prompt available in the debugger, open the **Debug** menu and +choose **Break.** Once the browser has crashed or been stopped, continue +with the steps below. + + +After the crash or hang +~~~~~~~~~~~~~~~~~~~~~~~ + +You need to capture the debug information to include in a bug comment or +support request. Enter these three commands, one at a time, to get the +stacktrace, crash/hang analysis and log of loaded modules. (Again, press +Enter after each command.) + +:: + + ~* kp + !analyze -v -f + lm + +After these steps are completed, find the file +**c:\temp\firefox-debug-(Today's Date).txt** on your hard drive. To +provide the information to the development community, submit this file +with a `support request <https://support.mozilla.com/>`__ or attach it +to a related bug on `Bugzilla <https://bugzilla.mozilla.org/>`__. + + +Producing a minidump +~~~~~~~~~~~~~~~~~~~~ + +Sometimes the stacktrace alone is not enough information for a developer +to figure out what went wrong. A developer may ask you for a "minidump" +or a "full memory dump", which are files containing more information +about the process. :ref:`You can easily produce minidumps from WinDBG and +provide them to developers <Capturing a minidump>`. + +FAQ + +Q: I am running Windows 7 (32-bit or 64-bit) and I see an exception in +the WinDbg command window that says 'ntdll32!LdrpDoDebuggerBreak+0x2c' +or 'ntdll32!LdrpDoDebuggerBreak+0x30'. What do I do now? + +A: If you see 'int 3' after either of those exceptions, you will need to +execute the following commands in WinDbg. + +:: + + bp ntdll!LdrpDoDebuggerBreak+0x30 + bp ntdll!LdrpDoDebuggerBreak+0x2c + eb ntdll!LdrpDoDebuggerBreak+0x30 0x90 + eb ntdll!LdrpDoDebuggerBreak+0x2c 0x90 + +| Make sure you enter them one at a time and press enter after each one. + If you use the 64-bit version of Windows, you need to replace "ntdll" + in these commands with "ntdll32". +| Q: The first four frames of my stack trace look like this: + +:: + + 0012fe20 7c90e89a ntdll!KiFastSystemCallRet + 0012fe24 7c81cd96 ntdll!ZwTerminateProcess+0xc + 0012ff20 7c81cdee kernel32!_ExitProcess+0x62 + + 0012ff34 6000179e kernel32!ExitProcess+0x14 + +This looks wrong to me?! + +A: You ran the application without the "Debug child processes also" +check box being checked. You need to detach the debugger and open the +application again, this time with the check box being checked. + +Q: WinDbg tells me that it is unable to verify checksum for firefox.exe. +Is this normal? + +A: Yes, this is normal and can be ignored. + +Q: Should I click yes or no when WinDbg asks me to "Save information for +workspace?" + +A: Click yes and WinDbg will save you from having to enter in the symbol +location for Firefox.exe in the future. Click no if you'd rather not +having WinDbg save this information. + +Q: I'm seeing "wow64" on top of each thread, is that ok ? + +A: No, you are running a 64 bit version of Windbg and trying to debug a +32 bit version of the mozilla software. Redownload and install the 32 +bit version of windbg. + + +Troubleshooting: Symbols will not download +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If symbols will not download no matter what you do, the problem may be +that Internet Explorer has been set to the **Work Offline** mode. You +will not receive any warnings of this in Windbg, Visual C++ or Visual +Studio. Even using the command line with symchk.exe to download symbols +will fail. This is because Microsoft uses Internet Explorer's internet & +proxy settings to download the symbol files. Check the File menu of +Internet Explorer to ensure "Work Offline" is unchecked. + + +See also +-------- + +- :ref:`symbol server <Using The Mozilla Symbol Server>` Maps addresses to human readable strings. +- :ref:`source server <Using The Mozilla Source Server>` Maps addresses to source code lines diff --git a/docs/contributing/debugging/understanding_crash_reports.rst b/docs/contributing/debugging/understanding_crash_reports.rst new file mode 100644 index 0000000000..9f8e5bbe03 --- /dev/null +++ b/docs/contributing/debugging/understanding_crash_reports.rst @@ -0,0 +1,325 @@ +Understanding Crash Reports +=========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +If a user experiences a crash they will be prompted to submit a raw +crash report, which is generated by Breakpad. The raw crash report is +received by `Socorro <https://github.com/mozilla/socorro>`__ which +`creates <https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py>`__ +a processed crash report. The processed crash report is based on the raw +crash report but also has a signature, classifications, and a number of +improved fields (e.g. OS, product, version). Many of the fields in both +the raw crash report and the processed crash report are viewable and +searchable on `crash-stats <https://crash-stats.mozilla.org/>`__. +Although there are two distinct crash reports, the raw and the +processed, people typically talk about a single "crash report" because +crash-stats mostly presents them in a combined way. + +Each crash report contains a wealth of data about the crash +circumstances. Despite this, many crash reports lack sufficient data for +a developer to understand why the crash occurred. As well as providing a +general overview, this page aims to highlight parts of a crash report +that may provide non-obvious insights. + +Note that most crash report fields are visible, but a few +privacy-sensitive parts of it are only available to users who are logged +in and have "minidump access". A relatively small number of users have +minidump access, and they are required to follow certain rules. For +access, see the `Protected Data Access docs on Crash Stats +<https://crash-stats.mozilla.org/documentation/protected_data_access/>`__. + +Each crash report has the following tabs: Details, Metadata, Modules, +Raw Dump, Extensions, and (optional) Correlations. + +Details tab +----------- + +The Details tab is the first place to look because it contains the most +important pieces of information. + +Primary fields +~~~~~~~~~~~~~~ + +| The first part of the Details tab shows a table containing the most + important crash report fields. It includes such things as when the + crash occurred, in which product and version, the crash kind, and + various details about the OS and configuration of the machine on which + the crash occurred. The following screenshot shows some of these + fields. +| |Example fields in the "Details" tab of a crash report| + +All fields have a tool-tip. For many fields, the tool-tip describes its +meaning. For all fields, the tool-tip indicates the key to use when you +want to do searches involving this field. (The field name is usually but +not always similar to the search key. E.g. the field "Adapter Device ID" +has the search key "adapter_device_id".) These descriptions are shown in +the `SuperSearchFields +API <https://crash-stats.mozilla.org/api/SuperSearchFields/>`__ and can be +`modified in super_search_fields.py <https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py>`__ +or by writing up a `bug in Socorro <https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro>`__. + +The fields present in this tab vary depending on the crash kind. Not all +fields are always present. + +The "Signature" field is the main identifier or label for a crash report. +Rather than considering each crash report in isolation, we want to put +crash reports into clusters so we can deal with groups of them at once. +An ideal clustering algorithm would put all crash reports with the same +root cause into a single cluster, and all crash reports with different +root causes into different clusters. The crash signature is our +imperfect but still useful attempt at such an algorithm. Most crash +signatures are based on the crashing stack trace, but some +special-purpose annotations are used to indicate particular kinds of +crashes. + +- ``Abort``: A controlled abort, e.g. via ``NS_RUNTIMEABORT``. + (Controlled aborts that occur via ``MOZ_CRASH`` or + ``MOZ_RELEASE_ASSERT`` currently don't get an ``Abort`` annotation, + but they do get a "MOZ_CRASH Reason" field.) +- ``OOM | <size>``, where ``<size>`` is one of ``large``, ``small``, + ``unknown``: an out-of-memory (OOM) abort. The ``<size>`` annotation + is determined by the "OOM Allocation Size" field; if that field is + missing ``<size>`` will be ``unknown``. +- ``hang``: a hang prior to shutdown. +- ``shutdownhang``: a hang during shutdown. +- ``IPCError-browser``: a problem involving IPC. If the parent Firefox + process detects that the child process has sent broken or + unprocessable IPDL data, or is not shutting down in a timely manner, + it kills the child process with a crash report. These crashes will + now have a signature that indicates why the process was killed, + rather than the child stack at the moment. + +When no special-purpose annotation is present and the signature begins +with a stack frame, it's usually a vanilla uncontrolled crash. The crash +cause can be determined from the "Crash Reason" field. Most commonly +it's a bad memory access. In that case, on Windows you can tell from the +reason field if the crash occurred while reading, writing or executing +memory (e.g. ``EXCEPTION_VIOLATION_ACCESS_READ`` indicates a bad memory +read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you +cannot tell from this field what kind of memory access it was. + +See `this +file <https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst>`__ +for a detailed explanation of the crash report signature generation +procedure, and for information on how modify this procedure. + +There are no fields that uniquely identify the user that a crash report +came from, but if you want to know if multiple crashes come from a +single user the "Install Time" field is a good choice. Use it in +conjunction with other fields that don't change, such as those +describing the OS or graphics card, for additional confidence. + +For bad memory accesses, the "Crash Address" field can give additional +indications what went wrong. + +- 0x0 is probably a null pointer deference[*]. +- Small addresses like 0x8 can indicate an object access (e.g. + ``this->mFoo``) via a null ``this`` pointer. +- Addresses like 0xfffffffffd8 might be stack accesses, depending on + the platform[*]. +- Addresses like 0x80cdefd3 might be heap accesses, depending on the + platform. +- Addresses may be poisoned: 0xe4 indicates the address comes from + memory that has been allocated by jemalloc but not yet initialized; + 0xe5 indicates the address comes from memory freed by jemalloc. The + JS engine also has multiple poison values defined in + ``js/src/jsutil.h``. + +[*] Note that due to the way addressing works on x86-64, if the crash +address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for +a Windows crash report, it's highly likely that the value is incorrect. +(There is a `bug +report <https://bugzilla.mozilla.org/show_bug.cgi?id=1493342>`__ open +for this problem.) You can sanity-check these crashes by looking at the +raw dump or minidump in the Raw Dump tab (see below). + +Note that for non-release builds the "Version" field represents multiple +different builds since nightly and beta version numbers are reused for +builds created over a series of days until the version number is bumped. +(The "Build ID" field can disambiguate.) It's not currently possible to +`restrict searches to a given version or +later <https://bugzilla.mozilla.org/show_bug.cgi?id=1401517>`__ (using +>= with a build ID and a given release channel may work around this). + +Some fields, such as "URL" and "Email Address", are privacy-sensitive +and are only visible to users with minidump access. + +The Windows-only "Total Virtual Memory" field indicates if the Firefox +build and OS are 32-bit or 64-bit. + +- A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows. +- A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows + (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox. +- A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit + Firefox. (The "Build Architecture" field should be "amd64" in this + case.) + +Some crash reports might contain a memory report. This memory report will +have been made some time before the crash, at a time when available +memory was low. In this case, a number of key measurements from the +memory report are shown in the Details tab, each one having a field name +starting with "MR:", short for "memory report". The full memory report +can be obtained in the Raw Dump tab (see below). + +Bug-related information +~~~~~~~~~~~~~~~~~~~~~~~ + +The second part of the Details tab shows bug-related information, as the +following screenshot shows. + +|Information relating to bug reports in the "Details" tab of a crash +report| + +The "Report this bug in" links can be used to easily file bug reports. +Each one links to a Bugzilla bug report creation page that has various +fields pre-filled, such as the crash signature. + +The "Related Bugs" section shows related bug reports, as determined by +the crash signature. + +Stack traces +~~~~~~~~~~~~ + +The third part of the Details tab shows the stack trace and thread +number of the crashing thread, as the following screenshot shows. + +|Information relating to threads in the "Details" tab of a crash report| + +Each stack frame has a link to the source code, when possible. If a +crash is new, the regressing changeset can often be identified by +looking for recent changes in the blame annotations for one or more of +the top stack frames. Blame annotations are also good for identifying +who might know about the code in question. + +Sometimes the highlighted source code is puzzling, e.g. the identified +line may not touch memory even though the crash is memory-related. This +can be caused by compiler optimizations. It's often better to look at +the disassembly (e.g. in a minidump) to understand exactly what code is +being executed. + +Stack frame entries take on a variety of forms. + +- The simplest are functions names, such as ``NS_InitXPCOM2``. +- Name/address pairs such as ``nss3.dll@0x1eb720`` are within system + libraries. +- Names such as ``F1398665248_____________________________`` ('F' + followed by many numbers then many underscores) are in Flash. +- Addresses such as ``@0xe1a850ac`` may indicate an address that wasn't + part of any legitimate code. If an address such as this occurs in the + first stack frame, the crash may be + `exploitable <https://developer.mozilla.org/en-US/docs/Mozilla/Security/Exploitable_crashes>`__. + +Stack traces for other threads can be viewed by clicking on the small +"Show other threads" link. + +If the crash report is for a hang, the crashing thread will be the +"watchdog" thread, which exists purely to detect hangs; its top stack +frame will be something +like\ :literal:`mozilla::`anonymous namespace'::RunWatchdog`. In that +case you should look at the other threads' stack traces to determine the +problem; many of them will be waiting on some kind of response, as shown +by a top stack frame containing a function like +``NtWaitForSingleObject`` or ``ZwWaitForMultipleObjects``. + +Metadata tab +------------ + +The Metadata tab is similar to the first part of the Details tab, +containing a table with various fields. These are the fields from the +raw crash report, ordered alphabetically by field name, but with +privacy-sensitive fields shown only to users with minidump access. There +is some overlap with the fields shown in the Details tab. + +Modules tab +----------- + +The modules tab shows all the system libraries loaded at the time of the +crash, as the following screenshot shows. + +|Table of modules in the "Modules" tab of a crash report| + +On Windows these are mostly DLLs, on Mac they are mostly ``.dylib`` +files, and on Linux they are mostly ``.so`` files. + +This information is most useful for Windows crashes, because DLLs loaded +by antivirus software or malware often cause Firefox to crash. +Correlations between loaded modules and crash signatures can be seen in +the "Correlations" tab (see below). + +`This page <https://support.mozilla.org/en-US/kb/helping-crashes>`__ +says that files lacking version/debug identifier/debug filename are +likely to be malware. + +Raw Dump tab +------------ + +The first part of the Raw Dump tab shows the raw crash report, in JSON +format. Once again, privacy-sensitive fields are shown only to users +with minidump access. + +|JSON data in the "Raw Dump" tab of a crash report| + +For users with minidump access, the second part of the Raw Dump tab has +some links, as the following screenshot shows. + +|Links to downloadable files in the "Raw Dump" tab of a crash report| + +These links are to the following items. + +#. A minidump. Minidumps can be extremely useful in understanding a + crash report; see :ref:`this page <Debugging A Minidump>` for an + explanation how to use them. +#. The aforementioned JSON raw crash report. +#. The memory report contained within the crash report. +#. The unredacted crash report, which has additional information. + +Extensions tab +-------------- + +The Extensions tab shows which extensions are installed and enabled. + +|Table of extensions in the "Extensions" tab of a crash report| + +Usually it just shows an ID rather than the proper extension name. + +Note that several extensions ship by default with Firefox and so will be +present in almost all crash reports. (The exact set of default +extensions depends on the release channel.) The least obvious of these +has an Id of ``{972ce4c6-7e08-4474-a285-3208198ce6fd}``, which is the +default Firefox theme. Some (but not all) of the other extensions +shipped by default have the following Ids: ``webcompat@mozilla.org``, +``e10srollout@mozilla.org``, ``firefox@getpocket.com``, +``flyweb@mozilla.org``, ``loop@mozilla.org``. + +If an extension only has a hexadecimal identifier, a Google search of +that identifier is usually enough to identify the extension's name. + +This information is useful because some crashes are caused by +extensions. Correlations between extensions and crash signatures can be +seen in the "Correlations" tab (see below). + +Correlations tab +---------------- + +This tab is only shown when crash-stats identifies correlations between +a crash and modules or extensions that are present, which happens +occasionally. + +See also +-------- + +- `A talk about understanding crash + reports <https://air.mozilla.org/a-talk-about-understanding-crash-reports/>`__, + by David Baron, from March 2016. +- :ref:`A guide to searching crash reports` + +.. |Example fields in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13579/Details1.png +.. |Information relating to bug reports in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13581/Details2.png +.. |Information relating to threads in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13583/Details3.png +.. |Table of modules in the "Modules" tab of a crash report| image:: https://mdn.mozillademos.org/files/13593/Modules1.png +.. |JSON data in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/13595/RawDump1.png +.. |Links to downloadable files in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/14047/raw-dump-links.png +.. |Table of extensions in the "Extensions" tab of a crash report| image:: https://mdn.mozillademos.org/files/13599/Extensions1.png |
