diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 19:33:14 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-07 19:33:14 +0000 |
| commit | 36d22d82aa202bb199967e9512281e9a53db42c9 (patch) | |
| tree | 105e8c98ddea1c1e4784a60a5a6410fa416be2de /docs/contributing/debugging | |
| parent | Initial commit. (diff) | |
| download | firefox-esr-36d22d82aa202bb199967e9512281e9a53db42c9.tar.xz firefox-esr-36d22d82aa202bb199967e9512281e9a53db42c9.zip | |
Adding upstream version 115.7.0esr.upstream/115.7.0esr
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'docs/contributing/debugging')
25 files changed, 2859 insertions, 0 deletions
diff --git a/docs/contributing/debugging/capturing_minidump.rst b/docs/contributing/debugging/capturing_minidump.rst new file mode 100644 index 0000000000..4c8f7591f5 --- /dev/null +++ b/docs/contributing/debugging/capturing_minidump.rst @@ -0,0 +1,259 @@ +Capturing a minidump +==================== + +*Minidumps* are files created by various Windows tools which record the +complete state of a program as it's running, or as it was at the moment +of a crash. Small minidumps are created by the Breakpad :ref:`crash +reporting <Crash Reporter>` tool, but sometimes that's not +sufficient to diagnose a problem. For example, if the application is +hanging (not responding to input, but hasn't crashed) then Breakpad is +not triggered, and it can be difficult to determine where the problem +lies. Sometimes a more complete form of minidump is needed to see +additional details about a crash, in which case manual capture of a +minidump is desired. + +This page describes how to capture these minidumps on Windows, to permit +better debugging. + + +Privacy and minidumps +--------------------- + +.. warning:: + + **Warning!** Unlike the minidumps submitted by Breakpad, these + minidumps contain the **complete** contents of program memory. They + are therefore much more likely to contain private information, if + there is any in the browser. For this reason, you may prefer to + generate minidumps against a `clean + profile <http://support.mozilla.com/en-US/kb/Managing%20profiles>`__ + where possible. + + +Capturing a minidump: application crash +--------------------------------------- + +To capture a full minidump for an application crash, you can use a tool called +**Debugging Tools for Windows**, which is provided by Microsoft for free. + + +Install Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please follow `these instructions +<https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/>`__. +You can install the Debugging Tools for Windows alone, but you cannot +download it alone. To install it, download Windows Software Development Kit +(SDK) first, start the installer, and then select only **Debugging Tools for +Windows** in the list of features to install. + +.. image:: img/sdk-installer.png + +The latest installer installs the tools for all CPU architectures (X86, +X64, ARM, and ARM64). You need to choose a tool of the architecture +matching Firefox you want to capture a minidump from. For example, if +you want to capture a minidump from 32-bit Firefox on 64-bit Windows, +use the X86 version of tools, not X64 tools. + +The default install path of SDK for 64-bit Windows is +``%ProgramFiles(x86)%\Windows Kits\10``. The debugging tools can be found in +the folder named **Debuggers** under your install path of SDK. + +The Debugging Tools for Windows contains both graphical and command line +debugging tools that can be used to capture a minidump. If you prefer +a graphical tool, please follow `Capture a minidump in a graphical way +<#capture-a-minidump-in-a-graphical-way>`__. If you prefer a command +line tool, please follow `Capture a minidump from the command line +<#capture-a-minidump-from-the-command-line>`__. + + +Capture a minidump in a graphical way +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#. Launch the graphical debugger named WinDbg from the Start Menu. WinDbg + icons are registered under "Windows Kits" as below. + + |WinDbg in Start Menu| + +#. Connect Firefox to the debugger. + + a. If Firefox is not already running, open the **"File"** menu on WinDbg + and choose **"Open Executable..."**. In the file chooser window that + appears, open the firefox.exe executable. If you're not sure about where + it is, please see `How to find the location of firefox.exe + <#how-to-find-the-location-of-firefox-exe>`__. + + b. If Firefox is already running and you know which process you want to + capture a minidump from, open the **"File"** menu on WinDbg and choose + **"Attach to a Process..."**. In the "Attach to Process" dialog that + appears, select the process. To identify a process, please see + `Identify a process to attach a debugger to + <#identify-a-process-to-attach-a-debugger-to>`__. + +#. You should now see a "Command" text window with debug output at the + top and an input box at the bottom. From the menu, select + ``Debug → Go``, and Firefox should start. If the debugger spits out + some text right away and Firefox doesn't come up, select + ``Debug → Go`` again. + +#. When the program is about to crash, WinDbg will spit out more data, + and the prompt at the bottom will change from saying "``*BUSY*``" to + having a number in it. At this point, you should type + "``.dump /ma c:\temp\firefoxcrash.dmp``" -- without the quotes, but + don't forget the dot at the beginning. Once it completes, which can + take a fair while, you will have a very large file at + ``c:\temp\firefoxcrash.dmp`` that can be used to help debug your + problem. File size will depend on this size of Firefox running in + your environment, which could several GB. + +#. Ask in the relevant bug or thread how best to share this very large + file! + + +Capture a minidump from the command line +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox is not already running, open the Command Prompt and run the following +command. This command launches all Firefox processes under a debugger. This +technique is useful when you have a startup crash or when you're not sure about +which process will crash. + +To find out where firefox.exe is, please see `How to find the location +of firefox.exe <#how-to-find-the-location-of-firefox-exe>`__. + +.. code:: + + <path to debugger>\cdb.exe -g -G -o <path to firefox>\firefox.exe + + +For example, if both the debugging tools and Firefox are installed in the +default folder and you want to capture a minidump of 64-bit Firefox, +the command will be like this. Please note that you need to add double +quotes when a path contains one or more whitespaces. + +.. code:: + + "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o "C:\Program Files\Mozilla Firefox\firefox.exe" + + +If a Firefox process you want to capture a minidump from is already running, +attach the debugger to it with the following command. To identify a process, +please see `Identify a process to attach a debugger to +<#identify-a-process-to-attach-a-debugger-to>`__. + +.. code:: + + <path to debugger>\cdb.exe -g -G -p <PID> + +When the process crashes, the debugger tool captures it and waits for your +command. At this point, you should type ``.dump /ma c:\temp\firefoxcrash.dmp`` +-- don't forget the dot at the beginning. Once it completes, which can take +a fair while, you will have a very large file at ``c:\temp\firefoxcrash.dmp`` +that can be used to help debug your problem. File size will depend on this +size of Firefox running in your environment, which could several GB. + +After a minidump is generated, type ``q`` and press Enter to quit the debugger. + + +Capturing a minidump: application hang +-------------------------------------- + +On Windows Vista and Windows 7, you can follow `these +instructions <http://support.microsoft.com/kb/931673>`__ to capture a +dump file and locate it after it's been saved. + + +Identify a process to attach a debugger to +------------------------------------------ + +When you're running Firefox, even if you have only a single tab, you may have +noticed a bunch of firefox.exe instances in Task Manager. This means Firefox +consists of multiple processes. Since an application crash happens per process +and a minidump is generated per process, you need to identify which process will +crash before starting a debugger. + +Identify a process type +~~~~~~~~~~~~~~~~~~~~~~~ + +Each instance of firefox.exe has a type. To identify a process to attach +a debugger to, you need to know its process type first. + +When a crash happens, if all Firefox windows are suddenly gone and Mozilla +Crash Reporter window is opend, a process that crashed is the main process. + +.. image:: img/crashreporter.png + +When a crash happens, if you see the following page, a process that crashed +is a tab (content) process. + +.. image:: img/tabcrashed.png + +There are more process types, but there isn't an easy way to detect a crash in +a process of those types because the symptom varies. If you cannot be sure +about the type of a crashing process, terminate Firefox and launch a new +instance of Firefox under a debugger in the way described above. + +If a GPU process crashes, you may see a window is not rendered correctly as +below. Since the main process relaunches a GPU process, this symptom will be +transient and the window will be rendered correctly again. + +.. image:: img/crash-gpu.png + +If a GMP (= Gecko Media Plugin) process crashes, you will see an information +bar will be displayed below the address bar. + +.. image:: img/crash-gmp.png + +If an RDD (= Remote Data Decoder) process crashes, Firefox may stop playing +a video as below, but not limited to this symptom. + +.. image:: img/crash-rdd.png + + +Identify a process ID (PID) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you understand what type of process crashes, the next step is to get a +process ID (PID), which is a value to specify in the debugger command we +discussed above. We present two ways to get a PID here. + +The first way is to use Firefox itself. Open a new tab and go to the +**about:processes** page. This page shows the list of all processes and their +PIDs. In the example below, the PID of the main process is **6308** and the +PID of the tab process hosting a page of mozilla.org is **6748**. + +.. image:: img/about-processes.png + +The second way is to use `Process Explorer +<https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer>`__, +which is provided by Microsoft for free. You may need this technique to attach +a debugger to a hung process or when you cannot open a new tab in the existing +Firefox for some reason. + +Process Explorer is basically an advanced version of Task Manager. Since it +displays processes in a hierarchical tree view, you can easily locate the main +process, which is the parent of all child processes. In the example below, the +PID of the main process is **6308** and all processes, including +plugin-container.exe, under the main process is child processes. + +Another helpful feature of Process Explorer is that when you hover the mouse +cursor on a process, it displays a tooltip window showing the process's command +line string. For a child process of firefox.exe, the command line's last +keyword shows the process type, so you can tell which process is which process +type. In the example below, the tooltip window displays a command line string +of a GPU process. + +.. image:: img/process-explorer.png + + +How to find the location of firefox.exe +--------------------------------------- + +If you're not sure about the location of the executable file (firefox.exe) of +Firefox you run, you can find it in the **about:support** page. In the +"Application Basics" section, the path to firefox.exe is displayed in the row +of "Application Binary". + +.. image:: img/about-support.png + +.. |WinDbg in Start Menu| image:: img/windbg-in-startmenu.png + :width: 50% diff --git a/docs/contributing/debugging/debugging_a_hang_on_macos.rst b/docs/contributing/debugging/debugging_a_hang_on_macos.rst new file mode 100644 index 0000000000..ee2bed16b2 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_hang_on_macos.rst @@ -0,0 +1,10 @@ +Debugging A Hang On macOS +========================= + +See `How to Report a Hung +Firefox <https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox>`_. + +See also +~~~~~~~~ + +`Debugging on macOS <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_macOS>`__ diff --git a/docs/contributing/debugging/debugging_a_minidump.rst b/docs/contributing/debugging/debugging_a_minidump.rst new file mode 100644 index 0000000000..5638c29d85 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_minidump.rst @@ -0,0 +1,202 @@ +Debugging A Minidump +==================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +The +`minidump <http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx>`__ +file format contains data about a crash on Windows. It is used by +`rust-minidump <https://github.com/luser/rust-minidump>`__, +`Breakpad <https://wiki.mozilla.org/Breakpad>`__, and also by various +Windows debugging tools. Each minidump includes the following data. + +- Details about the exception which led to the crash. +- Information about each thread in the process: the address which was + executing and the register state at the time the process stopped. +- A list of shared libraries loaded into the process at the time of the + crash. +- The stack memory of each thread. +- The memory right around the crashing address. +- (Optional) Other memory regions, if requested by the application. +- (Optional) Other platform-specific data. + +Accessing minidumps from crash reports +-------------------------------------- + +Minidumps are not available to everyone. For details on how to gain +access and where to find minidump files for crash reports, consult the +:ref:`crash report documentation <Understanding Crash Reports>` + +Using rust-minidump's tooling +----------------------------------- + +Most of our crash-reporting infrastructure is based on rust-minidump. +The primary tool for this is the +`minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ +CLI application, which includes extensive user documentation. + +That documentation includes a +`dedicated section <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk#analyzing-firefox-minidumps>`__ +on locally analyzing Firefox crashreports and minidumps. + +If you're looking for minidump_dump, it's included as part of +minidump-stackwalk. + +Using the MS Visual Studio debugger +----------------------------------- + +#. Set up the debugger to :ref:`use the Mozilla symbol + server <Using The Mozilla Symbol Server>` and + :ref:`source server <Using The Mozilla Source Server>`. +#. Double-click on the minidump file to open it in the debugger. +#. When it loads, click the green icon in the visual studio debugger + toolbar that looks like a play button. + +For Firefox releases older than Firefox 41, you will also need to +install the relevant release of Firefox (for example from +`here <https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/>`__), +and add the directory it is in (e.g., "C:\Program Files\Mozilla +Firefox 3.6 Beta 1\") to the same dialog in which you set up the +symbol server (in case the binary location in the minidump is not the +same as the one on your machine). Note that you can install the +relevant release anywhere. Just make sure to configure the symbol +server to the directory where you installed it. For releases from 41 +onward, the binaries are available on the symbol server. + +If this doesn't work, downloading the exact build and crashreporter +symbols full files. These can be found in treeherder / build folder. +Load Visual Studio, and go to file -> open -> minidump location. Click +on "Run Native", and Visual Studio will ask for the corresponding symbol +files. For each .dll you wish to have symbols for, you must go to a +console and go to the corresponding directory. E.g. (xul.dll should go +to xul.pdf in the crashreporter symbols directory). Each directory will +have a .pd\_ file. In a command shell run: "expand /r foo.pd\_". Then +point Visual Studio to this directory. + +Then you'll be able to examine: + ++------------------+-------------------------------------------------------------------------+ +| stack trace | The debugger shows the stack trace. You can right-click on any frame | +| | in the stack, and then choose "go to disassembly" or "go to source". | +| | (Choosing "go to disassembly" from the source view might not get you | +| | to the right place due to optimizations.) When looking at the | +| | source, beware that the debugging information will associate all | +| | inlined functions as part of the line into which they were inlined, | +| | and compiler (with PGO) inlines *very* aggressively (including | +| | inlining virtual functions). You can often figure out where you | +| | *really* are by reading the disassembly. | ++------------------+-------------------------------------------------------------------------+ +| registers | In the Registers tab (Debug->Windows->Registers if you don't have | +| | it open), you can look at the registers associated with each stack | +| | frame, but only at the current state (i.e., the time of the crash). | +| | Registers that Visual Studio can't figure out will be grayed-out and | +| | have the value 00000000. | ++------------------+-------------------------------------------------------------------------+ +| stack memory | You open a window (Memory 1, etc.) that shows contiguous segments of | +| | memory using the Debug->Windows->Memory menu item. You can then | +| | enter the address of the stack pointer (ESP register) in this window | +| | and look at the memory on the stack. (The minidump doesn't have the | +| | memory on the heap.) It's a good idea to change the "width" dropdown | +| | in the top right corner of the window from its default "Auto" to | +| | either "8" or "16" so that the memory display is word-aligned. If | +| | you're interested in pointers, which is usually the case, you can | +| | right click in this window and change the display to show 4-byte | +| | words (so that you don't have to reverse the order due to | +| | little-endianness). This view, combined with the disassembly, can | +| | often be used to reconstruct information beyond what in shown the | +| | function parameters. | ++------------------+-------------------------------------------------------------------------+ +| local variables | In the Watch 1 (etc.) window (which, if you don't have open, you can | +| | iget from Debug->Windows->Watch), you can type an expression | +| | (e.g., the name of a local variable) and the debugger will show you | +| | its value (although it sometimes gets confused). If you're looking | +| | at a pointer to a variable that happens to be on the stack, you can | +| | even examine member variables by just typing expressions. If Visual | +| | Studio can't figure something out from the minidump, it might show | +| | you 00000000 (is this true?). | ++------------------+-------------------------------------------------------------------------+ + +Using minidump-2-core on Linux +------------------------------ + +The `Breakpad +source <https://chromium.googlesource.com/breakpad/breakpad/+/master/>`__ +contains a tool called +`minidump-2-core <https://chromium.googlesource.com/breakpad/breakpad/+/master/src/tools/linux/md2core/>`__, +which converts Linux minidumps into core files. If you checkout and +build Breakpad, the binary will be at +``src/tools/linux/md2core/minidump-2-core``. Running the binary with the +path to a Linux minidump will generate a core file on stdout which can +then be loaded in gdb as usual. You will need to manually download the +matching Firefox binaries, but then you can use the :ref:`GDB Python +script <Downloading symbols on Linux / Mac OS X>` to download symbols. + +The ``minidump-2-core`` source does not currently handle processing +minidumps from a different CPU architecture than the system it was +built for. If you want to use it on an ARM dump, for example, you may +need to build the tool for ARM and run it under QEMU. + +Using other tools to inspect minidump data +------------------------------------------ + +Ted has a few tools that can be built against an already-built copy of +Breakpad to do more targeted inspection. All of these tools assume you +have checked out their source in a directory next to the breakpad +checkout, and that you have built Breakpad in an objdir named +``obj-breakpad`` at the same level. + +- `stackwalk-http <https://hg.mozilla.org/users/tmielczarek_mozilla.com/stackwalk-http/>`__ + is a version of the breakpad's minidump_stackwalk that can fetch symbols + over HTTP, and also has the Mozilla symbol server URL baked in. If you + run it like ``stackwalk /path/to/dmp /tmp/syms`` it will print the stack + trace and save the symbols it downloaded in ``/tmp/syms``. Note that + symbols are only uploaded to the symbol server for nightly and + release builds, not per-change builds. +- `dump-lookup <https://hg.mozilla.org/users/tmielczarek_mozilla.com/dump-lookup/>`__ + takes a minidump and prints values on the stack that are potential + return addresses. This is useful when a stack trace looks truncated + or otherwise wrong. It needs symbol files to produce useful output, + so you will generally want to have run ``stackwalk-http`` to download + them first. +- `get-minidump-instructions <https://hg.mozilla.org/users/tmielczarek_mozilla.com/get-minidump-instructions/>`__ + retrieves and displays the memory range surrounding the faulting + instruction pointer from a minidump. You will almost always want to + run it with the ``--disassemble`` option, which will make it send the + bytes through ``objdump`` to display the disassembled instructions. + If you also give it a path to symbols (see ``stackwalk-http`` above) + it can download the matching source files from hg.mozilla.org and + display source interleaved with the disassembly. +- `minidump-modules <http://hg.mozilla.org/users/tmielczarek_mozilla.com/minidump-modules>`__ + takes a minidump and prints the list of modules from the crashed + process. It will print the full path to each module, whereas the + Socorro UI only prints the filename for each module for privacy + reasons. It also accepts a -v option to print the debug ID for each + module, and a -d option to print relative paths to the symbol files + that would be used instead of the module filenames. + +Getting a stack trace from a crashed B2G process +------------------------------------------------ + +#. Get the minidump file in the phone at + /data/b2g/mozilla/\*.default/minidump/. You can use `adb + pull <http://developer.android.com/tools/help/adb.html>`__ for that. +#. Build the debug symbols using the command ./build.sh buildsymbols + inside the B2G tree. The symbol files will be generated in + $OBJDIR/dist/crashreporter-symbols. +#. Build and install + `google-breakpad <https://code.google.com/p/google-breakpad/>`__. +#. Use the + `minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ + tool to get the stack trace. + +.. code:: bash + + Example: + + $ cd B2G + $ adb pull /data/b2g/mozilla/*.default/minidump/*.dmp . + $ls *.dmp + 71788789-197e-d769-67167423-4e7aef32.dmp + $ minidump-stackwalk 71788789-197e-d769-67167423-4e7aef32.dmp objdir-debug/dist/crashreporter-symbols/ diff --git a/docs/contributing/debugging/debugging_firefox_with_gdb.rst b/docs/contributing/debugging/debugging_firefox_with_gdb.rst new file mode 100644 index 0000000000..8b188edc85 --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_gdb.rst @@ -0,0 +1,501 @@ +Debugging Firefox with GDB +========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This page details how you can more easily debug Firefox and work around +some GDB problems. + +Use GDB 5, or higher. A more recent version of GDB can be obtained from +`sourceware <https://sourceware.org/gdb/>`__ or your Linux distro repo. +If you are running less than 256 MB of RAM, be sure to see `Using gdb on +wimpy computers <https://developer.mozilla.org/en/Using_gdb_on_wimpy_computers>`__. + +Where can I find general gdb documentation? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using GDB is beyond the scope of this document. Documentation is likely +available on your system if you have GDB installed, in the form of +**info,** **man** pages, or the gnome help browser. Additionally, you +can use a graphical front-end to GDB like +`ddd <https://www.gnu.org/software/ddd/>`__ or +`insight <https://sourceware.org/insight/>`__. For more information see +https://sourceware.org/gdb/current/onlinedocs/gdb/ + +How do I run Firefox under gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The preferred method, is using the +:ref:`mach` command-line tool to run the +debugger, which can bypass several optional defaults. Use "mach help +run" to get more details. If inside the source directory, you would use +"./mach". Please note that :ref:`mach is aware of mozconfigs <mach_and_mozconfigs>`. + +.. code:: bash + + $ ./mach run --debug [arguments to pass to firefox] + +If you need to direct arguments to gdb, you can use '--debugger-args' +options via the command line parser, taking care to adhere to shell +splitting rules. For example, if you wanted to run the command 'show +args' when gdb starts, you would use: + +.. code:: bash + + $ ./mach run --debug --debugger-args "-ex 'show args'" + +Alternatively, you can run gdb directly against Firefox. However, you +won't get some of the more useful capabilities this way. For example, +mach sets an environment variable (see below) to stop the JS engine from +generating synthetic segfaults to support the slower script dialoging +mechanism. + +.. code:: + + (gdb) OBJDIR/dist/bin/firefox + +How do I pass arguments in prun? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the arguments in GDB before calling prun. Here's an example on how +to do that: + +.. code:: + + (gdb) set args https://www.mozilla.org + (gdb) prun + +How do I set a breakpoint in a library that hasn't been loaded? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +GDB 6.1 and above has support for "pending breakpoints". This is +controlled by the "``set breakpoint pending``" setting, and is enabled +by default. If a breakpoint cannot be immediately resolved, it will be +re-checked each time a shared library is loaded, by the process being +debugged. If your GDB is older than this, you should upgrade. + +In older versions, there isn't a way to set breakpoints in a library +that has not yet been loaded. See more on `setting a breakpoint when a +component is +loaded <#How_do_I_set_a_breakpoint_when_a_component_is_loaded.3F>`__. If +you have to set a breakpoint you can set a breakpoint in ``_dl_open``. +This function is called when a new library is loaded, when you can +finally set your breakpoint. + +How do I set a breakpoint when a component is loaded? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Firefox Version 57 (and possibly earlier) XPCOM_BREAK_ON_LOAD does +not seem to exist. + +There's a facility in XPCOM which allows you to set an environment +variable to drop into the debugger when loading a certain component. You +have to set ``XPCOM_BREAK_ON_LOAD`` variable before you run Firefox, +setting it to a string containing the names of libraries you want to +load. For example, if you wish to stop when a library named ``raptor`` +or ``necko`` is loaded, you set the variable to ``raptor:necko``. Here's +an example: + +.. code:: + + (gdb) set env XPCOM_BREAK_ON_LOAD raptor:necko + (gdb) prun + +Why can't I set a breakpoint? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You probably can't set a breakpoint because its library hasn't been +loaded. Most Firefox functionality is in libraries loaded mid-way +through the ``main()``\ function. If you break on ``main(),``\ and step +through until the libraries are loaded, with a call to +``InitCOMGlue()``, you should be able to set breakpoints on many more +symbols, source files, and continue running. + +.. code:: + + (gdb) break main + (gdb) run + Breakpoint 1, main(argc=4, argv=0x7fffffffde98, envp=0x7ffffffffdec0) ..... + 256 { + (gdb) next + ... + 293 nsresult rv = InitXPCOMGlue() + (gdb) next + +If you still can't set the breakpoints, you need to confirm the library +has loaded. You can't proceed until the library loads. See more on +`loading shared libraries <#How_do_I_load_shared_libraries.3F>`__. If +you wish to break as soon as the library is loaded, see the section on +`breaking when a component is +loaded <#How_do_I_set_a_breakpoint_when_a_component_is_loaded.3F>`__ and +`breaking on a library +load <#How_do_I_set_a_breakpoint_when_a_component_is_loaded.3F>`__. + +How do I display PRUnichar's? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One suggestion is this: + +.. code:: + + (gdb) print ((PRUnichar*)uri.mBuffer)[0]@16 + $47 = {114, 100, 102, 58, 110, 117, 108, 108, 0, 0, 8, 0, 0, 0, 37432, + 16514} + + + +.. code:: + + (gdb) print aURI + $1 = (const PRUnichar *) 0x855e6e0 + (gdb) x/32ch aURI + 0x855e6e0: 104 'h' 116 't' 116 't' 112 'p' 58 ':' 47 '/' 47 '/' 119 'w' + 0x855e6f0: 119 'w' 119 'w' 46 '.' 109 'm' 111 'o' 122 'z' 105 'i' 108 'l' + 0x855e700: 108 'l' 97 'a' 46 '.' 111 'o' 114 'r' 103 'g' 47 '/' 115 's' + 0x855e710: 116 't' 97 'a' 114 'r' 116 't' 47 '/' 0 '\0' 25 '\031' 0 '\0' + (gdb) + +- Define helper functions in your .gdbinit + +.. code:: + + # Define a "pu" command to display PRUnichar * strings (100 chars max) + # Also allows an optional argument for how many chars to print as long as + # it's less than 100. + def pu + set $uni = $arg0 + if $argc == 2 + set $limit = $arg1 + if $limit > 100 + set $limit = 100 + end + else + set $limit = 100 + end + # scratch array with space for 100 chars plus null terminator. Make + # sure to not use ' ' as the char so this copy/pastes well. + set $scratch = "____________________________________________________________________________________________________" + set $i = 0 + set $scratch_idx = 0 + while (*$uni && $i++ < $limit) + if (*$uni < 0x80) + set $scratch[$scratch_idx++] = *(char*)$uni++ + else + if ($scratch_idx > 0) + set $scratch[$scratch_idx] = '\0' + print $scratch + set $scratch_idx = 0 + end + print /x *(short*)$uni++ + end + end + if ($scratch_idx > 0) + set $scratch[$scratch_idx] = '\0' + print $scratch + end + end + + # Define a "ps" command to display subclasses of nsAC?String. Note that + # this assumes strings as of Gecko 1.9 (well, and probably a few + # releases before that as well); going back far enough will get you + # to string classes that this function doesn't work for. + def ps + set $str = $arg0 + if (sizeof(*$str.mData) == 1 && ($str.mFlags & 1) != 0) + print $str.mData + else + pu $str.mData $str.mLength + end + end + +`This is hard. Give me a .gdbinit that already has the +functions. <#This_is_hard._Give_me_a_.gdbinit_that_works.>`__ + +- Define a small helper function "punichar" in #ifdef NS_DEBUG code + somewhere. + +How do I display an nsString? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can call the ToNewCString() method on the nsString. It leaks a +little memory but it shouldn't hurt anything if you only do it a few +times in one gdb session. (via akkana@netscape.com) + +.. code:: + + (gdb) p string.ToNewCString() + +Another method (via bent) is the following (replace ``n`` with: the +returned length of your string): + +.. code:: + + (gdb) p string.Length() + $1 = n + (gdb) x/ns string.BeginReading() + +You can of course use any of the above unichar-printing routines instead +of x/s. + +This is hard. Give me a .gdbinit that works. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See `Boris Zbarsky's +.gdbinit <http://web.mit.edu/bzbarsky/www/gdbinit>`__. It contained +several function definitions including: + +- "prun" to start the browser and disable library loading. +- "pu" which will display a (PRUnichar \*) string. +- "ps" which will display a nsString. + +How do I determine the concrete type of an object pointed to by an interface pointer? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can determine the concrete type of any object pointed to, by an +XPCOM interface pointer, by looking at the mangled name of the symbol +for the object's vtable: + +.. code:: + + (gdb) p aKidFrame + $1 = (nsIFrame *) 0x85058d4 + (gdb) x/wa *(void**)aKidFrame + 0x4210d380 <__vt_14nsRootBoxFrame>: 0x0 + (gdb) p *(nsRootBoxFrame*)aKidFrame + [ all the member variables of aKidFrame ] + +If you're using gcc 3.x, the output is slightly different from the gcc +2.9x output above. Pay particular attention to the vtable symbol, in +this case ``__vt_14nsRootBoxFrame``. You won't get anything useful if +the shared library containing the object is not loaded. See `How do I +load shared libraries? <#How_do_I_load_shared_libraries.3F>`__ and `How +do I see what libraries I already have +loaded? <#How_do_I_see_what_libraries_I_already_have_loaded.3F>`__ + +Or use the gdb command ``set print object on``. + +How can I debug JavaScript from gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have JavaScript Engine code on the stack, you'll probably want a +JS stack in addition to the C++ stack. + +.. code:: + + (gdb) call DumpJSStack() + +See `https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_JavaScript <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_JavaScript>`__ +for more JS debugging tricks. + +How can I debug race conditions and/or how can I make something different happen at NS_ASSERTION time? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +| [submitted by Dan Mosedale] +| As Linux is unable to generate useful core files for multi-threaded + applications, tracking down race-conditions which don't show up under + the debugger can be a bit tricky. Unless you've given the + ``--enable-crash-on-assert`` switch to ``configure``, you can now + change the behavior of ``NS_ASSERTION`` (nsDebug::Break) using the + ``XPCOM_DEBUG_BREAK`` environment variable. + +How do I run the debugger in emacs/xemacs? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Emacs and XEmacs contain modes for doing visual debugging. However, you +might want to set up environment variables, specifying the loading of +symbols and components. The easiest way to set up these is to use the +``run-mozilla.sh`` script, located in the dist/bin directory of your +build. This script sets up the environment to run the editor, shell, +debugger, or defining a preferred setup and running any commands you +wish. For example: + +.. code:: bash + + $ ./run-mozilla.sh /bin/bash + MOZILLA_FIVE_HOME=/home/USER/src/mozilla/build/dist/bin + LD_LIBRARY_PATH=/home/USER/src/mozilla/build/dist/bin + LIBRARY_PATH=/home/USER/src/mozilla/build/dist/bin + SHLIB_PATH=/home/USER/src/mozilla/build/dist/bin + LIBPATH=/home/USER/src/mozilla/build/dist/bin + ADDON_PATH=/home/USER/src/mozilla/build/dist/bin + MOZ_PROGRAM=/bin/bash + MOZ_TOOLKIT= + moz_debug=0 + moz_debugger= + +GDB 5 used to work for me, but now Firefox won't start. What can I do? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A recent threading change (see `bug +57051 <https://bugzilla.mozilla.org/show_bug.cgi?id=57051>`__ for +details) caused a problem on some systems. Firefox would get part-way +through its initialization, then stop before showing a window. A recent +change to gdb has fixed this. Download and build `the latest version of +Insight <https://sources.redhat.com/insight/>`__, or if you don't want a +GUI, `the latest version of gdb <https://sources.redhat.com/gdb/>`__. + +"run" or "prun" in GDB fails with "error in loading shared libraries." +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Running mozilla-bin inside GDB fails with an error message like: + +.. code:: + + Starting program: + /u/dmose/s/mozilla/mozilla-all/mozilla/dist/bin/./mozilla-bin + /u/dmose/s/mozilla/mozilla-all/mozilla/dist/bin/./mozilla-bin: error + in loading shared libraries: libraptorgfx.so: cannot open shared + object file: No such file or directory + +Your LD_LIBRARY_PATH is probably being reset by your .cshrc or .profile. +From the GDB manual: + +*\*Warning:\* GDB runs your program using the shell indicated by your +'SHELL' environment variable if it exists (or '/bin/sh' if not). If your +'SHELL' variable names a shell that runs an initialization file -- such +as '.cshrc' for C-shell, or '.bashrc' for BASH--any variables you set in +that file affect your program. You may wish to move the setting of +environment variables to files that are only run when you sign on, such +as '.login' or '.profile'.* + +Debian's GDB doesn't work. What do I do? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Debian's unstable distribution currently uses glibc 2.1 and GDB 4.18. +However, there is no package of GDB for Debian with the appropriate +threads patches that will work with glibc 2.1. I was able to get this to +work by getting the GDB 4.18 RPM from Red Hat's rawhide server and +installing that. It has all of the patches necessary for debugging +threaded software. These fixes are expected to be merged into GDB, which +will fix the problem for Debian Linux. (via `Bruce +Mitchener <mailto:bruce@cybersight.com>`__) + +Firefox is aborting. Where do I set a breakpoint to find out where it is exiting? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On Linux there are two possible symbols that are causing this: +``PR_ASSERT()`` and ``NS_ASSERTION()``. To see where it's asserting you +can stop at two places: + +.. code:: + + (gdb) b abort + (gdb) b exit + +I keep getting a SIGSEGV in JS/JIT code under gdb even though there is no crash when gdb is not attached. How do I fix it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the JS_DISABLE_SLOW_SCRIPT_SIGNALS environment variable (in FF33, +the shorter and easier-to-remember JS_NO_SIGNALS). For an explanation, +read `Jan's blog +post <https://www.jandemooij.nl/blog/2014/02/18/using-segfaults-to-interrupt-jit-code/>`__. + +I keep getting a SIG32 in the debugger. How do I fix it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are getting a SIG32 while trying to debug Firefox you might have +turned off shared library loading before the pthreads library was +loaded. For example, ``set auto-solib-add 0`` in your ``.gdbinit`` file. +In this case, you can either: + +- Remove it and use the method explained in the section about `GDB's + memory + usage <#The_debugger_uses_a_lot_of_memory._How_do_I_fix_it.3F>`__ +- Use ``handle SIG32 noprint`` either in gdb or in your ``.gdbinit`` + file + +Alternatively, the problem might lie in your pthread library. If this +library has its symbols stripped, then GDB can't hook into thread +events, and you end up with SIG32 signals. You can check if your +libpthread is stripped in ``file /lib/libpthread*`` and looking for +``'stripped'.``\ To fix this problem on Gentoo Linux, you can re-emerge +glibc after adding ``"nostrip"`` to your ``FEATURES`` in +``/etc/make.conf``. + +How do I get useful stack traces inside system libraries? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many Linux distributions provide separate packages with debugging +information for system libraries, such as gdb, Valgrind, profiling +tools, etc., to give useful stack traces via system libraries. + +Fedora +^^^^^^ + +On Fedora, you need to enable the debuginfo repositories, as the +packages are in separate repositories. Enable them permanently, so when +you get updates you also get security updates for these packages. A way +to do this is edit ``/etc/yum.repos.d/fedora.repo`` and +``fedora-updates.repo`` to change the ``enabled=0`` line in the +debuginfo section to ``enabled=1``. This may then flag a conflict when +upgrading to a new distribution version. You would the need to perform +this edit again. + +You can finally install debuginfo packages with yum or other package +management tools. The best way is install the ``yum-utils`` package, and +then use the ``debuginfo-install`` command to install all the debuginfo: + +.. code:: bash + + $ yum install yum-utils + $ debuginfo-install firefox + +This can be done manually using: + +.. code:: bash + + $ yum install GConf2-debuginfo ORBit2-debuginfo atk-debuginfo \ + cairo-debuginfo dbus-debuginfo dbus-glib-debuginfo expat-debuginfo \ + fontconfig-debuginfo freetype-debuginfo gcc-debuginfo glib2-debuginfo \ + glibc-debuginfo gnome-vfs2-debuginfo gtk2-debuginfo gtk2-engines-debuginfo \ + hal-debuginfo libX11-debuginfo libXcursor-debuginfo libXext-debuginfo \ + libXfixes-debuginfo libXft-debuginfo libXi-debuginfo libXinerama-debuginfo \ + libXrender-debuginfo libbonobo-debuginfo libgnome-debuginfo \ + libselinux-debuginfo pango-debuginfo popt-debuginfo scim-bridge-debuginfo + +Debugging electrolysis (e10s) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``mach run`` and ``mach test`` both accept a ``--disable-e10s`` +argument. Some debuggers can't catch child-process crashes without it. + +You can find some (outdated) information on +https://wiki.mozilla.org/Electrolysis/Debugging. You may also like to +read +https://mikeconley.ca/blog/2014/04/25/electrolysis-debugging-child-processes-of-content-for-make-benefit-glorious-browser-of-firefox +for a more up-to-date blog post. + +To get the child process id use: + +.. code:: + + MOZ_DEBUG_CHILD_PROCESS=1 mach run + +See also +~~~~~~~~~ + +- `Debugging <https://developer.mozilla.org/En/Debugging>`__ +- `Performance tools <https://wiki.mozilla.org/Performance:Tools>`__ +- `Fun with + gdb <https://blog.mozilla.com/sfink/2011/02/22/fun-with-gdb/>`__ by + Steve Fink +- `Archer pretty printers for + SpiderMonkey <https://hg.mozilla.org/users/jblandy_mozilla.com/archer-mozilla>`__ + (`blog + post <https://itcouldbesomuchbetter.wordpress.com/2010/12/20/debugging-spidermonkey-with-archer-2/>`__) +- `More pretty + printers <https://hg.mozilla.org/users/josh_joshmatthews.net/archer-mozilla/>`__ + for Gecko internals (`blog + post <https://www.joshmatthews.net/blog/2011/06/nscomptr-has-never-been-so-pretty/>`__) + +.. container:: originaldocinfo + + .. rubric:: Original Document Information + :name: Original_Document_Information + + - `History <http://bonsai-www.mozilla.org/cvslog.cgi?file=mozilla-org/html/unix/debugging-faq.html&rev=&root=/www/>`__ + - Copyright Information: © 1998-2008 by individual mozilla.org + contributors; content available under a `Creative Commons + license <https://www.mozilla.org/foundation/licensing/website-content.html>`__ diff --git a/docs/contributing/debugging/debugging_firefox_with_lldb.rst b/docs/contributing/debugging/debugging_firefox_with_lldb.rst new file mode 100644 index 0000000000..99ae5a60c0 --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_lldb.rst @@ -0,0 +1,80 @@ +Debugging Firefox with LLDB +=========================== + +See http://lldb.llvm.org/index.html. + +Mozilla-specific lldb settings +------------------------------ + +There's an +``.lldbinit`` `file <https://searchfox.org/mozilla-central/source/.lldbinit>`_ +in the Mozilla source tree, which applies recommended settings and +includes a few type summaries and Mozilla-specific debugging commands +via the lldbutils module (see +`python/lldbutils/README.txt <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__). +For information about available features see the links above and the `Using +LLDB to debug Gecko <http://mcc.id.au/blog/2014/01/lldb-gecko>`__ blog +post. + +The in-tree ``.lldbinit`` should be loaded automatically in most cases +when running lldb from the command line (e.g. using +:ref:`mach`), but **not** +when using Xcode. See :ref:`Debugging on macOS` for information on setting up +Xcode. + +.. warning:: + + LLDB warning: Xcode 5 only comes with lldb (gdb is gone). The + introduction and use of UNIFIED_SOURCES in the source starting around + November 2013 has broken the default LLDB configuration so that it + will not manage to resolve breakpoints in files that are build using + UNIFIED_SOURCES (the breakpoints will be listed as "pending", and + lldb will not stop at them). To fix this add the following to your + $HOME/.lldbinit file: + + .. code:: + + # Mozilla's use of UNIFIED_SOURCES to include multiple source files into a + # single compiled file breaks lldb breakpoint setting. This works around that. + # See http://lldb.llvm.org/troubleshooting.html for more. + settings set target.inline-breakpoint-strategy always + + Restart Xcode/lldb and restart your debugging session. If that still + doesn't fix things then try closing Xcode/lldb, doing a clobber + build, reopening Xcode/lldb, and restarting your debugging session. + +Starting a debugging session +---------------------------- + +Attaching to an existing process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can attach to Firefox with following command: + +.. code:: + + (lldb) process attach --name firefox + +Some versions of lldb causes crashes after attaching to Firefox. + +Running a new process +~~~~~~~~~~~~~~~~~~~~~ + +To start Firefox under the debugger, run ``lldb`` followed by "--", +followed by the command line you'd like to run, like this: + +.. code:: bash + + $ lldb -- obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox-bin -no-remote -profile /path/to/profile + +Then set breakpoints you need and start the process: + +.. code:: + + (lldb) breakpoint set --name nsInProcessTabChildGlobal::InitTabChildGlobal + Breakpoint created: 1: name = 'nsInProcessTabChildGlobal::InitTabChildGlobal', locations = 0 (pending) + WARNING: Unable to resolve breakpoint to any actual locations. + + (lldb) r + Process 7602 launched: '/.../obj-ff-opt/dist/Nightly.app/Contents/MacOS/firefox-bin' (x86_64) + 1 location added to breakpoint 1 diff --git a/docs/contributing/debugging/debugging_firefox_with_rr.rst b/docs/contributing/debugging/debugging_firefox_with_rr.rst new file mode 100644 index 0000000000..ef68d9c6ea --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_rr.rst @@ -0,0 +1,98 @@ +Debugging Firefox with rr +========================= + +This page is intended to help Firefox/Gecko developers get started using rr to debug Firefox. + +Prerequisites +------------- + +You must have Linux installed with a recent kernel. If you're not running Linux already, an option is set up a virtual machine in which to record Firefox. Be forewarned though that + + * rr requires a VM hypervisor that virtualizes CPU performance counters. VMWare Workstation supports that. + * there's a 20% or so performance hit from running in a VM; generally speaking recorder overhead increases from ~1.2x to ~1.4x. (It's a feather in the cap of the hypervisor authors that the hit is that small, though!) + * Some features (reverse execution) `may not work well in VMWare <https://robert.ocallahan.org/2014/09/vmware-cpuid-conditional-branch.html>`__ due to a VMWare optimization that can be disabled `this way <http://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html>`__. + +When using VSCode, consider adding the `Midas <https://github.com/farre/midas>`__ addon to debug rr traces from the editor. You can use Midas to install / build rr and to configure the :code:`auto-load safe path` via the setup commands. + +Ensure that you've `installed <http://rr-project.org/>`__ or `built <https://github.com/mozilla/rr/wiki/Building-And-Installing>`__ rr and have `used it successfully <https://github.com/mozilla/rr/wiki/Usage>`__. Check that you have the latest version. + +You likely need to configure your :code:`auto-load safe path` for rr / gdb to work correctly. If you are on Linux and your builds are in ~/moz, then add the following to ~/.gdbinit + +.. code:: + + add-auto-load-safe-path ~/moz + +Firefox developers are strongly encouraged to build rr from source. If your Firefox patch triggers a bug in rr, rr developers will fix that bug with high priority. You might be able to pull a fix within a few hours or days instead of waiting for the next release. + +Recording Firefox +----------------- + + +To record Firefox running normally, simply launch it under rr as you would if running it under valgrind or gdb + +.. code:: bash + + $ rr $ff-objdir/dist/bin/firefox ... + +or use mach + +.. code:: bash + + $ ./mach run --debugger=rr + +This will save a trace to your working directory as described in the `usage instructions <https://github.com/mozilla/rr/wiki/Usage>`__. Please refer to `those instructions <https://github.com/mozilla/rr/wiki/Usage>`__ for details on how to debug the recording, which isn't covered in this document. + +Sandboxing reduces recording performance because of the SIGSYS signals and extra syscall. Disabling it can help. + +The background hang monitor might also be making things worse by causing a lot of extra syscalls. It can be disabled by setting +toolkit.content-background-hang-monitor.disabled=true. + +SIGSYS +------ + +When recording and replaying Firefox running with the Linux sandbox, you will get SIGSYS signals frequently. This is expected behavior caused by the sandbox. In gdb, use handle SIGSYS noprint nostop to suppress the signals. + +Recording test suites +--------------------- + +You can use the test runners' --debugger feature to punch rr down through the layers of python script to where Firefox is launched. This is used in the same way you would use --debugger to run valgrind or gdb, for example: + +.. code:: bash + + $ ./mach mochitest --debugger=rr ... + +The test harnesses disable the slow-script timeout when the --debugger argument is passed. That's usually sensible, because you don't want those warnings being generated while Firefox is stopped in gdb. However, this has been `observed to change Gecko behavior <https://bugzilla.mozilla.org/show_bug.cgi?id=986673>`__. rr doesn't need to have the slow-script timeout disabled, so to avoid those kinds of pitfalls, pass the --slowscript argument to the test harness. + +To run rr in chaos mode: + +.. code:: bash + + $ ./mach mochitest --debugger=rr --debugger-args="record --chaos" + +You can also run the entire test harness in rr: + +.. code:: bash + + $ rr ./mach mochitest ... + +The trace will contain many processes, so to debug the correct one, you'll want to use rr ps or rr replay -p firefox etc. + +Working with multiple processes +------------------------------ + +rr should work out of the box with multi-process Firefox. Once you have a recording you can use rr ps to show all the process that were recorded and rr replay -p <pid> to attach to a particular process. + +If you want to debug a particular part of code, you can use the :code:`MOZ_DBG` macro and :code:`getpid()` function to write the process id to stderr. `MOZ_LOG <https://firefox-source-docs.mozilla.org/xpcom/logging.html>`__ will include the pid in log messages by default. + +You can combine that with the -M and -g flags to jump to a particular point in a particular process's lifetime. + +Get help! +--------- + +If you encounter a problem with rr, please `file an issue <https://github.com/mozilla/rr/issues>`__. Firefox bugs are high priority, so usually your issue can be fixed very quickly. + +If you want to chat with rr developers, because you need more help or want to contribute or want to complain, we hang out in the `#rr channel <https://chat.mozilla.org/#/room/#rr:mozilla.org>`__. There is also a channel for `#midas <https://chat.mozilla.org/#/room/#midas:mozilla.org>`__. + +You also may find `these debugging protips <https://github.com/mozilla/rr/wiki/Debugging-protips>`__ helpful, though many are for rr developers, not users. + +Happy debugging! diff --git a/docs/contributing/debugging/debugging_firefox_with_valgrind.rst b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst new file mode 100644 index 0000000000..33aab3638a --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst @@ -0,0 +1,177 @@ +Debugging Firefox with Valgrind +=============================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This page describes how to use Valgrind (specifically, its Memcheck +tool) to find memory errors. + +Supported platforms +------------------- + +Valgrind runs desktop Firefox fine on Linux, especially on x86 and +x86-64. Firefox for Android and Firefox OS on ARMv7 should also run, +though perhaps not as smoothly. The other architectures supported by +Valgrind on Linux (AARCH64, PPC{32,64}, MIPS{32,64}, S390X) should also +work, in theory. + +MacOS X 10.10 (Yosemite), 64-bit only, works, although it can be a bit +of a rough ride. + +- Expect lower performance and a somewhat higher false positive error + rate than on Linux. +- Valgrind's handling of malloc zones on Yosemite is imperfect. Regard + leak reports with caution. +- Valgrind has been known to cause kernel panics, for unknown reasons. + +Where to get Valgrind +--------------------- + +Linux: Download `Valgrind <https://valgrind.org/>`__ directly, or use +your distribution's package manager (if it has a recent enough version). + +MacOSX: `Get Valgrind trunk from +SVN <https://valgrind.org/downloads/repository.html>`__ and build it. +Don't use 3.10.x or any other tarball. + +Make sure you have Valgrind 3.14 or later, version 3.16.1 is known to work, +3.13.0 did not. Newer versions tend to have better compatibility with both +Firefox's JITs and newer toolchain components (compiler, libc and linker +versions). + +Basics +------ + +Build +~~~~~ + +Build Firefox with the following options, which maximize speed and +accuracy. + +.. code:: + + ac_add_options --disable-jemalloc + ac_add_options --disable-strip + ac_add_options --enable-valgrind + ac_add_options --enable-optimize="-g -O2" + ac_add_options --disable-sandbox + +Run +~~~ + +Note that programs run *much* more slowly under Valgrind than they do +natively. Slow-downs of 20x or 30x aren't unexpected, and it's slower on +Mac than on Linux. Don't try this on an underpowered machine. + +Linux +^^^^^ + +On Linux, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --read-inline-info=yes + +The ``--smc-check`` and ``--vex-iropt-register-updates`` options are +necessary to avoid crashes in JIT-generated code. + +The ``--show-mismatched-frees`` option is necessary due to inconsistent +inlining of ``new`` and ``delete`` -- i.e. one gets inlined but the +other doesn't -- which lead to false-positive mismatched-free errors. + +The ``--read-inline-info`` option improves stack trace readability in +the presence of inlining. + +Also, run with the following environment variable set. + +.. code:: + + G_SLICE=always-malloc + +This is necessary to get the Gnome system libraries to use plain +``malloc`` instead of pool allocators. + +Mac +^^^ + +On Mac, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --dsymutil=yes + +The ``--dsymutil`` option ensures line number information is present in +stack traces. + +Advanced usage +-------------- + +Shared suppression files +~~~~~~~~~~~~~~~~~~~~~~~~ + +`/build/valgrind/ <https://searchfox.org/mozilla-central/source/build/valgrind/>`__ +contains the suppression files used by the periodic Valgrind jobs on +Treeherder. Some of these files are platform-specific. + +Running mochitests under Valgrind? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To run a mochitest under Valgrind, use the following command. + +.. code:: bash + + $ ./mach mochitest-plain --debugger="valgrind" --debugger-args="$VALGRIND_OPTIONS" relative/path/to/tests + +Where ``$VALGRIND_OPTIONS`` are the options described +:ref:`above <Debugging Firefox With Valgrind>`. You might also +need ``--trace-children=yes`` to trace into child processes. + +As of December 2014 it is possible to do a complete run of +mochitests-plain on Valgrind in about 8 CPU hours on a Core i4910 +(Haswell) machine. Maximum process size is 5.4G, of which about 80% is +in memory. Runs of small subsets of mochitests take far less memory. + +Bits and pieces +~~~~~~~~~~~~~~~ + +For un-released Linux distros (Fedora Rawhide, etc.) you'll need to use +a version of Valgrind trunk build, because fixes for the latest gcc and +glibc versions appear there first. Without them you'll be flooded with +false errors from Memcheck, and have debuginfo reading problems. + +On Linux, code compiled by LLVM at high optimisation levels can cause +Memcheck to report false uninitialised value errors. See +`here <https://bugs.kde.org/show_bug.cgi?id=242137#c3>`__ for an easy +workaround. On Mac, Valgrind has this workaround built in. + +You can make stack traces easier to read by asking for source file names +to be given relative to the root of your source tree. Do this by using +``--fullpath-after=`` to specify the rightmost part of the absolute path +that you don't want to see. For example, if your source tree is rooted +at ``/home/sewardj/MC-20-12-2014``, use ``--fullpath-after=2014/`` to +get path names relative to the source directory. + +The ``--track-origins=yes`` slows down Valgrind greatly, so don't use it +unless you are hunting down a specific uninitialised value error. But if +you are hunting down such an error, it's extremely helpful and worth +waiting for. + +Additional help +--------------- + +The `Valgrind Quick Start +Guide <http://www.valgrind.org/docs/manual/quick-start.html>`__ is short +and worth reading. The `User +Manual <https://valgrind.org/docs/manual/manual.html>`__ is also useful. + +If Valgrind asserts, crashes, doesn't do what you expect, or otherwise +acts up, first of all read this page and make sure you have both Firefox +and Valgrind correctly configured. If that's all OK, try using the +`Valgrind trunk from +SVN <http://www.valgrind.org/downloads/repository.html>`__. Oftentimes +bugs are fixed in the trunk before most users fall across them. If that +doesn't help, consider `filing a bug +report <http://www.valgrind.org/support/bug_reports.html>`__, and/or +mailing Julian Seward or Nick Nethercote. diff --git a/docs/contributing/debugging/debugging_on_macos.rst b/docs/contributing/debugging/debugging_on_macos.rst new file mode 100644 index 0000000000..ef42c162f9 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_macos.rst @@ -0,0 +1,359 @@ +Debugging On macOS +================== + +This document explains how to debug Gecko-based applications such as +Firefox, Thunderbird, and SeaMonkey on macOS using Xcode. If you want to +debug from the terminal see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. For specific +information on a way to debug hangs, see :ref:`Debugging a hang on macOS <Debugging A Hang On macOS>`. + +Creating a debuggable build +--------------------------- + +First, you need to build the application you're going to debug using +this in your .mozconfig + +.. code:: + + ac_add_options --disable-optimize + ac_add_options --enable-debug-symbols + +you can also add this flag if you want assertions etc. compiled in + +.. code:: + + ac_add_options --enable-debug + +See :ref:`Building Firefox for macOS <Building Firefox On MacOS>` +if you need help creating your own build. + +Debugging Firefox on macOS 10.14+ +--------------------------------- + +macOS 10.14 introduced Notarization and Hardened Runtime features for +improved application security. macOS 10.15 went further, requiring +applications to be Notarized with Hardened Runtime enabled in order to +launch (ignoring workarounds). When run on earlier macOS versions, +Notarization and Hardened Runtime settings have no effect. + +Official Builds +~~~~~~~~~~~~~~~ + +At this time, official builds of Firefox 69 and later are Notarized. +**As a result, it is not possible to attach a debugger to these official +Firefox releases on macOS 10.14+ without disabling System Integrity +Protection (SIP).** This is due to Notarization requiring Hardened +Runtime to be enabled with the ``com.apple.security.get-task-allow`` +entitlement disallowed. **Rather than disabling SIP (which has security +implications), it is recommended to debug with try builds or local +builds. The differences are explained below.** + +try Server Builds +~~~~~~~~~~~~~~~~~ + +In most cases, developers needing to debug a build as close as possible +to the production environment should use a :ref:`try +build <Pushing to Try>`. These +builds enable Hardened Runtime and only differ from production builds in +that they are not Notarized which should not otherwise affect +functionality, (other than the ability to easily launch the browser on +macOS 10.15+ -- see quarantine note below). At this time, developers can +obtain a Hardened Runtime build with the +``com.apple.security.get-task-allow`` entitlement allowed by submitting +a try build and downloading the dmg generated by the "Rpk" shippable +build job. A debugger can be attached to Firefox processes of these +builds. try builds use the ``developer.entitlements.xml`` file from the +source tree while production builds use ``production.entitlements.xml``. +**On macOS 10.15+, downloaded try builds will not launch by default +because Notarization is required. To workaround this problem, remove the +quarantine extended attribute from the downloaded Nightly:** + + ``$ xattr -r -d com.apple.quarantine /Path/to/Nightly.app`` + +Local Builds +~~~~~~~~~~~~ + +Local builds of mozilla-central do not enable Hardened Runtime and hence +do not have debugging restrictions. As a result, some functionality will +be permitted on local builds, but blocked on production builds which +have Hardened Runtime enabled. `Bug +1522409 <https://bugzilla.mozilla.org/show_bug.cgi?id=1522409>`__ was +filed to automate codesigning local builds to enable Hardened Runtime by +default and eliminate this discrepancy. + +To obtain a Hardened Runtime build without using try infrastructure, a +developer can manually codesign builds using the macOS ``codesign(1)`` +command with the ``developer.entitlements.xml`` file from the tree. This +requires creating a codesigning identity. + +Disabling System Integrity Protection (SIP) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If debugging a production build is required, follow Apple's documented +steps for disabling System Integrity Protection (SIP). Note that +disabling SIP bypasses Hardened Runtime restrictions which can mask some +bugs that only occur with Hardened Runtime so it is recommended to test +fixes with SIP enabled. **Disabling SIP has system security implications +that should be understood before taking this step.** + +Creating an Xcode project +------------------------- + +If you try to create a new Xcode project in an existing directory +then Xcode will delete its existing contents (Xcode will warn you +beforehand). To work around that, the steps below have you initialize +the project outside the Mozilla source tree, close the project, copy +the .xcodeproj project "file" into the source tree, and then reopen +the project to finish setting it up. + +Note also that since Xcode 7.3.1 it doesn't seem to be possible to +have the Xcode project live outside the source tree. If you try to do +that then Xcode will simply **copy** the source files under the +project directory rather than link to them which breaks debugging and the +possibility to modify-rebuild-relaunch from inside Xcode. + +These steps were last updated for Xcode 10.3: + +#. Open Xcode, and create a new Project with File > New Project. Select + the "Cross-platform" tab then under the "Other" template group select + the "Empty" project type. the click Next. Name the project and click + Next. Create/select a temporary directory to contain the project and + then click Create. +#. Before going any further, close the project (File > Close Project) + and open Finder. Find the \*.xcodejproj directory in the temporary + directory, move it into your Mozilla source tree, and then + double-click on it to reopen it. +#. In the left-hand pane in Xcode you should see a tree item where the + root item has the project name. If the temporary directory that you + originally created the Xcode project in is under that, right click it + and delete it. Now, right click on the root item, select 'Add files + to "<project-name>"', select all the files and directories in your + source directory, untick "Copy items if needed", then click Add. + (These will then be progressively added under the root item + <project-name> in the left-hand pane. Note that subdirectories may + initially appear to be empty, but they too will progressively be + populated as Xcode processes the sourse files. Once done, you should + be able to open any file quickly by hitting Cmd-Shift-O and typing in + the name of a file.) +#. In the Product menu, select Scheme > New Scheme and name your scheme + (for example, "Debug"). After you click OK, Xcode should open the + settings window for the new scheme. (If not, then open its settings + from the Product > Edit Scheme menu.) +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on "None" and + selecting "Other...". A new dialog titled "Choose an executable to + launch" will pop up. Browse to the ``.app`` file that you want to + debug (``Firefox.app``, ``Nightly``\ ``Debug.app`` etc). The ``.app`` + file is typically found inside the ``dist`` folder in your build + directory. +#. If you are debugging Firefox, Thunderbird, or some other application + that supports multiple profiles, using a separate profile for + debugging purposes is recommended. See "Having a profile for + debugging purposes" below. Select the "Arguments" tab in the scheme + editor, and click the '+' below the "Arguments passed on launch" + field. Add "-P *profilename*", where *profilename* is the name of a + profile you created previously. Repeat that to also add the argument + "-no-remote". +#. Also in the "Arguments" panel, you may want to add an environment + variable MOZ_DEBUG_CHILD_PROCESS set to the value 1 to help with + debugging e10s. +#. Select "Build" from the left of the scheme editor window, and check + that there is nothing listed under Targets (otherwise it may cause + problems when you try to run the executable for debugging since you + will get build errors). +#. Click "Close" to close the scheme editor. + +At this point you can run the application from Xcode, and when you pause +or hit breakpoints it should show open the correct source file at the +correct line. + +Setting up lldb +--------------- + +``lldb`` is the debugger Xcode provides/uses. + +.. warning:: + + One important issue that the Mozilla .lldbinit file fixes is that by + default some breakpoints will be listed as "pending", and Xcode will + not stop at them. If you don't include the Mozilla's .lldbinit, you + must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Firefox with + lldb <Debugging Firefox with LLDB>`. + +The +`.lldbinit <http://searchfox.org/mozilla-central/source/.lldbinit>`__ +file in the source tree imports many useful `Mozilla specific lldb +settings, commands and +formatters <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__ +into ``lldb``, but you may need to take one of the following steps to +make sure this file is used. + +If you are using ``lldb`` on the command line (independently of Xcode) +and you will always run it from either the top source directory, the +object directory or else the dist/bin subdirectory of the object +directory, then adding the following setting to your ``$HOME/.lldbinit`` +is sufficient: + +:: + + settings set target.load-cwd-lldbinit true + +*However*, if you will run lldb from a different directory, or if you +will be running it indirectly by debugging in Xcode (Xcode always runs +lldb from "/"), then this setting will not help you. Instead, add the +following to your ``$HOME/.lldbinit``: + +:: + + # This automatically sources the Mozilla project's .lldbinit as soon as lldb + # starts or attaches to a Mozilla app (that's in an object directory). + # + # This is mainly a workaround for Xcode not providing a way to specify that + # lldb should be run from a given directory. (Xcode always runs lldb from "/", + # regardless of what directory Xcode was started from, and regardless of the + # value of the "Custom working directory" field in the Scheme's Run options. + # Therefore setting `settings set target.load-cwd-lldbinit true` can't help us + # without Xcode providing that functionality.) + # + # The following works by setting a one-shot breakpoint to break on a function + # that we know will both run early (which we want when we start first start the + # app) and run frequently (which we want so that it will trigger ASAP if we + # attach to an already running app). The breakpoint runs some commands to + # figure out the object directory path from the attached target and then + # sources the .lldbinit from there. + # + # NOTE: This scripts actions take a few seconds to complete, so the custom + # formatters, commands etc. that are added may not be immediately available. + # + breakpoint set --name nsThread::ProcessNextEvent --thread-index 1 --auto-continue true --one-shot true + breakpoint command add -s python + # This script that we run does not work if we try to use the global 'lldb' + # object, since it is out of date at the time that the script runs (for + # example, `lldb.target.executable.fullpath` is empty). Therefore we must + # get the following objects from the 'frame' object. + target = frame.GetThread().GetProcess().GetTarget() + debugger = target.GetDebugger() + + # Delete our breakpoint (not actually necessary with `--one-shot true`): + target.BreakpointDelete(bp_loc.GetBreakpoint().GetID()) + + # For completeness, find and delete the dummy breakpoint (the breakpoint + # lldb creates when it can't initially find the method to set the + # breakpoint on): + # BUG WORKAROUND! GetID() on the *dummy* breakpoint appears to be returning + # the breakpoint index instead of its ID. We have to add 1 to correct for + # that! :-( + dummy_bp_list = lldb.SBBreakpointList(target) + debugger.GetDummyTarget().FindBreakpointsByName("nsThread::ProcessNextEvent", dummy_bp_list) + dummy_bp_id = dummy_bp_list.GetBreakpointAtIndex(0).GetID() + 1 + debugger.GetDummyTarget().BreakpointDelete(dummy_bp_id) + + # "source" the Mozilla project .lldbinit: + os.chdir(target.executable.fullpath.split("/dist/")[0]) + debugger.HandleCommand("command source -s true " + os.path.join(os.getcwd(), ".lldbinit")) + DONE + +see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. for more information. + +Having a profile for debugging purposes +--------------------------------------- + +It is recommended to create a separate profile to debug with, whatever +your task, so that you don't lose precious data like Bookmarks, saved +passwords, etc. So that you're not bothered with the profile manager +every time you start to debug, expand the "Executables" branch of the +"Groups & Files" list and double click on the Executable you added for +Mozilla. Click the plus icon under the "Arguments" list and type "-P +<profile name>" (e.g. "-P MozillaDebug"). Close the window when you're +done. + +Running a debug session +----------------------- + +Make sure breakpoints are active (which implies running under the +debugger) by opening the Product menu and selecting "Debug / Activate +Breakpoints" (also shown by the "Breakpoints" button in the top right +section of the main window). Then click the "Run" button or select "Run" +from the Product menu. + +Setting breakpoints +~~~~~~~~~~~~~~~~~~~ + +Setting a breakpoint is easy. Just open the source file you want to +debug in Xcode, and click in the margin to the left of the line of code +where you want to break. + +During the debugging session, each time that line is executed, the +debugger will break there, and you will be able to debug it. + +.. warning:: + + Note that with the default configuration, some breakpoints will be + listed as "pending", and Xcode will not stop at them. If you don't + include the Mozilla's .lldbinit, you must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with + lldb <Debugging Firefox with LLDB>`. + +Using Firefox-specific lldb commands +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you included the .lldbinit when `Setting up +lldb <#setting-up-lldb>`__, you can use Mozilla-specific lldb commands +in the console, located in the Debug area of Xcode. For example, type +``js`` to see the JavaScript stack. For more information, see :ref:`Debugging +Mozilla with lldb <Debugging Firefox with LLDB>`. + +Debugging e10s child processes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using Xcode to debug child processes created by an e10s-enabled browser +is a little trickier than debugging a single-process browser, but it can +be done. These directions were written using Xcode 6.3.1 + +#. Complete all the steps above under "Creating the Project" +#. From the "Product" menu, ensure the scheme you created is selected + under "Scheme", then choose "Scheme > Edit Scheme" +#. In the resulting popup, click "Duplicate Scheme" +#. Give the resulting scheme a more descriptive name than "Copy of + Scheme" +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on the + "Executable" drop-down, and selecting the ``plugin-container.app`` + that is inside the app bundle of the copy of Firefox you want to + debug. +#. On the same tab, under "Launch" select "Wait for executable to be + launched" +#. On the "Arguments" tab, remove all arguments passed on launch. + +Now you're ready to start debugging: + +#. From the "Product" menu, ensure the scheme you created above is + selected under "Scheme" +#. Click the "Run" button. The information area at the top of the window + will show "Waiting for plugin-container to launch" +#. From a command line, run your build of Firefox. When that launches a + child process (for example, when you start to load a webpage), Xcode + will notice and attach to that child process. You can then debug the + child process like you would any other process. +#. When you are done debugging, click the "Stop" button and quit the + instance of Firefox that you were debugging in the normal way. + +For some help on using lldb see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. + +Other resources +--------------- + +Apple has an extensive list of `debugging tips and +techniques <https://developer.apple.com/library/mac/#technotes/tn2124/_index.html>`__. + +Questions? Problems? +~~~~~~~~~~~~~~~~~~~~ + +Try asking in our Element channels +`#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ or +`#macdev <https://chat.mozilla.org/#/room/#macdev:mozilla.org>`__. diff --git a/docs/contributing/debugging/debugging_on_windows.rst b/docs/contributing/debugging/debugging_on_windows.rst new file mode 100644 index 0000000000..3213299199 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_windows.rst @@ -0,0 +1,330 @@ +Debugging On Windows +==================== + +This document explains how to debug Gecko based applications such as +Firefox, Thunderbird, and SeaMonkey on Windows using the Visual Studio IDE. + +If VS and your Gecko application hang shortly after you launch the +application under the debugger, see `Problems Loading Debug +Symbols <#problems-loading-debug-symbols>`__. + +Ways to start the debugger +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First of all, it's necessary to install a Visual Studio extension to be +able to follow child processes as they are created. Firefox, in general, +and even in non-e10s mode, does not start the main process directly, it +starts it via a Launcher Process. This means that Visual Studio will +only attach to the first process it finds, and will not hit any +break-point (and even notifies you that it cannot find their location). +`Microsoft Child Process Debugging Power +Tool <https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool>`__ +allows automatically attaching to child processes, such as Web Content +process, GPU process, etc. Enable it by going its configuration menu in +"Debug > Other debugging targets > Child process debugging settings", +and ticking the box. + +If you have followed the steps in :ref:`Building Firefox for +Windows <Building Firefox On Windows>` +and have a local debug build, you can **execute this command from same command line.** + +.. code:: + + ./mach run --debug + +It would open Visual Studio with Firefox's +run options configured. You can **click "Start" button** to run Firefox +then, already attached in the debugger. + +Alternatively, if you have generated the Visual Studio solution, via +``./mach build-backend -b VisualStudio``, opening this solution allows +you to run ``firefox.exe`` directly in the debugger. To make it the +startup project, right click on the project and select ``Set As Startup +Project``. It appears bold when it's the case. Breakpoints are kept +across runs, this can be a good way to debug startup issues. + +**Run the program until you hit an assertion.** You will get a dialog +box asking if you would like to debug. Hit "Cancel". The MSDEV IDE will +launch and load the file where the assertion happened. This will also +create a Visual Studio Mozilla project in the directory of the executable +by default. + +**Attach the debugger to an existing Mozilla process**. In the Visual +Studio, select Debug > Attach to Process. If you want to debug a content +process, you can **hover on the tab** of page you want to debug, which +would show the pid. You can then select the process from dialog opened +from "Attach to Process". You can open ``about:processes`` to see the pid +for all subprocesses, including tabs but also GPU, networking etc. +For more information, see `Attach to Running Processes with the Visual Studio +Debugger <http://msdn.microsoft.com/en-us/library/vstudio/3s68z0b3.aspx>`__. + +**Starting an MSIX installed Firefox with the debugger**. In Visual +Studio, select Debug -> Other Debug Targets -> Debug Installed App Package. +In the dialog, select the installed Firefox package you wish to debug +and click "Start". + +Debugging Release and Nightly Builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the steps to :ref:`use the Mozilla symbol +server <Using The Mozilla Symbol Server>` and :ref:`source +server <Using The Mozilla Source Server>` + +Creating a Visual Studio project for Firefox +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please refer to :ref:`this <Visual Studio Projects>`. + +Changing/setting the executable to debug +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To change or set the executable to debug, go to Project > Properties > +Debugging > Command. (As of Visual Studio 2022.) + +It should show the executable you are debugging. If it is empty or +incorrect, manually add the correct path to the executable. + +Command line parameters and environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To change or set the command line options, go to Project > Properties > +Debugging > Command Arguments. + +Some common options would be the URL of the file you want the browser to +open as soon as it starts, starting the Profile Manager, or selecting a +profile. You can also redirect the console output to a file (by adding +"``> filename.txt``" for example, without the quotes). + +Customizing the debugger's variable value view +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can customize how Visual Studio displays classes in the variable view. +By default VS displays "{...}" and you need to click the small + icon +to expand the members. You can change this behaviour, and make Visual +Studio display whatever data member you want in whatever order, formatted +however you like instead of just "{...}". + +You need to locate a file called "gecko.natvis" under toolkit/library. +The file contains a list of types and how they should be displayed in +the debugger. It is XML and after a little practice you should be well +on your way. + +To understand the file in detail refer to `Create custom views of C++ +objects in the debugger using the Natvis framework +<https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects>`__ + +The file already comes with a number of entries that will make your life +easier, like support for several string types. If you need to add a custom +type, or want to change an existing entry for debugging purposes, you can +easily edit the file. For your convenience it is included in all generated +Visual Studio projects, and if you edit and save it within Visual Studio, it +will pick up the changes immediately. + +Handling multiple processes in Visual Studio +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Turn off "Break all processes when one process breaks" to single step a single +process. + +Turning off "Break all processes when one process breaks" adds "Step Into +Current Process", "Step Over Current Process" and "Step Out Current Process" to +the "Debug" menu. + +To single step a single process with the other processes paused: + +- Turn on "Break all processes when one process breaks" +- Hit a breakpoint which stops all processes +- Turn off "Break all processes when one process breaks" +- Now using "Step Into Current Process" will leave the other processes stopped + and just advance the current one. + +Obtaining ``stdout`` and other ``FILE`` handles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Running the following command in the Command Window in Visual Studio +returns the value of ``stdout``, which can be used with various +debugging methods (such as ``nsGenericElement::List``) that take a +``FILE*`` param: + +.. code:: + + Debug.EvaluateStatement {,,msvcr80d}(&__iob_func()[1]) + +(Alternatively you can evaluate ``{,,msvcr80d}(&__iob_func()[1])`` in +the Immediate window) + +Similarly, you can open a file on the disk using ``fopen``: + +.. code:: + + >Debug.EvaluateStatement {,,msvcr80d}fopen("c:\\123", "w") + 0x10311dc0 { ..snip.. } + >Debug.EvaluateStatement ((nsGenericElement*)0x03f0e710)->List((FILE*)0x10311dc0, 1) + <void> + >Debug.EvaluateStatement {,,msvcr80d}fclose((FILE*)0x10311dc0) + 0x00000000 + +Note that you may not see the debugging output until you flush or close +the file handle. + +Disabling ASSERTIONS +~~~~~~~~~~~~~~~~~~~~ + +There are basically two ways to disable assertions. One requires setting +an environment variable, while the other affects only the currently +running program instance in memory. + +Environment variable +^^^^^^^^^^^^^^^^^^^^ + +There is an environment variable that can disable breaking for +assertions. This is how you would normally set it: + +.. code:: + + set XPCOM_DEBUG_BREAK=warn + +The environment variable takes also other values besides ``warn``, see +``XPCOM_DEBUG_BREAK`` for more details. + +Note that unlike Unix, the default for Windows is not warn, it's to pop +up a dialog. To set the environment variable for Visual Studio, use +Project > Properties > Debugging > Environment and click the little box. +Then use + +.. code:: + + XPCOM_DEBUG_BREAK=warn + +Changing running code +^^^^^^^^^^^^^^^^^^^^^ + +You normally shouldn't need to do this (just quit the application, set +the environment variable described above, and run it again). And this +can be **dangerous** (like **trashing your hard disc and corrupting your +system**). So unless you feel comfortable with this, don't do it. **You +have been warned!** + +It is possible to change the interrupt code in memory (which causes you +to break into debugger) to be a NOP (no operation). + +You do this by running the program in the debugger until you hit an +assertion. You should see some assembly code. One assembly code +instruction reads "int 3". Check the memory address for that line. Now +open memory view. Type/copy/drag the memory address of "int 3" into the +memory view to get it to update on that part of the memory. Change the +value of the memory to "90", close the memory view and hit "F5" to +continue. + +Automatically handling ASSERTIONS without a debugger attached +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When an assertion happens and there is not a debugger attached, a small +helper application +(```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__) +is run. That application can automatically select a response to the "Do +you want to debug" dialog instead of prompting if you configure it, for +more info, see +```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__. + +Debugging optimized builds +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To effectively debug optimized builds, you should enable debugging +information which effectively leaves the debug symbols in optimized code +so you can still set breakpoints etc. Because the code is optimized, +stepping through the code may occasionally provide small surprises when +the debugger jumps over something. + +You need to make sure this configure parameter is set: + +.. code:: + + ac_add_options --enable-debug + +You can also choose to include or exclude specific modules. + +Console debugging +~~~~~~~~~~~~~~~~~ + +When printing to STDOUT from a content process, the console message will +not appear on Windows. One way to view it is simply to disable e10s +(``./mach run --disable-e10s``) but in order to debug with e10s enabled +one can run + +:: + + ./mach run ... 2>&1 | tee + +It may also be necessary to disable the content sandbox +(``MOZ_DISABLE_CONTENT_SANDBOX=1 ./mach run ...``). + +Running two instances of Mozilla simultaneously +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can run two instances of Mozilla (e.g. debug and optimized) +simultaneously by setting the environment variable ``MOZ_NO_REMOTE``: + +.. code:: + + set MOZ_NO_REMOTE=1 + +Or, starting with Firefox 2 and other Gecko 1.8.1-based applications, +you can use the ``-no-remote`` command-line switch instead (implemented +in +`bug 325509 <https://bugzilla.mozilla.org/show_bug.cgi?id=325509>`__). + +You can also specify the profile to use with the ``-P profile_name`` +command-line argument. + +Debugging JavaScript +~~~~~~~~~~~~~~~~~~~~ + +You can use helper functions from +`nsXPConnect.cpp <https://searchfox.org/mozilla-central/source/js/xpconnect/src/nsXPConnect.cpp>`__ +to inspect and modify the state of JavaScript code from the MSVS +debugger. + +For example, to print current JavaScript stack to stdout, evaluate this +in Immediate window: + +.. code:: + + {,,xul}DumpJSStack() + +Visual Studio will show you something in the quick watch window, but +not the stack, you have to look in the OS console for the output. + +Also this magical command only works when you have JS on the VS stack. + +Debugging minidumps +~~~~~~~~~~~~~~~~~~~ + +See :ref:`debugging a minidump <Debugging A Minidump>`. + +Problems post-mortem debugging on Windows 7 SP1 x64? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you attempt to use ``NS_DebugBreak`` etc to perform post-mortem +debugging on a 64bit Windows 7, but as soon as you try and continue +debugging the program crashes with an Access Violation, you may be +hitting a Windows bug relating to AVX support. For more details, +including a work-around see `this blog +post <http://www.os2museum.com/wp/?p=960>`__ or `this social.msdn +thread <http://social.msdn.microsoft.com/Forums/vstudio/en-US/392ca62c-e502-42d9-adbc-b4e22d5da0c3/jit-debugging-32bit-app-crashing-with-access-violation>`__. +(And just in-case those links die, the work-around is to execute + +:: + + bcdedit /set xsavedisable 1 + +from an elevated command-prompt to disable AVX support.) + +Got a tip? +~~~~~~~~~~ + +If you think you know a cool Mozilla debugging trick, feel free to +discuss it with `#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ and +then post it here. + +.. |Screenshot of disabling assertions| image:: https://developer.mozilla.org/@api/deki/files/420/=Win32-debug-nop.png + :class: internal diff --git a/docs/contributing/debugging/img/about-processes.png b/docs/contributing/debugging/img/about-processes.png Binary files differnew file mode 100644 index 0000000000..6f1563bcd1 --- /dev/null +++ b/docs/contributing/debugging/img/about-processes.png diff --git a/docs/contributing/debugging/img/about-support.png b/docs/contributing/debugging/img/about-support.png Binary files differnew file mode 100644 index 0000000000..2b12d1027d --- /dev/null +++ b/docs/contributing/debugging/img/about-support.png diff --git a/docs/contributing/debugging/img/crash-gmp.png b/docs/contributing/debugging/img/crash-gmp.png Binary files differnew file mode 100644 index 0000000000..595e439307 --- /dev/null +++ b/docs/contributing/debugging/img/crash-gmp.png diff --git a/docs/contributing/debugging/img/crash-gpu.png b/docs/contributing/debugging/img/crash-gpu.png Binary files differnew file mode 100644 index 0000000000..9e40b63917 --- /dev/null +++ b/docs/contributing/debugging/img/crash-gpu.png diff --git a/docs/contributing/debugging/img/crash-rdd.png b/docs/contributing/debugging/img/crash-rdd.png Binary files differnew file mode 100644 index 0000000000..5548672d27 --- /dev/null +++ b/docs/contributing/debugging/img/crash-rdd.png diff --git a/docs/contributing/debugging/img/crashlist.jpg b/docs/contributing/debugging/img/crashlist.jpg Binary files differnew file mode 100644 index 0000000000..4cb376d0f2 --- /dev/null +++ b/docs/contributing/debugging/img/crashlist.jpg diff --git a/docs/contributing/debugging/img/crashreporter.png b/docs/contributing/debugging/img/crashreporter.png Binary files differnew file mode 100644 index 0000000000..3137c71c78 --- /dev/null +++ b/docs/contributing/debugging/img/crashreporter.png diff --git a/docs/contributing/debugging/img/process-explorer.png b/docs/contributing/debugging/img/process-explorer.png Binary files differnew file mode 100644 index 0000000000..26e7837a8d --- /dev/null +++ b/docs/contributing/debugging/img/process-explorer.png diff --git a/docs/contributing/debugging/img/sdk-installer.png b/docs/contributing/debugging/img/sdk-installer.png Binary files differnew file mode 100644 index 0000000000..c7dcc1f1a4 --- /dev/null +++ b/docs/contributing/debugging/img/sdk-installer.png diff --git a/docs/contributing/debugging/img/tabcrashed.png b/docs/contributing/debugging/img/tabcrashed.png Binary files differnew file mode 100644 index 0000000000..9982e527ed --- /dev/null +++ b/docs/contributing/debugging/img/tabcrashed.png diff --git a/docs/contributing/debugging/img/windbg-in-startmenu.png b/docs/contributing/debugging/img/windbg-in-startmenu.png Binary files differnew file mode 100644 index 0000000000..0819d4b424 --- /dev/null +++ b/docs/contributing/debugging/img/windbg-in-startmenu.png diff --git a/docs/contributing/debugging/local_symbols.rst b/docs/contributing/debugging/local_symbols.rst new file mode 100644 index 0000000000..875717e1a1 --- /dev/null +++ b/docs/contributing/debugging/local_symbols.rst @@ -0,0 +1,64 @@ +Symbolicating TreeHerder stacks locally +======================================= + +When using tools like the :ref:`Dark Matter Detector (DMD)` or +:ref:`refcount logging<Refcount Tracing and Balancing>` to +investigate issues occurring on TreeHerder that you can't reproduce locally, you +can often end up with unsymbolicated stacks. Fortunately, there is a way to +symbolicate these stacks on your own machine. + +These instructions are for a Linux TreeHerder build for MacOS, so they might +require some modifications for other combinations of platforms. + +Download ``target.tar.bz2`` and ``target.crashreporter-symbols.zip`` from the +Build job. **Note that these files are very large so you'll want to delete +them and the extracted files when you are done.** + +These files each contain a large number of files, so I'd recommend creating +a directory for each of them. Call these ``<TARGET_DIR>`` and ``<SYMB_DIR>``, +and move the prior two files into these two directories, respectively. + +Go to ``<TARGET_DIR>`` and run something like + +.. code-block:: shell + + tar xf target.tar.bz2 + +then go to ``<SYMB_DIR>`` and run something like + +.. code-block:: shell + + unzip target.crashreporter-symbols.zip + +You should be able to delete the two original files now. + +Next we need to ensure that the locations of binaries are rewritten from +where they are on TreeHerder to where we have them locally. We'll do this by +editing ``fix_stacks.py``. This file is located in the ``tools/rb/`` directory of +the Firefox source directory. You need to add these two lines to the function +``fixSymbols``, after ``line_str`` is defined and before it is written to +``fix_stacks.stdin``. I've done this right before the definition of +``is_missing_newline``. + +.. code-block:: python + + line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox", + "<TARGET_DIR>/firefox/firefox-bin") + line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so", + "<TARGET_DIR>/firefox/libxul.so") + +The initial locations should appear verbatim in the stack you are trying to +symbolicate, so double check that they match. Also, ``<TARGET_DIR>`` of course +needs to be replaced with the actual local directories where those files are +located. Note that the ``firefox`` executable is changed to ``firefox-bin``. +I don't know why that is necessary, but only the latter existed for me. + +Finally, we need to make it so that the stack fixer can find the location of +the breakpad symbols we downloaded. If you are running ``fix_stacks.py`` via +``dmd.py`` or directly (in a recent version), you can do this by running with the +environment variable ``BREAKPAD_SYMBOLS_PATH`` set to the ``<SYMB_DIR>`` from above. +If that doesn't work, you'll have to edit ``initFixStacks`` in ``fix_stacks.py`` to +set ``breakpadSymsDir`` to ``<SYMB_DIR>``. + +With all of that done, you should now be able to run ``dmd.py`` or ``fix_stacks.py`` +to fix the stacks. Note that the stack fixing process can take a minute or two. diff --git a/docs/contributing/debugging/process_dump_task_manager.rst b/docs/contributing/debugging/process_dump_task_manager.rst new file mode 100644 index 0000000000..d345171856 --- /dev/null +++ b/docs/contributing/debugging/process_dump_task_manager.rst @@ -0,0 +1,69 @@ +How to get a process dump with Windows Task Manager +=================================================== + +Introduction +------------ + +When tracking down the causes of process hangs, it is often helpful to +obtain a process dump while the process is experiencing a hang. This +article describes how to get a process dump with Task Manager on +Windows. (To get a process dump for Thunderbird or some other product, +substitute the product name where ever you see Firefox in these +instructions.) + + +Caution +------- + +The memory dump that will be created through this process is a complete +snapshot of the state of Firefox when you create the file, so it +contains URLs of active tabs, history information, and possibly even +passwords depending on what you are doing when the snapshot is taken. It +is advisable to create a new, blank profile to use when reproducing the +hang and capturing the memory dump. Please ask for help doing this! + + +Requirements +------------ + +Windows + To get a process dump, you need to be using Windows Vista or above. +A Firefox nightly or release + You need a Firefox version for which symbols are available from the + :ref:`symbol server <Using The Mozilla Symbol Server>`. You + can use any `official nightly + build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released + version of Firefox from Mozilla. You can find the latest trunk + nightly builds under + `http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/>`__. + + +Creating the Dump File +---------------------- + +Ensure that Firefox is not already running. + + +Run Firefox, reproduce the hang +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Start Firefox and perform whatever steps are necessary to cause Firefox +to hang. Once the browser hangs, continue with the steps below. + + +After the hang +~~~~~~~~~~~~~~ + +#. Open Windows Task Manager (CTRL+SHIFT+ESC). +#. Find Firefox.exe among the list of processes. +#. Right-click Firefox.exe and select "Create dump file". Task manager + should indicate where the dump file was written to. + + +See also +-------- + +- :ref:`How to get a stacktrace for a bug report <How to get a stacktrace for a bug report>` +- `How to create a user-mode process dump file in Windows Vista and in + Windows 7 + (MSDN) <https://docs.microsoft.com/en-us/windows/client-management/generate-kernel-or-complete-crash-dump#manually-generate-a-memory-dump-file>`__ diff --git a/docs/contributing/debugging/stacktrace_report.rst b/docs/contributing/debugging/stacktrace_report.rst new file mode 100644 index 0000000000..2cae230745 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_report.rst @@ -0,0 +1,153 @@ +How to get a stacktrace for a bug report +======================================== + +If you file a bug report in Bugzilla about a crash you should include a +stacktrace (call stack) in your report. A stacktrace will tell Mozilla +developers what crashed and provide a starting point for investigating +its cause. This article describes how to use the Mozilla Crash Reporter +(Breakpad) to get a crash ID, which our engineers can use to get a +stacktrace, and alternative ways to get a stacktrace if you can't get a +crash ID. + +Requirements +------------ + +You need a binary build of Firefox from +`Mozilla.org <https://www.mozilla.org/firefox/>`__. SeaMonkey and +Thunderbird also support crash reporting. + +Mozilla's crash report server currently only has debug information for +Mozilla builds and thus the crash reporter cannot work if you use a +build from a Linux distribution or if you compile from source code. In +these cases you will need to use one of the :ref:`alternative +methods <Alternative ways to get a stacktrace>` listed below. + +.. note:: + + **Note:** When filing a crash report, it is important to know whether + the crash occurs with `Firefox safe + mode <http://support.mozilla.com/kb/Safe+Mode>`__. This helps + engineers determine whether a particular + `extension <http://support.mozilla.com/kb/Troubleshooting+extensions+and+themes>`__ + or + `plugin <http://support.mozilla.com/kb/Troubleshooting+plugins>`__ + is the cause of the crash. + + +How to get a crash ID with the Mozilla Crash Reporter +----------------------------------------------------- + +1 - Crash and submit a report to the system. + +.. image:: img/crashreporter.png + +The Mozilla Crash Reporter window should automatically come up after Firefox crashes. +If you have any additional information about the crash, such as additional detail on +what you were doing at the time that may have triggered the crash, please enter it +into the comments box. Be sure that you **check the "Tell Mozilla about this crash"** +checkbox and click the restart button. The crash reporter should now submit the +crash report and Firefox should open again. + +.. note:: + + The "Details" button gives additional data about the incident, + however this is not useful in a bug report. + + +2 - Tell us the ID of the report you submitted. + +.. image:: img/crashlist.jpg + +To access all of your submitted reports type "about:crashes" into the Firefox address bar +and press enter. Firefox should open a list of IDs for your submitted crash reports. +Copy two or three of the IDs for the appropriate crashes and paste them into your +Bugzilla report. Please check the listed times to avoid copying the ID of an unrelated +crash report. + +.. note:: + + You can prefix a "bp-" to the beginning of an ID to make Bugzilla turn it + into a link: bp-a70759c6-1295-4160-aa30-bc4772090918 + + +How to get the crash ID if Firefox crashes on startup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox crashes on startup you can still access your submitted crash +reports. Crash reports are accessible from all Firefox profiles, so if a +`new +profile <https://support.mozilla.org/kb/profile-manager-create-remove-switch-firefox-profiles>`__ +does not crash you can use it to access them through "about:crashes" as above. + + +Accessing crash report IDs outside of Firefox +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you cannot load Firefox at all you can find the crash report files at +this location depending on your operating system: + +* Windows : ``%APPDATA%\Mozilla\Firefox\Crash Reports\submitted\`` +* macOS : ``~/Library/Application Support/Firefox/Crash Reports/submitted/`` +* Linux : ``~/.mozilla/firefox/Crash Reports/submitted/`` + +Each file in this folder contains one submitted crash report ID. You can +check the modified or creation time for each file to discern which crash +reports are relevant to your bug report. + +.. _Alternative ways to get a stacktrace: + +Alternative ways to get a stacktrace +------------------------------------ + +If the Mozilla crash reporter doesn't come up or isn't available you +will need to obtain a stacktrace manually: + + +Windows +~~~~~~~ + +See the article :ref:`Create a stacktrace with Windbg <How to get a stacktrace with WinDbg>` for information +on how to do this. + +For a full process dump, see :ref:`How to get a process dump with Windows +Task Manager`. + + +macOS +~~~~~ + +Run /Applications/Utilities/Console.app. Expand "~/Library/Logs" and +"CrashReporter", then look for logs for "firefox-bin". + + +Linux +~~~~~ + +Note that for most distros, the package you need to get symbols for will +be something like "xulrunner", not "firefox". + + +Crash reports files on your computer +------------------------------------ + +When Breakpad initially catches a crash it first writes crash report +files (e.g. .dump and .extra files) into the 'pending' subdirectory of +its 'Crash Reports' directory. + +If Breakpad successfully sends the crash report to the reporting server +then, by default, the files added to the 'pending' subdirectory for the +crash are removed, and a .txt file is placed in the 'submitted' +directory containing the crash ID created by the reporting server. + +If you want Breakpad to leave the .dump and .extra files on your +computer so that you can examine them locally, then set the +MOZ_CRASHREPORTER_NO_DELETE_DUMP environment variable to 1. + +- Ubuntu: `Instructions from the Ubuntu + Team <https://wiki.ubuntu.com/MozillaTeam/Bugs#Obtain%20a%20backtrace%20from%20an%20apport%20crash%20report%20(using%20gdb)>`__ +- openSUSE: `General instructions from + openSUSE <https://en.opensuse.org/openSUSE:Bugreport_application_crashed>`__ +- Fedora: `Capturing Stack + Traces <https://fedoraproject.org/wiki/StackTraces>`__ +- Gentoo: `Debugging using + GDB <https://wiki.gentoo.org/wiki/Debugging_with_GDB>`__ diff --git a/docs/contributing/debugging/stacktrace_windbg.rst b/docs/contributing/debugging/stacktrace_windbg.rst new file mode 100644 index 0000000000..a0d2abfc25 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_windbg.rst @@ -0,0 +1,232 @@ +How to get a stacktrace with WinDbg +=================================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Introduction +------------ + +Sometimes you need to get a stacktrace (call stack) for a crash or hang +but `Breakpad <http://kb.mozillazine.org/Breakpad>`__ fails because it's +a special crash or a hang. This article describes how to get a +stacktrace in those cases with WinDbg on Windows. (To get a stacktrace +for Thunderbird or some other product, substitute the product name where +ever you see Firefox in this instructions.) + +Requirements +------------ + +To get such a stacktrace you need to install the following software: + +Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Microsoft distributes the Debugging Tools for Windows for free, those +include WinDbg which you will need here. Download it from `Install +Debugging Tools for +Windows <https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk>`__. +(*You'll want the 32-bit version*, even if you are using a 64-bit +version of Windows) Then install it, the standard settings in the +installation process are fine. + +A Firefox nightly or release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You need a Firefox version for which symbols are availables from the +:ref:`symbol server <Using The Mozilla Symbol Server>` to use +with WinDbg. You can use any `official nightly +build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released +version of Firefox from Mozilla. You can find the latest trunk nightly +builds under +`http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/>`__. + + +Debugging +--------- + +To begin debugging, ensure that Firefox is not already running and open +WinDbg from the Start menu. (Start->All Programs->Debugging Tools for +Windows->WinDbg) Next, open the **"File"** menu and choose **"Open +Executable..."**. In the file chooser window that appears, open the +firefox.exe executable in your Firefox program folder (C:\Program +Files\Mozilla Firefox). + +You should now see a "Command" text window with debug output at the top +and an input box at the bottom. Before debugging can start, several +commands must be entered into the one-line input box at the bottom of +the Command window. + +.. note:: + + Tip: All commands must be entered exactly as written, one line at a + time, into the bottom of the Command box. + + - Copying and pasting each line is the easiest method to avoid + mistakes + - Some commands start with a period (.) or a pipe character (|), + which is required. (The keystroke for a pipe character on US + keyboards is SHIFT+\) + - Submit the log file on a bug or via the support site, even if + nothing seems to happen during the debug process + + +Start debugging +~~~~~~~~~~~~~~~ + +Now that Firefox is opened in the debugger, you need to configure your +WinDbg to download symbols from the Mozilla symbol server. To load the +symbols, enter the three commands below, pressing enter after each one. +(More details are available at :ref:`symbol server <Using The Mozilla Symbol Server>`.) + +:: + + .sympath SRV*c:\symbols*http://symbols.mozilla.org/firefox;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols + .symfix+ c:\symbols + .reload /f + +Now wait for the symbols to download. This may take some time depending +on your connection speed; the total size of the Mozilla and Microsoft +symbols download is around 1.4GB. WinDbg will show "Busy" at the bottom +of the application window until the download is complete. + +Once the download is complete, you need to configure WinDbg to examine +child processes, ignore a specific event caused by Flash Player, and +record a log of loaded modules. You will also want to open a log file to +save data you collect. To do this, enter these four commands, pressing +enter after each one. + +:: + + .logopen /t c:\temp\firefox-debug.log + .childdbg 1 + .tlist + sxn gp + lm + +If you see firefox.exe listed in the output from .tlist more than once, +then you are already running the application and need to close the +running instance first before you start debugging, otherwise you won't +get useful results. + +Now run Firefox by opening the **Debug** menu and clicking **Go**. +**While Firefox is running, you will not be able to type any commands +into the debugger.** After it starts, try to reproduce the crash or +hanging issue that you are seeing. + +.. note:: + + If Firefox fails to start, and you see lines of text followed by a + command prompt in the debugger, a "breakpoint" may have been + triggered. If you are prompted for a command but don't see an error + about a crash, go back to the **Debug** menu and press **Go**. + +Once the browser crashes, you will see an error (such as "Access +violation") in the WinDbg Command window. If Firefox hangs and there is +no command prompt available in the debugger, open the **Debug** menu and +choose **Break.** Once the browser has crashed or been stopped, continue +with the steps below. + + +After the crash or hang +~~~~~~~~~~~~~~~~~~~~~~~ + +You need to capture the debug information to include in a bug comment or +support request. Enter these three commands, one at a time, to get the +stacktrace, crash/hang analysis and log of loaded modules. (Again, press +Enter after each command.) + +:: + + ~* kp + !analyze -v -f + lm + +After these steps are completed, find the file +**c:\temp\firefox-debug-(Today's Date).txt** on your hard drive. To +provide the information to the development community, submit this file +with a `support request <https://support.mozilla.com/>`__ or attach it +to a related bug on `Bugzilla <https://bugzilla.mozilla.org/>`__. + + +Producing a minidump +~~~~~~~~~~~~~~~~~~~~ + +Sometimes the stacktrace alone is not enough information for a developer +to figure out what went wrong. A developer may ask you for a "minidump" +or a "full memory dump", which are files containing more information +about the process. :ref:`You can easily produce minidumps from WinDBG and +provide them to developers <Capturing a minidump>`. + +FAQ + +Q: I am running Windows 7 (32-bit or 64-bit) and I see an exception in +the WinDbg command window that says 'ntdll32!LdrpDoDebuggerBreak+0x2c' +or 'ntdll32!LdrpDoDebuggerBreak+0x30'. What do I do now? + +A: If you see 'int 3' after either of those exceptions, you will need to +execute the following commands in WinDbg. + +:: + + bp ntdll!LdrpDoDebuggerBreak+0x30 + bp ntdll!LdrpDoDebuggerBreak+0x2c + eb ntdll!LdrpDoDebuggerBreak+0x30 0x90 + eb ntdll!LdrpDoDebuggerBreak+0x2c 0x90 + +| Make sure you enter them one at a time and press enter after each one. + If you use the 64-bit version of Windows, you need to replace "ntdll" + in these commands with "ntdll32". +| Q: The first four frames of my stack trace look like this: + +:: + + 0012fe20 7c90e89a ntdll!KiFastSystemCallRet + 0012fe24 7c81cd96 ntdll!ZwTerminateProcess+0xc + 0012ff20 7c81cdee kernel32!_ExitProcess+0x62 + + 0012ff34 6000179e kernel32!ExitProcess+0x14 + +This looks wrong to me?! + +A: You ran the application without the "Debug child processes also" +check box being checked. You need to detach the debugger and open the +application again, this time with the check box being checked. + +Q: WinDbg tells me that it is unable to verify checksum for firefox.exe. +Is this normal? + +A: Yes, this is normal and can be ignored. + +Q: Should I click yes or no when WinDbg asks me to "Save information for +workspace?" + +A: Click yes and WinDbg will save you from having to enter in the symbol +location for Firefox.exe in the future. Click no if you'd rather not +having WinDbg save this information. + +Q: I'm seeing "wow64" on top of each thread, is that ok ? + +A: No, you are running a 64 bit version of Windbg and trying to debug a +32 bit version of the mozilla software. Redownload and install the 32 +bit version of windbg. + + +Troubleshooting: Symbols will not download +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If symbols will not download no matter what you do, the problem may be +that Internet Explorer has been set to the **Work Offline** mode. You +will not receive any warnings of this in Windbg, Visual C++ or Visual +Studio. Even using the command line with symchk.exe to download symbols +will fail. This is because Microsoft uses Internet Explorer's internet & +proxy settings to download the symbol files. Check the File menu of +Internet Explorer to ensure "Work Offline" is unchecked. + + +See also +-------- + +- :ref:`symbol server <Using The Mozilla Symbol Server>` Maps addresses to human readable strings. +- :ref:`source server <Using The Mozilla Source Server>` Maps addresses to source code lines diff --git a/docs/contributing/debugging/understanding_crash_reports.rst b/docs/contributing/debugging/understanding_crash_reports.rst new file mode 100644 index 0000000000..9f8e5bbe03 --- /dev/null +++ b/docs/contributing/debugging/understanding_crash_reports.rst @@ -0,0 +1,325 @@ +Understanding Crash Reports +=========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +If a user experiences a crash they will be prompted to submit a raw +crash report, which is generated by Breakpad. The raw crash report is +received by `Socorro <https://github.com/mozilla/socorro>`__ which +`creates <https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py>`__ +a processed crash report. The processed crash report is based on the raw +crash report but also has a signature, classifications, and a number of +improved fields (e.g. OS, product, version). Many of the fields in both +the raw crash report and the processed crash report are viewable and +searchable on `crash-stats <https://crash-stats.mozilla.org/>`__. +Although there are two distinct crash reports, the raw and the +processed, people typically talk about a single "crash report" because +crash-stats mostly presents them in a combined way. + +Each crash report contains a wealth of data about the crash +circumstances. Despite this, many crash reports lack sufficient data for +a developer to understand why the crash occurred. As well as providing a +general overview, this page aims to highlight parts of a crash report +that may provide non-obvious insights. + +Note that most crash report fields are visible, but a few +privacy-sensitive parts of it are only available to users who are logged +in and have "minidump access". A relatively small number of users have +minidump access, and they are required to follow certain rules. For +access, see the `Protected Data Access docs on Crash Stats +<https://crash-stats.mozilla.org/documentation/protected_data_access/>`__. + +Each crash report has the following tabs: Details, Metadata, Modules, +Raw Dump, Extensions, and (optional) Correlations. + +Details tab +----------- + +The Details tab is the first place to look because it contains the most +important pieces of information. + +Primary fields +~~~~~~~~~~~~~~ + +| The first part of the Details tab shows a table containing the most + important crash report fields. It includes such things as when the + crash occurred, in which product and version, the crash kind, and + various details about the OS and configuration of the machine on which + the crash occurred. The following screenshot shows some of these + fields. +| |Example fields in the "Details" tab of a crash report| + +All fields have a tool-tip. For many fields, the tool-tip describes its +meaning. For all fields, the tool-tip indicates the key to use when you +want to do searches involving this field. (The field name is usually but +not always similar to the search key. E.g. the field "Adapter Device ID" +has the search key "adapter_device_id".) These descriptions are shown in +the `SuperSearchFields +API <https://crash-stats.mozilla.org/api/SuperSearchFields/>`__ and can be +`modified in super_search_fields.py <https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py>`__ +or by writing up a `bug in Socorro <https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro>`__. + +The fields present in this tab vary depending on the crash kind. Not all +fields are always present. + +The "Signature" field is the main identifier or label for a crash report. +Rather than considering each crash report in isolation, we want to put +crash reports into clusters so we can deal with groups of them at once. +An ideal clustering algorithm would put all crash reports with the same +root cause into a single cluster, and all crash reports with different +root causes into different clusters. The crash signature is our +imperfect but still useful attempt at such an algorithm. Most crash +signatures are based on the crashing stack trace, but some +special-purpose annotations are used to indicate particular kinds of +crashes. + +- ``Abort``: A controlled abort, e.g. via ``NS_RUNTIMEABORT``. + (Controlled aborts that occur via ``MOZ_CRASH`` or + ``MOZ_RELEASE_ASSERT`` currently don't get an ``Abort`` annotation, + but they do get a "MOZ_CRASH Reason" field.) +- ``OOM | <size>``, where ``<size>`` is one of ``large``, ``small``, + ``unknown``: an out-of-memory (OOM) abort. The ``<size>`` annotation + is determined by the "OOM Allocation Size" field; if that field is + missing ``<size>`` will be ``unknown``. +- ``hang``: a hang prior to shutdown. +- ``shutdownhang``: a hang during shutdown. +- ``IPCError-browser``: a problem involving IPC. If the parent Firefox + process detects that the child process has sent broken or + unprocessable IPDL data, or is not shutting down in a timely manner, + it kills the child process with a crash report. These crashes will + now have a signature that indicates why the process was killed, + rather than the child stack at the moment. + +When no special-purpose annotation is present and the signature begins +with a stack frame, it's usually a vanilla uncontrolled crash. The crash +cause can be determined from the "Crash Reason" field. Most commonly +it's a bad memory access. In that case, on Windows you can tell from the +reason field if the crash occurred while reading, writing or executing +memory (e.g. ``EXCEPTION_VIOLATION_ACCESS_READ`` indicates a bad memory +read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you +cannot tell from this field what kind of memory access it was. + +See `this +file <https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst>`__ +for a detailed explanation of the crash report signature generation +procedure, and for information on how modify this procedure. + +There are no fields that uniquely identify the user that a crash report +came from, but if you want to know if multiple crashes come from a +single user the "Install Time" field is a good choice. Use it in +conjunction with other fields that don't change, such as those +describing the OS or graphics card, for additional confidence. + +For bad memory accesses, the "Crash Address" field can give additional +indications what went wrong. + +- 0x0 is probably a null pointer deference[*]. +- Small addresses like 0x8 can indicate an object access (e.g. + ``this->mFoo``) via a null ``this`` pointer. +- Addresses like 0xfffffffffd8 might be stack accesses, depending on + the platform[*]. +- Addresses like 0x80cdefd3 might be heap accesses, depending on the + platform. +- Addresses may be poisoned: 0xe4 indicates the address comes from + memory that has been allocated by jemalloc but not yet initialized; + 0xe5 indicates the address comes from memory freed by jemalloc. The + JS engine also has multiple poison values defined in + ``js/src/jsutil.h``. + +[*] Note that due to the way addressing works on x86-64, if the crash +address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for +a Windows crash report, it's highly likely that the value is incorrect. +(There is a `bug +report <https://bugzilla.mozilla.org/show_bug.cgi?id=1493342>`__ open +for this problem.) You can sanity-check these crashes by looking at the +raw dump or minidump in the Raw Dump tab (see below). + +Note that for non-release builds the "Version" field represents multiple +different builds since nightly and beta version numbers are reused for +builds created over a series of days until the version number is bumped. +(The "Build ID" field can disambiguate.) It's not currently possible to +`restrict searches to a given version or +later <https://bugzilla.mozilla.org/show_bug.cgi?id=1401517>`__ (using +>= with a build ID and a given release channel may work around this). + +Some fields, such as "URL" and "Email Address", are privacy-sensitive +and are only visible to users with minidump access. + +The Windows-only "Total Virtual Memory" field indicates if the Firefox +build and OS are 32-bit or 64-bit. + +- A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows. +- A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows + (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox. +- A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit + Firefox. (The "Build Architecture" field should be "amd64" in this + case.) + +Some crash reports might contain a memory report. This memory report will +have been made some time before the crash, at a time when available +memory was low. In this case, a number of key measurements from the +memory report are shown in the Details tab, each one having a field name +starting with "MR:", short for "memory report". The full memory report +can be obtained in the Raw Dump tab (see below). + +Bug-related information +~~~~~~~~~~~~~~~~~~~~~~~ + +The second part of the Details tab shows bug-related information, as the +following screenshot shows. + +|Information relating to bug reports in the "Details" tab of a crash +report| + +The "Report this bug in" links can be used to easily file bug reports. +Each one links to a Bugzilla bug report creation page that has various +fields pre-filled, such as the crash signature. + +The "Related Bugs" section shows related bug reports, as determined by +the crash signature. + +Stack traces +~~~~~~~~~~~~ + +The third part of the Details tab shows the stack trace and thread +number of the crashing thread, as the following screenshot shows. + +|Information relating to threads in the "Details" tab of a crash report| + +Each stack frame has a link to the source code, when possible. If a +crash is new, the regressing changeset can often be identified by +looking for recent changes in the blame annotations for one or more of +the top stack frames. Blame annotations are also good for identifying +who might know about the code in question. + +Sometimes the highlighted source code is puzzling, e.g. the identified +line may not touch memory even though the crash is memory-related. This +can be caused by compiler optimizations. It's often better to look at +the disassembly (e.g. in a minidump) to understand exactly what code is +being executed. + +Stack frame entries take on a variety of forms. + +- The simplest are functions names, such as ``NS_InitXPCOM2``. +- Name/address pairs such as ``nss3.dll@0x1eb720`` are within system + libraries. +- Names such as ``F1398665248_____________________________`` ('F' + followed by many numbers then many underscores) are in Flash. +- Addresses such as ``@0xe1a850ac`` may indicate an address that wasn't + part of any legitimate code. If an address such as this occurs in the + first stack frame, the crash may be + `exploitable <https://developer.mozilla.org/en-US/docs/Mozilla/Security/Exploitable_crashes>`__. + +Stack traces for other threads can be viewed by clicking on the small +"Show other threads" link. + +If the crash report is for a hang, the crashing thread will be the +"watchdog" thread, which exists purely to detect hangs; its top stack +frame will be something +like\ :literal:`mozilla::`anonymous namespace'::RunWatchdog`. In that +case you should look at the other threads' stack traces to determine the +problem; many of them will be waiting on some kind of response, as shown +by a top stack frame containing a function like +``NtWaitForSingleObject`` or ``ZwWaitForMultipleObjects``. + +Metadata tab +------------ + +The Metadata tab is similar to the first part of the Details tab, +containing a table with various fields. These are the fields from the +raw crash report, ordered alphabetically by field name, but with +privacy-sensitive fields shown only to users with minidump access. There +is some overlap with the fields shown in the Details tab. + +Modules tab +----------- + +The modules tab shows all the system libraries loaded at the time of the +crash, as the following screenshot shows. + +|Table of modules in the "Modules" tab of a crash report| + +On Windows these are mostly DLLs, on Mac they are mostly ``.dylib`` +files, and on Linux they are mostly ``.so`` files. + +This information is most useful for Windows crashes, because DLLs loaded +by antivirus software or malware often cause Firefox to crash. +Correlations between loaded modules and crash signatures can be seen in +the "Correlations" tab (see below). + +`This page <https://support.mozilla.org/en-US/kb/helping-crashes>`__ +says that files lacking version/debug identifier/debug filename are +likely to be malware. + +Raw Dump tab +------------ + +The first part of the Raw Dump tab shows the raw crash report, in JSON +format. Once again, privacy-sensitive fields are shown only to users +with minidump access. + +|JSON data in the "Raw Dump" tab of a crash report| + +For users with minidump access, the second part of the Raw Dump tab has +some links, as the following screenshot shows. + +|Links to downloadable files in the "Raw Dump" tab of a crash report| + +These links are to the following items. + +#. A minidump. Minidumps can be extremely useful in understanding a + crash report; see :ref:`this page <Debugging A Minidump>` for an + explanation how to use them. +#. The aforementioned JSON raw crash report. +#. The memory report contained within the crash report. +#. The unredacted crash report, which has additional information. + +Extensions tab +-------------- + +The Extensions tab shows which extensions are installed and enabled. + +|Table of extensions in the "Extensions" tab of a crash report| + +Usually it just shows an ID rather than the proper extension name. + +Note that several extensions ship by default with Firefox and so will be +present in almost all crash reports. (The exact set of default +extensions depends on the release channel.) The least obvious of these +has an Id of ``{972ce4c6-7e08-4474-a285-3208198ce6fd}``, which is the +default Firefox theme. Some (but not all) of the other extensions +shipped by default have the following Ids: ``webcompat@mozilla.org``, +``e10srollout@mozilla.org``, ``firefox@getpocket.com``, +``flyweb@mozilla.org``, ``loop@mozilla.org``. + +If an extension only has a hexadecimal identifier, a Google search of +that identifier is usually enough to identify the extension's name. + +This information is useful because some crashes are caused by +extensions. Correlations between extensions and crash signatures can be +seen in the "Correlations" tab (see below). + +Correlations tab +---------------- + +This tab is only shown when crash-stats identifies correlations between +a crash and modules or extensions that are present, which happens +occasionally. + +See also +-------- + +- `A talk about understanding crash + reports <https://air.mozilla.org/a-talk-about-understanding-crash-reports/>`__, + by David Baron, from March 2016. +- :ref:`A guide to searching crash reports` + +.. |Example fields in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13579/Details1.png +.. |Information relating to bug reports in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13581/Details2.png +.. |Information relating to threads in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13583/Details3.png +.. |Table of modules in the "Modules" tab of a crash report| image:: https://mdn.mozillademos.org/files/13593/Modules1.png +.. |JSON data in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/13595/RawDump1.png +.. |Links to downloadable files in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/14047/raw-dump-links.png +.. |Table of extensions in the "Extensions" tab of a crash report| image:: https://mdn.mozillademos.org/files/13599/Extensions1.png |
