Symbolicating TreeHerder stacks locally
=======================================

When using tools like the :ref:`Dark Matter Detector (DMD)` or
:ref:`refcount logging<Refcount Tracing and Balancing>` to
investigate issues occurring on TreeHerder that you can't reproduce locally, you
can often end up with unsymbolicated stacks. Fortunately, there is a way to
symbolicate these stacks on your own machine.

These instructions are for a Linux TreeHerder build for MacOS, so they might
require some modifications for other combinations of platforms.

Download ``target.tar.bz2`` and ``target.crashreporter-symbols.zip`` from the
Build job. **Note that these files are very large so you'll want to delete
them and the extracted files when you are done.**

These files each contain a large number of files, so I'd recommend creating
a directory for each of them. Call these ``<TARGET_DIR>`` and ``<SYMB_DIR>``,
and move the prior two files into these two directories, respectively.

Go to ``<TARGET_DIR>`` and run something like

.. code-block:: shell

    tar xf target.tar.bz2

then go to ``<SYMB_DIR>`` and run something like

.. code-block:: shell

    unzip target.crashreporter-symbols.zip

You should be able to delete the two original files now.

Next we need to ensure that the locations of binaries are rewritten from
where they are on TreeHerder to where we have them locally. We'll do this by
editing ``fix_stacks.py``. This file is located in the ``tools/rb/`` directory of
the Firefox source directory. You need to add these two lines to the function
``fixSymbols``, after ``line_str`` is defined and before it is written to
``fix_stacks.stdin``. I've done this right before the definition of
``is_missing_newline``.

.. code-block:: python

    line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox",
                                "<TARGET_DIR>/firefox/firefox-bin")
    line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so",
                                "<TARGET_DIR>/firefox/libxul.so")

The initial locations should appear verbatim in the stack you are trying to
symbolicate, so double check that they match. Also, ``<TARGET_DIR>`` of course
needs to be replaced with the actual local directories where those files are
located. Note that the ``firefox`` executable is changed to ``firefox-bin``.
I don't know why that is necessary, but only the latter existed for me.

Finally, we need to make it so that the stack fixer can find the location of
the breakpad symbols we downloaded. If you are running ``fix_stacks.py`` via
``dmd.py`` or directly (in a recent version), you can do this by running with the
environment variable ``BREAKPAD_SYMBOLS_PATH`` set to the ``<SYMB_DIR>`` from above.
If that doesn't work, you'll have to edit ``initFixStacks`` in ``fix_stacks.py`` to
set ``breakpadSymsDir`` to ``<SYMB_DIR>``.

With all of that done, you should now be able to run ``dmd.py`` or ``fix_stacks.py``
to fix the stacks. Note that the stack fixing process can take a minute or two.