summaryrefslogtreecommitdiffstats
path: root/tools/sanitizer/docs
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-07 19:33:14 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-07 19:33:14 +0000
commit36d22d82aa202bb199967e9512281e9a53db42c9 (patch)
tree105e8c98ddea1c1e4784a60a5a6410fa416be2de /tools/sanitizer/docs
parentInitial commit. (diff)
downloadfirefox-esr-36d22d82aa202bb199967e9512281e9a53db42c9.tar.xz
firefox-esr-36d22d82aa202bb199967e9512281e9a53db42c9.zip
Adding upstream version 115.7.0esr.upstream/115.7.0esr
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'tools/sanitizer/docs')
-rw-r--r--tools/sanitizer/docs/asan.rst378
-rw-r--r--tools/sanitizer/docs/asan_nightly.rst204
-rw-r--r--tools/sanitizer/docs/index.rst33
-rw-r--r--tools/sanitizer/docs/memory_sanitizer.rst173
-rw-r--r--tools/sanitizer/docs/tsan.rst327
5 files changed, 1115 insertions, 0 deletions
diff --git a/tools/sanitizer/docs/asan.rst b/tools/sanitizer/docs/asan.rst
new file mode 100644
index 0000000000..a241908ced
--- /dev/null
+++ b/tools/sanitizer/docs/asan.rst
@@ -0,0 +1,378 @@
+Address Sanitizer
+=================
+
+What is Address Sanitizer?
+--------------------------
+
+Address Sanitizer (ASan) is a fast memory error detector that detects
+use-after-free and out-of-bound bugs in C/C++ programs. It uses a
+compile-time instrumentation to check all reads and writes during the
+execution. In addition, the runtime part replaces the ``malloc`` and
+``free`` functions to check dynamically allocated memory. More
+information on how ASan works can be found on `the Address Sanitizer
+wiki <https://github.com/google/sanitizers/wiki/AddressSanitizer>`__.
+
+A `meta bug called asan-maintenance <https://bugzilla.mozilla.org/show_bug.cgi?id=asan-maintenance>`__
+is maintained to keep track of all the bugs found with ASan.
+
+Downloading artifact builds
+---------------------------
+
+For Linux and Windows users, the easiest way to get Firefox builds with
+Address Sanitizer is to download a continuous integration asan build of
+mozilla-central (updated at least daily):
+
+- mozilla-central optimized builds:
+ `linux <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-asan-opt/artifacts/public/build/target.tar.bz2>`__
+ \|
+ `windows <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.win64-asan-opt/artifacts/public/build/target.zip>`__
+ (recommended for testing)
+- mozilla-central debug builds:
+ `linux <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-asan-debug/artifacts/public/build/target.tar.bz2>`__
+ \|
+ `windows <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.win64-asan-debug/artifacts/public/build/target.zip>`__
+ (recommended for debugging if the optimized builds don't do the job)
+
+The fuzzing team also offers a tool called ``fuzzfetch`` to download these and many
+other CI builds. It makes downloading and unpacking these builds much easier and
+can be used not just for fuzzing but for all purposes that require a CI build download.
+
+You can install ``fuzzfetch`` from
+`Github <https://github.com/MozillaSecurity/fuzzfetch>`__ or
+`via pip <https://pypi.org/project/fuzzfetch/>`__.
+
+Afterwards, you can run e.g.
+
+::
+
+ $ python -m fuzzfetch --asan -n firefox-asan
+
+to get the optimized Linux ASan build mentioned above unpacked into a directory called ``firefox-asan``.
+The ``--debug`` and ``--os`` switches can be used to get the other variants listed above.
+
+Creating Try builds
+-------------------
+
+If for some reason you can't use the pre-built binaries mentioned in the
+previous section (e.g. you want a non-Linux build or you need to test a
+patch), you can either build Firefox yourself (see the following
+section) or use the :ref:`try server <Pushing to Try>` to
+create the customized build for you. Pushing to try requires L1 commit
+access. If you don't have this access yet you can request access (see
+`Becoming A Mozilla
+Committer <https://www.mozilla.org/about/governance/policies/commit/>`__
+and `Mozilla Commit Access
+Policy <https://www.mozilla.org/about/governance/policies/commit/access-policy/>`__
+for the requirements).
+
+The tree contains `several mozconfig files for creating asan
+builds <https://searchfox.org/mozilla-central/search?q=&case=true&path=browser%2Fconfig%2Fmozconfigs%2F*%2F*asan*>`__
+(the "nightly-asan" files create release builds, whereas the
+"debug-asan" files create debug+opt builds). For Linux builds, the
+appropriate configuration file is used by the ``linux64-asan`` target.
+If you want to create a macOS or Windows build, you'll need to copy the
+appropriate configuration file over the regular debug configuration
+before pushing to try. For example:
+
+::
+
+ cp browser/config/mozconfigs/macosx64/debug-asan browser/config/mozconfigs/macosx64/debug
+
+You can then `push to Try in the usual
+way </tools/try/index.html#using-try>`__
+and, once the build is complete, download the appropriate build
+artifact.
+
+Creating local builds on Windows
+--------------------------------
+
+On Windows, ASan is supported only in 64-bit builds.
+
+Run ``mach bootstrap`` to get an updated clang-cl in your
+``~/.mozbuild`` directory, then use the following
+:ref:`mozconfig <Configuring Build Options>`:
+
+::
+
+ ac_add_options --enable-address-sanitizer
+ ac_add_options --disable-jemalloc
+
+ export LDFLAGS="clang_rt.asan_dynamic-x86_64.lib clang_rt.asan_dynamic_runtime_thunk-x86_64.lib"
+ CLANG_LIB_DIR="$(cd ~/.mozbuild/clang/lib/clang/*/lib/windows && pwd)"
+ export MOZ_CLANG_RT_ASAN_LIB_PATH="${CLANG_LIB_DIR}/clang_rt.asan_dynamic-x86_64.dll"
+
+If you launch an ASan build under WinDbg, you may see spurious
+first-chance Access Violation exceptions. These come from ASan creating
+shadow memory pages on demand, and can be ignored. Run ``sxi av`` to
+ignore these exceptions. (You will still catch second-chance Access
+Violation exceptions if you actually crash.)
+
+LeakSanitizer (LSan) is not supported on Windows.
+
+Creating local builds on Linux or Mac
+-------------------------------------
+
+Build prerequisites
+~~~~~~~~~~~~~~~~~~~
+
+LLVM/Clang
+^^^^^^^^^^
+
+The ASan instrumentation is implemented as an LLVM pass and integrated
+into Clang. Any clang version that is capable of compiling Firefox has
+everything needed to do an ASAN build.
+
+Building Firefox
+~~~~~~~~~~~~~~~~
+
+Getting the source
+^^^^^^^^^^^^^^^^^^
+
+Using that or any later revision, all you need to do is to :ref:`get yourself
+a clone of mozilla-central <Mercurial overview>`.
+
+Adjusting the build configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Create the build configuration file ``mozconfig`` with the following
+content in your mozilla-central directory:
+
+::
+
+ # Combined .mozconfig file for ASan on Linux+Mac
+
+ mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-asan
+
+ # Enable ASan specific code and build workarounds
+ ac_add_options --enable-address-sanitizer
+
+ # These three are required by ASan
+ ac_add_options --disable-jemalloc
+ ac_add_options --disable-crashreporter
+ ac_add_options --disable-elf-hack
+
+ # Keep symbols to symbolize ASan traces later
+ export MOZ_DEBUG_SYMBOLS=1
+ ac_add_options --enable-debug-symbols
+ ac_add_options --disable-install-strip
+
+ # Settings for an opt build (preferred)
+ # The -gline-tables-only ensures that all the necessary debug information for ASan
+ # is present, but the rest is stripped so the resulting binaries are smaller.
+ ac_add_options --enable-optimize="-O2 -gline-tables-only"
+ ac_add_options --disable-debug
+
+ # Settings for a debug+opt build
+ #ac_add_options --enable-optimize
+ #ac_add_options --enable-debug
+
+ # MacOSX only: Uncomment and adjust this path to match your SDK
+ # ac_add_options --with-macos-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk
+
+You may also need this, as seen in
+``browser/config/mozconfigs/linux64/nightly-asan`` (the config file used
+for Address Sanitizer builds used for automated testing):
+
+::
+
+ # ASan specific options on Linux
+ ac_add_options --enable-valgrind
+
+Starting the build process
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now you start the build process using the regular ``./mach build``
+command.
+
+Starting Firefox
+^^^^^^^^^^^^^^^^
+
+After the build has completed, ``./mach run`` with the usual options for
+running in a debugger (``gdb``, ``lldb``, ``rr``, etc.) work fine, as do
+the ``--disable-e10s`` and other options.
+
+Building only the JavaScript shell
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to build only the JavaScript shell instead of doing a full
+Firefox build, the build script below will probably help you to do so.
+Execute this script in the ``js/src/`` subdirectory and pass a directory
+name as the first parameter. The build will then be created in a new
+subdirectory with that name.
+
+::
+
+ #! /bin/sh
+
+ if [ -z $1 ] ; then
+ echo "usage: $0 <dirname>"
+ elif [ -d $1 ] ; then
+ echo "directory $1 already exists"
+ else
+ autoconf2.13
+ mkdir $1
+ cd $1
+ CC="clang" \
+ CXX="clang++" \
+ CFLAGS="-fsanitize=address" \
+ CXXFLAGS="-fsanitize=address" \
+ LDFLAGS="-fsanitize=address" \
+ ../configure --enable-debug --enable-optimize --enable-address-sanitizer --disable-jemalloc
+ fi
+
+Getting Symbols in Address Sanitizer Traces
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, ASan traces are unsymbolized and only print the
+binary/library and a memory offset instead. In order to get more useful
+traces, containing symbols, there are two approaches.
+
+Using the LLVM Symbolizer (recommended)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+LLVM ships with a symbolizer binary that ASan will readily use to
+immediately output symbolized traces. To use it, just set the
+environment variable ``ASAN_SYMBOLIZER_PATH`` to reflect the location of
+your ``llvm-symbolizer`` binary, before running the process. This
+program is usually included in an LLVM distribution. Stacks without
+symbols can also be post-processed, see below.
+
+.. warning::
+
+ .. note::
+
+ **Warning:** On OS X, the content sandbox prevents the symbolizer
+ from running. To use llvm-symbolizer on ASan output from a
+ content process, the content sandbox must be disabled. This can be
+ done by setting ``MOZ_DISABLE_CONTENT_SANDBOX=1`` in your run
+ environment. Setting this in .mozconfig has no effect.
+
+
+Post-Processing Traces with asan_symbolize.py
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Instead of using the llvm-symbolizer binary, you can also pipe the
+output through the ``asan_symbolize.py`` script, shipped with LLVM
+(``$LLVM_HOME/projects/compiler-rt/lib/asan/scripts/asan_symbolize.py``),
+often included in LLVM distributions. The disadvantage is that the
+script will need to use ``addr2line`` to get the symbols, which means
+that every library will have to be loaded into memory
+(including``libxul``, which takes a bit).
+
+However, in certain situations it makes sense to use this script. For
+example, if you have/received an unsymbolized trace, then you can still
+use the script to turn it into a symbolized trace, given that you can
+get the original binaries that produced the unsymbolized trace. In order
+for the script to work in such cases, you need to ensure that the paths
+in the trace point to the actual binaries, or change the paths
+accordingly.
+
+Since the output of the ``asan_symbolize.py`` script is still mangled,
+you might want to pipe the output also through ``c++filt`` afterwards.
+
+Troubleshooting / Known problems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Cannot specify -o when generating multiple output files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you get the error
+"``cannot specify -o when generating multiple output files"`` from
+clang, disable ``elf-hack`` in your ``mozconfig`` to work around the
+issue:
+
+::
+
+ ac_add_options --disable-elf-hack
+
+Optimized build
+^^^^^^^^^^^^^^^
+
+Since `an issue with -O2/-Os and
+ASan <https://github.com/google/sanitizers/issues/20>`__
+has been resolved, the regular optimizations used by Firefox should work
+without any problems. The optimized build has only a barely noticeable
+speed penalty and seems to be even faster than regular debug builds.
+
+No "AddressSanitizer: **libc** interceptors initialized" shows after running ./mach run
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+ $ ASAN_OPTIONS=verbosity=2 ./mach run
+
+Use the above command instead
+
+"An admin user name and password" is required to enter Developer Mode
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Please enable **Developer** **mode** by:
+
+::
+
+ $ /usr/sbin/DevToolsSecurity -enable
+ Developer mode is now enabled.
+
+Debugging issues that ASan finds
+--------------------------------
+
+When ASan discovers an issue it will simply print an error message and
+exit the app. To stop the app in a debugger before ASan exits it, set a
+breakpoint on ``__asan::ReportGenericError``. For more info on using
+ASan and debugging issues that it uncovers, see the page `Address
+sanitizer and a
+debugger <https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger>`__
+page on the upstream wiki.
+
+``__asan_describe_address(pointer)`` issued at the debugger prompt or
+even directly in the code allows outputting lots of information about
+this memory address (thread and stack of allocation, of deallocation,
+whether or not it is a bit outside a known buffer, thread and stack of
+allocation of this buffer, etc.). This can be useful to understand where
+some buffer that is not aligned was allocated, when doing SIMD work, for
+example.
+
+`rr <https://rr-project.org/>`__ (Linux x86 only) works great with ASan
+and combined, this combo allows doing some very powerful debugging
+strategies.
+
+LeakSanitizer
+-------------
+
+LeakSanitizer (LSan) is a special execution mode for regular ASan. It
+takes advantage of how ASan tracks the set of live blocks at any given
+point to print out the allocation stack of any block that is still alive
+at shutdown, but is not reachable from the stack, according to a
+conservative scan. This is very useful for detecting leaks of things
+such as ``char*`` that do not participate in the usual Gecko shutdown
+leak detection. LSan is supported on x86_64 Linux and OS X.
+
+LSan is enabled by default in ASan builds, as of more recent versions of
+Clang. To make an ASan build not run LSan, set the environment variable
+``ASAN_OPTIONS`` to ``detect_leaks=0`` (or add it as an entry to a
+``:``-separated list if it is already set to something). If you want to
+enable it when it is not for some reason, set it to 1 instead of 0. If
+LSan is enabled and you are using a non-debug build, you will also want
+to set the environment variable ``MOZ_CC_RUN_DURING_SHUTDOWN=1``, to
+ensure that we run shutdown GCs and CCs to avoid spurious leaks.
+
+If an object that is reported by LSan is intentionally never freed, a
+symbol can be added to ``build/sanitizers/lsan_suppressions.txt`` to get
+LSan to ignore it.
+
+For some more information on LSan, see the `Leak Sanitizer wiki
+page <https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer>`__.
+
+
+A `meta bug called lsan <https://bugzilla.mozilla.org/show_bug.cgi?id=lsan>`__
+is maintained to keep track of all the bugs found with LSan.
+
+
+
+Frequently Asked Questions about ASan
+-------------------------------------
+
+How does ASan work exactly?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+More information on how ASan works can be found on `the Address Sanitizer wiki <https://github.com/google/sanitizers/wiki/AddressSanitizer>`__.
diff --git a/tools/sanitizer/docs/asan_nightly.rst b/tools/sanitizer/docs/asan_nightly.rst
new file mode 100644
index 0000000000..790def3931
--- /dev/null
+++ b/tools/sanitizer/docs/asan_nightly.rst
@@ -0,0 +1,204 @@
+ASan Nightly
+============
+
+The **ASan Nightly Project** involves building a Firefox Nightly browser
+with the popular
+`AddressSanitizer <https://github.com/google/sanitizers/wiki/AddressSanitizer>`__
+tool and enhancing it with remote crash reporting capabilities for any
+errors detected.
+
+The purpose of the project is to find subtle memory corruptions
+occurring during regular browsing that would either not crash at all or
+crash in a way that we cannot figure out what the exact problem is just
+from the crash dump. We have a lot of inactionable crash reports and
+AddressSanitizer traces are usually a lot more actionable on their own
+(especially use-after-free traces). Part of this project is to figure
+out if and how many actionable crash reports ASan can give us just by
+surfing around. The success of the project of course also depends on the
+number of participants.
+
+You can download the latest build using one of the links below. The
+builds are self-updating daily like regular nightly builds (like with
+regular builds, you can go to *"Help"* → *"About Nightly"* to force an
+update check or confirm that you run the latest version).
+
+.. note::
+
+ If you came here looking for regular ASan builds (e.g. for fuzzing or
+ as a developer to reproduce a crash), you should probably go to the
+ :ref:`Address Sanitizer` doc instead.
+
+.. _Requirements:
+
+Requirements
+~~~~~~~~~~~~
+
+Current requirements are:
+
+- Windows or Linux-based Operating System
+- 16 GB of RAM recommended
+- Special ASan Nightly Firefox Build
+
+ - `Linux
+ Download <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.shippable.latest.firefox.linux64-asan-reporter-opt/artifacts/public/build/target.tar.bz2>`__
+ - `Windows
+ Download <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.shippable.latest.firefox.win64-asan-reporter-shippable-repackage-signing/artifacts/public/build/target.installer.exe>`__
+
+If you are already using regular Nightly, it should be safe to share the
+profile with the regular Nightly instance. If you normally use a beta or
+release build (and you would like to be able to switch back to these),
+you should consider using a second profile.
+
+.. warning::
+
+ **Windows Users:** Please note that the Windows builds currently show
+ an error during setup (see "*Known Issues*" section below), but
+ installation works nonetheless. We are working on the problem.
+
+.. note::
+
+ If you run in an environment with any sorts of additional security
+ restrictions (e.g. custom process sandboxing), please make sure that
+ your /tmp directory is writable and the shipped ``llvm-symbolizer``
+ binary is executable from within the Firefox process.
+
+Preferences
+~~~~~~~~~~~
+
+If you wish for your crash report to be identifiable, you can go to
+``about:config`` and set the **``asanreporter.clientid``** to your
+**valid email address**. This isn't mandatory, you can of course report
+crash traces anonymously. If you decide to send reports with your email
+address and you have a Bugzilla account, consider using the same email
+as your Bugzilla account uses. We will then Cc you on any bugs filed
+from your crash reports. If your email does not belong to a Bugzilla
+account, then we will not publish it but only use it to resolve
+questions about your crash reports.
+
+.. note::
+
+ Setting this preference helps us to get back to you in case we have
+ questions about your setup/OS. Please consider using it so we can get
+ back to you if necessary.
+
+Bug Bounty Program
+~~~~~~~~~~~~~~~~~~
+
+As a special reward for participating in the program, we decided to
+treat all submitted reports as if they were filed directly in Bugzilla.
+This means that reports that
+
+- indicate a security issue of critical or high rating
+- **and** that can be fixed by our developers
+
+are eligible for a bug bounty according to our `client bug bounty
+program
+rules <https://www.mozilla.org/security/client-bug-bounty/>`__. As
+the report will usually not include any steps to reproduce or a test
+case, it will most likely receive a lower-end bounty. Like with regular
+bug reports, we would typically reward the first (identifable) report of
+an issue.
+
+.. warning::
+
+ If you would like to participate in the bounty program, make sure you
+ set your **``asanreporter.clientid``** preference as specified above.
+ We cannot reward any reports that are submitted with no email
+ address.
+
+
+Known Issues
+~~~~~~~~~~~~
+
+This section lists all currently known limitations of the ASan Nightly
+builds that are considered bugs.
+
+- [STRIKEOUT:Flash is currently not working]
+- `Bug
+ 1477490 <https://bugzilla.mozilla.org/show_bug.cgi?id=1477490>`__\ [STRIKEOUT:-
+ Windows: Stack instrumentation disabled due to false positives]
+- `Bug
+ 1478096 <https://bugzilla.mozilla.org/show_bug.cgi?id=1478096>`__ -
+ **Windows:** Error during install with maintenanceservice_tmp.exe
+- It has been reported that ASan Nightly performance is particularly
+ bad if you run on a screen with 120hz refresh rate. Switching to 60hz
+ should improve performance drastically.
+
+Note that these bugs are **specific** to ASan Nightly as listed in the
+`tracking bug dependency
+list <https://bugzilla.mozilla.org/showdependencytree.cgi?id=1386297&hide_resolved=0>`__.
+For the full list of bugs found by this project, see `this
+list <https://bugzilla.mozilla.org/showdependencytree.cgi?id=1479399&hide_resolved=0>`__
+instead and note that some bugs might not be shown because they are
+security bugs.
+
+If you encounter a bug not listed here, please file a bug at
+`bugzilla.mozilla.org <https://bugzilla.mozilla.org/>`__ or send an
+email to
+`choller@mozilla.com <mailto:choller@mozilla.com?subject=%5BASan%20Nightly%20Project%5D%5BBug%20Report%5D>`__.
+When filing a bug, it greatly helps if you Cc that email address and
+make the bug block `bug
+1386297 <https://bugzilla.mozilla.org/show_bug.cgi?id=1386297>`__.
+
+FAQ
+~~~
+
+What additional data is collected?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The project only collects ASan traces and (if you set it in the
+preferences) your email address. We don't collect any other browser
+data, in particular not the sites you were visiting or page contents. It
+is really just crash traces submitted to a remote location.
+
+.. note::
+
+ The ASan Nightly browser also still has all the data collection
+ capabilities of a regular Nightly browser. The answer above only
+ refers to what this project collects **in addition** to what the
+ regular Nightly browser can collect.
+
+What's the performance impact?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ASan Nightly build only comes with a slight slowdown at startup and
+browsing, sometimes it is not even noticeable. The RAM consumption
+however is much higher than with a regular build. Be prepared to restart
+your browser sometimes, especially if you use a lot of tabs at once.
+Also, the updates are larger than the regular ones, so download times
+for updates will be higher, especially if you have a slower internet
+connection.
+
+.. warning::
+
+ If you experience performance issues, see also the *"Known Issues"*
+ section above, in particular the problem about screen refresh rate
+ slowing down Firefox.
+
+What about stability?
+^^^^^^^^^^^^^^^^^^^^^
+
+The browser is as stable as a regular Nightly build. Various people have
+been surfing around with it for their daily work for weeks now and we
+have barely received any crash reports.
+
+How do I confirm that I'm running the correct build?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you open ``about:config`` and type *"asanreporter"* into the search
+field, you should see an entry called ``asanreporter.apiurl`` associated
+with a URL. Do not modify this value.
+
+.. warning::
+
+ Since Firefox 64, the *"ASan Crash Reporter"*  feature is no longer
+ listed in ``about:support``
+
+Will there be support for Mac?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We are working on support for Mac, but it might take longer because we
+have no ASan CI coverage on Mac due to hardware constraints. If you work
+on Release Engineering and would like to help make e.g. Mac happen
+earlier, feel free to `contact
+me <mailto:choller@mozilla.com?subject=%5BASan%20Nightly%20Project%5D%20>`__.
diff --git a/tools/sanitizer/docs/index.rst b/tools/sanitizer/docs/index.rst
new file mode 100644
index 0000000000..3a1dc0806e
--- /dev/null
+++ b/tools/sanitizer/docs/index.rst
@@ -0,0 +1,33 @@
+Sanitizer
+=========
+
+.. toctree::
+ :maxdepth: 1
+ :hidden:
+ :glob:
+
+ *
+
+**Address Sanitizer**
+
+Address Sanitizer (ASan) is a fast memory error detector that detects use-after-free and out-of-bound bugs in C/C++ programs. It uses a compile-time instrumentation to check all reads and writes during the execution. In addition, the runtime part replaces the malloc and free functions to check dynamically allocated memory.
+
+:ref:`More information <Address Sanitizer>`
+
+**Thread Sanitizer**
+
+Thread Sanitizer (TSan) is a fast data race detector for C/C++ programs. It uses a compile-time instrumentation to check all non-race-free memory access at runtime. Unlike other tools, it understands compiler-builtin atomics and synchronization and therefore provides very accurate results with no false positives (except if unsupported synchronization primitives like inline assembly or memory fences are used).
+
+:ref:`More information <Thread Sanitizer>`
+
+**Memory Sanitizer**
+
+Memory Sanitizer (MSan) is a fast detector used for uninitialized memory in C/C++ programs. It uses a compile-time instrumentation to ensure that all memory access at runtime uses only memory that has been initialized.
+
+:ref:`More information <Memory Sanitizer>`
+
+**ASan Nightly Project**
+
+The ASan Nightly Project involves building a Firefox Nightly browser with the popular AddressSanitizer tool and enhancing it with remote crash reporting capabilities for any errors detected.
+
+:ref:`More information <ASan Nightly>`
diff --git a/tools/sanitizer/docs/memory_sanitizer.rst b/tools/sanitizer/docs/memory_sanitizer.rst
new file mode 100644
index 0000000000..161826fbc2
--- /dev/null
+++ b/tools/sanitizer/docs/memory_sanitizer.rst
@@ -0,0 +1,173 @@
+Memory Sanitizer
+================
+
++--------------------------------------------------------------------+
+| This page is an import from MDN and the contents might be outdated |
++--------------------------------------------------------------------+
+
+What is Memory Sanitizer?
+-------------------------
+
+Memory Sanitizer (MSan) is a fast detector used for uninitialized memory
+in C/C++ programs. It uses a compile-time instrumentation to ensure that
+all memory access at runtime uses only memory that has been initialized.
+Unlike most other sanitizers, MSan can easily cause false positives if
+not all libraries are instrumented. This happens because MSan is
+not able to observe memory initialization in uninstrumented libraries.
+More information on MSan can be found on `the Memory Sanitizer
+wiki <https://github.com/google/sanitizers/wiki/MemorySanitizer>`__.
+
+Public Builds
+-------------
+
+**Note:** No public builds are available at this time yet.
+
+Manual Build
+------------
+
+Build prerequisites
+~~~~~~~~~~~~~~~~~~~
+
+**Note:** MemorySanitizer requires **64-bit Linux** to work. Other
+platforms/operating systems are not supported.
+
+LLVM/Clang
+^^^^^^^^^^
+
+The MSan instrumentation is implemented as an LLVM pass and integrated
+into Clang. As MSan is one of the newer sanitizers, we recommend using a
+recent Clang version, such as Clang 3.7+.
+
+You can find precompiled binaries for LLVM/Clang on `the LLVM releases
+page <https://releases.llvm.org/download.html>`__.
+
+Building Firefox
+~~~~~~~~~~~~~~~~
+
+.. warning::
+
+ **Warning: Running Firefox with MemorySanitizer would require all
+ external dependencies to be built with MemorySanitizer as well. To
+ our knowledge, this has never been attempted yet, so the build
+ configuration provided here is untested and without an appropriately
+ instrumented userland, it will cause false positives.**
+
+Getting the source
+^^^^^^^^^^^^^^^^^^
+
+If you don't have a source code repository clone yet, you need to :ref:`get
+yourself a clone of Mozilla-central <Mercurial Overview>`.
+
+Adjusting the build configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Create the build configuration file ``.mozconfig`` with the following
+content in your Mozilla-central directory:
+
+.. code::
+
+ mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-msan
+
+ # Enable LLVM specific code and build workarounds
+ ac_add_options --enable-memory-sanitizer
+ # If clang is already in your $PATH, then these can simply be:
+ # export CC=clang
+ # export CXX=clang++
+ export CC="/path/to/clang"
+ export CXX="/path/to/clang++"
+
+ # llvm-symbolizer displays much more complete backtraces when data races are detected.
+ # If it's not already in your $PATH, then uncomment this next line:
+ #export LLVM_SYMBOLIZER="/path/to/llvm-symbolizer"
+
+ # Add MSan to our compiler flags
+ export CFLAGS="-fsanitize=memory"
+ export CXXFLAGS="-fsanitize=memory"
+
+ # Additionally, we need the MSan flag during linking. Normally, our C/CXXFLAGS would
+ # be used during linking as well but there is at least one place in our build where
+ # our CFLAGS are not added during linking.
+ # Note: The use of this flag causes Clang to automatically link the MSan runtime :)
+ export LDFLAGS="-fsanitize=memory"
+
+ # These three are required by MSan
+ ac_add_options --disable-jemalloc
+ ac_add_options --disable-crashreporter
+ ac_add_options --disable-elf-hack
+
+ # Keep symbols to symbolize MSan traces
+ export MOZ_DEBUG_SYMBOLS=1
+ ac_add_options --enable-debug-symbols
+ ac_add_options --disable-install-strip
+
+ # Settings for an opt build
+ ac_add_options --enable-optimize="-O2 -gline-tables-only"
+ ac_add_options --disable-debug
+
+Starting the build process
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now you start the build process using the regular ``make -f client.mk``
+command.
+
+Starting Firefox
+^^^^^^^^^^^^^^^^
+
+After the build has completed, you can start Firefox from the ``objdir``
+as usual.
+
+Building the JavaScript shell
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Note:** Unlike Firefox itself, the JavaScript shell does **not**
+require an instrumented userland. Calls to external libraries like
+zlib are handled with special annotations inside the engine.
+
+.. warning::
+
+ **Warning: Certain technologies used inside the JavaScript engine are
+ incompatible with MSan and must be disabled at runtime to prevent
+ false positives. This includes the JITs and asm.js. Therefore always
+ make sure to run with
+ ``--no-ion --no-baseline --no-asmjs --no-native-regexp``.**
+
+If you want to build only the JavaScript shell instead of doing a full
+Firefox build, the build script below will probably help you to do so.
+Before using it, you must, of course, adjust the path name for
+``LLVM_ROOT`` to match your setup. Once you have adjusted everything,
+execute this script in the ``js/src/`` subdirectory and pass a directory
+name as the first parameter. The build will then be created in a new
+subdirectory with that name.
+
+.. code::
+
+ #! /bin/sh
+
+ if [ -z $1 ] ; then
+ echo "usage: $0 <dirname>"
+ elif [ -d $1 ] ; then
+ echo "directory $1 already exists"
+ else
+ autoconf2.13
+ mkdir $1
+ cd $1
+ LLVM_ROOT="/path/to/llvm"
+ CC="$LLVM_ROOT/build/bin/clang" \
+ CXX="$LLVM_ROOT/build/bin/clang++" \
+ CFLAGS="-fsanitize=memory" \
+ CXXFLAGS="-fsanitize=memory" \
+ LDFLAGS=""-fsanitize=memory" \
+ ../configure --enable-debug --enable-optimize --enable-memory-sanitizer --disable-jemalloc --enable-posix-nspr-emulation
+ make -j 8
+ fi
+
+Using LLVM Symbolizer for faster/better traces
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, MSan traces are not symbolized.
+
+LLVM ships with the symbolizer binary ``llvm-symbolize`` that MSan will
+readily use to immediately output symbolized traces if the program is
+found on the ``PATH``. If your ``llvm-symbolizer`` lives outside the
+``PATH``, you can set the ``MSAN_SYMBOLIZER_PATH`` environment variable
+to point to your symbolizer binary.
diff --git a/tools/sanitizer/docs/tsan.rst b/tools/sanitizer/docs/tsan.rst
new file mode 100644
index 0000000000..77fb6c89d7
--- /dev/null
+++ b/tools/sanitizer/docs/tsan.rst
@@ -0,0 +1,327 @@
+Thread Sanitizer
+================
+
+What is Thread Sanitizer?
+--------------------------
+
+Thread Sanitizer (TSan) is a fast data race detector for C/C++ and Rust
+programs. It uses a compile-time instrumentation to check all non-race-free
+memory access at runtime. Unlike other tools, it understands compiler-builtin
+atomics and synchronization and therefore provides very accurate results
+with no false positives (except if unsupported synchronization primitives
+like inline assembly or memory fences are used). More information on how
+TSan works can be found on `the Thread Sanitizer wiki <https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm>`__.
+
+A `meta bug called tsan <https://bugzilla.mozilla.org/show_bug.cgi?id=tsan>`__
+is maintained to keep track of all the bugs found with TSan.
+
+A `blog post on hacks.mozilla.org <https://hacks.mozilla.org/2021/04/eliminating-data-races-in-firefox-a-technical-report/>`__ describes this project.
+
+Note that unlike other sanitizers, TSan is currently **only supported on Linux**.
+
+Downloading artifact builds
+---------------------------
+
+The easiest way to get Firefox builds with Thread Sanitizer is to download a
+continuous integration TSan build of mozilla-central (updated at least daily):
+
+- mozilla-central optimized builds:
+ `linux <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-tsan-opt/artifacts/public/build/target.tar.bz2>`__
+
+The fuzzing team also offers a tool called ``fuzzfetch`` to download this and many
+other CI builds. It makes downloading and unpacking these builds much easier and
+can be used not just for fuzzing but for all purposes that require a CI build download.
+
+You can install ``fuzzfetch`` from
+`Github <https://github.com/MozillaSecurity/fuzzfetch>`__ or
+`via pip <https://pypi.org/project/fuzzfetch/>`__.
+
+Afterwards, you can run
+
+::
+
+ $ python -m fuzzfetch --tsan -n firefox-tsan
+
+to get the build mentioned above unpacked into a directory called ``firefox-tsan``.
+
+Creating Try builds
+-------------------
+
+If for some reason you can't use the pre-built binaries mentioned in the
+previous section (e.g. you need to test a patch), you can either build
+Firefox yourself (see the following section) or use the :ref:`try server <Pushing to Try>`
+to create the customized build for you. Pushing to try requires L1 commit
+access. If you don't have this access yet you can request access (see
+`Becoming A Mozilla
+Committer <https://www.mozilla.org/about/governance/policies/commit/>`__
+and `Mozilla Commit Access
+Policy <https://www.mozilla.org/about/governance/policies/commit/access-policy/>`__
+for the requirements).
+
+Using ``mach try fuzzy --full`` you can select the ``build-linux64-tsan/opt`` job
+and related tests (if required).
+
+Creating local builds on Linux
+------------------------------
+
+Build prerequisites
+~~~~~~~~~~~~~~~~~~~
+
+LLVM/Clang/Rust
+^^^^^^^^^^^^^^^
+
+The TSan instrumentation is implemented as an LLVM pass and integrated
+into Clang. We strongly recommend that you use the Clang version supplied
+as part of the ``mach bootstrap`` process, as we backported several required
+fixes for TSan on Firefox.
+
+Sanitizer support in Rust is genuinely experimental,
+so our build system only works with a specially patched version of Rust
+that we build in our CI. To install that specific version (or update to a newer
+version), run the following in the root of your mozilla-central checkout:
+
+::
+
+ ./mach artifact toolchain --from-build linux64-rust-dev
+ rm -rf ~/.mozbuild/rustc-sanitizers
+ mv rustc ~/.mozbuild/rustc-sanitizers
+ rustup toolchain link gecko-sanitizers ~/.mozbuild/rustc-sanitizers
+ rustup override set gecko-sanitizers
+
+``mach artifact`` will always download the ``linux64-rust-dev`` toolchain associated
+with the current mozilla central commit you have checked out. The toolchain should
+mostly behave like a normal rust nightly but we don't recommend using it for anything
+other than building gecko, just in case. Also note that
+``~/.mozbuild/rustc-sanitizers`` is just a reasonable default location -- feel
+free to "install" the toolchain wherever you please.
+
+Building Firefox
+~~~~~~~~~~~~~~~~
+
+Getting the source
+^^^^^^^^^^^^^^^^^^
+
+Using that or any later revision, all you need to do is to :ref:`get yourself
+a clone of mozilla-central <Mercurial overview>`.
+
+Adjusting the build configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Create the build configuration file ``mozconfig`` with the following
+content in your mozilla-central directory:
+
+::
+
+ # Combined .mozconfig file for TSan on Linux+Mac
+
+ mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-tsan
+
+ # Enable ASan specific code and build workarounds
+ ac_add_options --enable-thread-sanitizer
+
+ # This ensures that we also instrument Rust code.
+ export RUSTFLAGS="-Zsanitizer=thread"
+
+ # rustfmt is currently missing in Rust nightly
+ unset RUSTFMT
+
+ # Current Rust Nightly has warnings
+ ac_add_options --disable-warnings-as-errors
+
+ # These are required by TSan
+ ac_add_options --disable-jemalloc
+ ac_add_options --disable-crashreporter
+ ac_add_options --disable-elf-hack
+ ac_add_options --disable-profiling
+
+ # The Thread Sanitizer is not compatible with sandboxing
+ # (see bug 1182565)
+ ac_add_options --disable-sandbox
+
+ # Keep symbols to symbolize TSan traces later
+ export MOZ_DEBUG_SYMBOLS=1
+ ac_add_options --enable-debug-symbols
+ ac_add_options --disable-install-strip
+
+ # Settings for an opt build (preferred)
+ # The -gline-tables-only ensures that all the necessary debug information for ASan
+ # is present, but the rest is stripped so the resulting binaries are smaller.
+ ac_add_options --enable-optimize="-O2 -gline-tables-only"
+ ac_add_options --disable-debug
+
+ # Settings for a debug+opt build
+ #ac_add_options --enable-optimize
+ #ac_add_options --enable-debug
+
+
+Starting the build process
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now you start the build process using the regular ``./mach build``
+command.
+
+Starting Firefox
+^^^^^^^^^^^^^^^^
+
+After the build has completed, ``./mach run`` with the usual options for
+running in a debugger (``gdb``, ``lldb``, ``rr``, etc.) work fine, as do
+the ``--disable-e10s`` and other options.
+
+Building only the JavaScript shell
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to build only the JavaScript shell instead of doing a full
+Firefox build, the build script below will probably help you to do so.
+Execute this script in the ``js/src/`` subdirectory and pass a directory
+name as the first parameter. The build will then be created in a new
+subdirectory with that name.
+
+::
+
+ #! /bin/sh
+
+ if [ -z $1 ] ; then
+ echo "usage: $0 <dirname>"
+ elif [ -d $1 ] ; then
+ echo "directory $1 already exists"
+ else
+ autoconf2.13
+ mkdir $1
+ cd $1
+ CC="/path/to/mozbuild/clang" \
+ CXX="/path/to/mozbuild/clang++" \
+ ../configure --disable-debug --enable-optimize="-O2 -gline-tables-only" --enable-thread-sanitizer --disable-jemalloc
+ fi
+
+Thread Sanitizer and Symbols
+----------------------------
+
+Unlike Address Sanitizer, TSan requires in-process symbolizing to work
+properly in the first place, as any kind of runtime suppressions will
+otherwise not work.
+
+Hence, it is required that you have a copy of ``llvm-symbolizer`` either
+in your ``PATH`` or pointed to by the ``TSAN_SYMBOLIZER_PATH`` environment
+variable. This binary is included in your local mozbuild directory, obtained
+by ``./mach bootstrap``.
+
+
+Runtime Suppressions
+--------------------
+
+TSan has the ability to suppress race reports at runtime. This can be used to
+silence a race while a fix is developed as well as to permanently silence a
+(benign) race that cannot be fixed.
+
+.. warning::
+ **Warning**: Many races *look* benign but are indeed not. Please read
+ the :ref:`FAQ section <Frequently Asked Questions about TSan>` carefully
+ and think twice before attempting to suppress a race.
+
+The runtime Suppression list is directly baked into Firefox at compile-time and
+located at `mozglue/build/TsanOptions.cpp <https://searchfox.org/mozilla-central/source/mozglue/build/TsanOptions.cpp>`__.
+
+.. warning::
+ **Important**: When adding a suppression, always make sure to include
+ the bug number. If the suppression is supposed to be permanent, please
+ add the string ``permanent`` in the same line as the bug number.
+
+.. warning::
+ **Important**: When adding a suppression for a *data race*, always make
+ sure to include a stack frame from **each** of the two race stacks.
+ Adding only one suppression for one stack can cause intermittent failures
+ that are later on hard to track. One exception to this rule is when suppressing
+ races on global variables. In that case, a single race entry with the name of
+ the variable is sufficient.
+
+Troubleshooting / Known Problems
+--------------------------------
+
+Known Sources of False Positives
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+TSan has a number of things that can cause false positives, namely:
+
+ * The use of memory fences (e.g. Rust Arc)
+ * The use of inline assembly for synchronization
+ * Uninstrumented code (e.g. external libraries) using compiler-builtins for synchronization
+ * A lock order inversion involving only a single thread can cause a false positive deadlock
+ report (see also https://github.com/google/sanitizers/issues/488).
+
+If none of these four items are involved, you should *never* assume that TSan is reporting
+a false positive to you without consulting TSan peers. It is very easy to misjudge a race
+to be a false positive because races can be highly complex and totally non-obvious due to
+compiler optimizations and the nature of parallel code.
+
+Intermittent Broken Stacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you intermittently see race reports where one stack is missing with a ``failed to restore the stack``
+message, this can indicate that a suppression is partially covering the race you are seeing.
+
+Any race where only one of the two stacks is matched by a runtime suppression will show up
+if that particular stack fails to symbolize for some reason. The usual solution is to search
+the suppressions for potential candidates and disable them temporarily to check if your race
+report now becomes mostly consistent.
+
+However, there are other reasons for broken TSan stacks, in particular if they are not intermittent.
+See also the ``history_size`` parameter in the `TSan flags <https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags>`__.
+
+Intermittent Race Reports
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Unfortunately, the TSan algorithm does not guarantee, that a race is detected 100% of the
+time. Intermittent failures with TSan are (to a certain degree) to be expected and the races
+involved should be filed and fixed to solve the problem.
+
+.. _Frequently Asked Questions about TSan:
+
+Frequently Asked Questions about TSan
+-------------------------------------
+
+Why fix data races?
+~~~~~~~~~~~~~~~~~~~
+
+Data races are undefined behavior and can cause crashes as well as correctness issues.
+Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.
+
+At Mozilla, we have already seen several dangerous races, causing random
+`use-after-free crashes <https://bugzilla.mozilla.org/show_bug.cgi?id=1580288>`__,
+`intermittent test failures <https://bugzilla.mozilla.org/show_bug.cgi?id=1602009>`__,
+`hangs <https://bugzilla.mozilla.org/show_bug.cgi?id=1607008>`__,
+`performance issues <https://bugzilla.mozilla.org/show_bug.cgi?id=1615045>`__ and
+`intermittent asserts <https://bugzilla.mozilla.org/show_bug.cgi?id=1601940>`__. Such problems do
+not only decrease the quality of our code and user experience, but they also waste countless hours
+of developer time.
+
+Since it is very hard to judge if a particular race could cause such a situation, we
+have decided to fix all data races wherever possible, since doing so is often cheaper
+than analyzing a race.
+
+My race is benign, can we ignore it?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While it is possible to add a runtime suppression to ignore the race, we *strongly* encourage
+you to not do so, for two reasons:
+
+ 1. Each suppressed race decreases the overall performance of the TSan build, as the race
+ has to be symbolized each time when it occurs. Since TSan is already in itself a slow
+ build, we need to keep the amount of suppressed races as low as possible.
+
+ 2. Deciding if a race is truly benign is surprisingly hard. We recommend to read
+ `this blog post <http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong>`__
+ and `this paper <https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf>`
+ on the effects of seemingly benign races.
+
+Valid reasons to suppress a confirmed benign race include performance problems arising from
+fixing the race or cases where fixing the race would require an unreasonable amount of work.
+
+Note that the use of atomics usually does not have the bad performance impact that developers
+tend to associate with it. If you assume that e.g. using atomics for synchronization will
+cause performance regressions, we suggest to perform a benchmark to confirm this. In many
+cases, the difference is not measurable.
+
+How does TSan work exactly?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+More information on how TSan works can be found on `the Thread Sanitizer wiki <https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm>`__.