diff options
Diffstat (limited to 'tools/sanitizer/docs')
| -rw-r--r-- | tools/sanitizer/docs/asan.rst | 379 | ||||
| -rw-r--r-- | tools/sanitizer/docs/asan_nightly.rst | 204 | ||||
| -rw-r--r-- | tools/sanitizer/docs/index.rst | 33 | ||||
| -rw-r--r-- | tools/sanitizer/docs/memory_sanitizer.rst | 173 | ||||
| -rw-r--r-- | tools/sanitizer/docs/tsan.rst | 327 |
5 files changed, 1116 insertions, 0 deletions
diff --git a/tools/sanitizer/docs/asan.rst b/tools/sanitizer/docs/asan.rst new file mode 100644 index 0000000000..0b20aa0311 --- /dev/null +++ b/tools/sanitizer/docs/asan.rst @@ -0,0 +1,379 @@ +Address Sanitizer +================= + +What is Address Sanitizer? +-------------------------- + +Address Sanitizer (ASan) is a fast memory error detector that detects +use-after-free and out-of-bound bugs in C/C++ programs. It uses a +compile-time instrumentation to check all reads and writes during the +execution. In addition, the runtime part replaces the ``malloc`` and +``free`` functions to check dynamically allocated memory. More +information on how ASan works can be found on `the Address Sanitizer +wiki <https://github.com/google/sanitizers/wiki/AddressSanitizer>`__. + +A `meta bug called asan-maintenance <https://bugzilla.mozilla.org/show_bug.cgi?id=asan-maintenance>`__ +is maintained to keep track of all the bugs found with ASan. + +Downloading artifact builds +--------------------------- + +For Linux and Windows users, the easiest way to get Firefox builds with +Address Sanitizer is to download a continuous integration asan build of +mozilla-central (updated at least daily): + +- mozilla-central optimized builds: + `linux <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-asan-opt/artifacts/public/build/target.tar.bz2>`__ + \| + `windows <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.win64-asan-opt/artifacts/public/build/target.zip>`__ + (recommended for testing) +- mozilla-central debug builds: + `linux <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-asan-debug/artifacts/public/build/target.tar.bz2>`__ + \| + `windows <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.win64-asan-debug/artifacts/public/build/target.zip>`__ + (recommended for debugging if the optimized builds don't do the job) + +The fuzzing team also offers a tool called ``fuzzfetch`` to download these and many +other CI builds. It makes downloading and unpacking these builds much easier and +can be used not just for fuzzing but for all purposes that require a CI build download. + +You can install ``fuzzfetch`` from +`Github <https://github.com/MozillaSecurity/fuzzfetch>`__ or +`via pip <https://pypi.org/project/fuzzfetch/>`__. + +Afterwards, you can run e.g. + +:: + + $ python -m fuzzfetch --asan -n firefox-asan + +to get the optimized Linux ASan build mentioned above unpacked into a directory called ``firefox-asan``. +The ``--debug`` and ``--os`` switches can be used to get the other variants listed above. + +Creating Try builds +------------------- + +If for some reason you can't use the pre-built binaries mentioned in the +previous section (e.g. you want a non-Linux build or you need to test a +patch), you can either build Firefox yourself (see the following +section) or use the :ref:`try server <Pushing to Try>` to +create the customized build for you. Pushing to try requires L1 commit +access. If you don't have this access yet you can request access (see +`Becoming A Mozilla +Committer <https://www.mozilla.org/about/governance/policies/commit/>`__ +and `Mozilla Commit Access +Policy <https://www.mozilla.org/about/governance/policies/commit/access-policy/>`__ +for the requirements). + +The tree contains `several mozconfig files for creating asan +builds <https://searchfox.org/mozilla-central/search?q=&case=true&path=browser%2Fconfig%2Fmozconfigs%2F*%2F*asan*>`__ +(the "nightly-asan" files create release builds, whereas the +"debug-asan" files create debug+opt builds). For Linux builds, the +appropriate configuration file is used by the ``linux64-asan`` target. +If you want to create a macOS or Windows build, you'll need to copy the +appropriate configuration file over the regular debug configuration +before pushing to try. For example: + +:: + + cp browser/config/mozconfigs/macosx64/debug-asan browser/config/mozconfigs/macosx64/debug + +You can then `push to Try in the usual +way </tools/try/index.html#using-try>`__ +and, once the build is complete, download the appropriate build +artifact. + +Creating local builds on Windows +-------------------------------- + +On Windows, ASan is supported only in 64-bit builds. + +Run ``mach bootstrap`` to get an updated clang-cl in your +``~/.mozbuild`` directory, then use the following +:ref:`mozconfig <Configuring Build Options>`: + +:: + + ac_add_options --enable-address-sanitizer + ac_add_options --disable-jemalloc + + export LDFLAGS="clang_rt.asan_dynamic-x86_64.lib clang_rt.asan_dynamic_runtime_thunk-x86_64.lib" + CLANG_LIB_DIR="$(cd ~/.mozbuild/clang/lib/clang/*/lib/windows && pwd)" + export MOZ_CLANG_RT_ASAN_LIB_PATH="${CLANG_LIB_DIR}/clang_rt.asan_dynamic-x86_64.dll" + export LIB=$LIB:$CLANG_LIB_DIR + +If you launch an ASan build under WinDbg, you may see spurious +first-chance Access Violation exceptions. These come from ASan creating +shadow memory pages on demand, and can be ignored. Run ``sxi av`` to +ignore these exceptions. (You will still catch second-chance Access +Violation exceptions if you actually crash.) + +LeakSanitizer (LSan) is not supported on Windows. + +Creating local builds on Linux or Mac +------------------------------------- + +Build prerequisites +~~~~~~~~~~~~~~~~~~~ + +LLVM/Clang +^^^^^^^^^^ + +The ASan instrumentation is implemented as an LLVM pass and integrated +into Clang. Any clang version that is capable of compiling Firefox has +everything needed to do an ASAN build. + +Building Firefox +~~~~~~~~~~~~~~~~ + +Getting the source +^^^^^^^^^^^^^^^^^^ + +Using that or any later revision, all you need to do is to :ref:`get yourself +a clone of mozilla-central <Mercurial overview>`. + +Adjusting the build configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Create the build configuration file ``mozconfig`` with the following +content in your mozilla-central directory: + +:: + + # Combined .mozconfig file for ASan on Linux+Mac + + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-asan + + # Enable ASan specific code and build workarounds + ac_add_options --enable-address-sanitizer + + # These three are required by ASan + ac_add_options --disable-jemalloc + ac_add_options --disable-crashreporter + ac_add_options --disable-elf-hack + + # Keep symbols to symbolize ASan traces later + export MOZ_DEBUG_SYMBOLS=1 + ac_add_options --enable-debug-symbols + ac_add_options --disable-install-strip + + # Settings for an opt build (preferred) + # The -gline-tables-only ensures that all the necessary debug information for ASan + # is present, but the rest is stripped so the resulting binaries are smaller. + ac_add_options --enable-optimize="-O2 -gline-tables-only" + ac_add_options --disable-debug + + # Settings for a debug+opt build + #ac_add_options --enable-optimize + #ac_add_options --enable-debug + + # MacOSX only: Uncomment and adjust this path to match your SDK + # ac_add_options --with-macos-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk + +You may also need this, as seen in +``browser/config/mozconfigs/linux64/nightly-asan`` (the config file used +for Address Sanitizer builds used for automated testing): + +:: + + # ASan specific options on Linux + ac_add_options --enable-valgrind + +Starting the build process +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now you start the build process using the regular ``./mach build`` +command. + +Starting Firefox +^^^^^^^^^^^^^^^^ + +After the build has completed, ``./mach run`` with the usual options for +running in a debugger (``gdb``, ``lldb``, ``rr``, etc.) work fine, as do +the ``--disable-e10s`` and other options. + +Building only the JavaScript shell +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you want to build only the JavaScript shell instead of doing a full +Firefox build, the build script below will probably help you to do so. +Execute this script in the ``js/src/`` subdirectory and pass a directory +name as the first parameter. The build will then be created in a new +subdirectory with that name. + +:: + + #! /bin/sh + + if [ -z $1 ] ; then + echo "usage: $0 <dirname>" + elif [ -d $1 ] ; then + echo "directory $1 already exists" + else + autoconf2.13 + mkdir $1 + cd $1 + CC="clang" \ + CXX="clang++" \ + CFLAGS="-fsanitize=address" \ + CXXFLAGS="-fsanitize=address" \ + LDFLAGS="-fsanitize=address" \ + ../configure --enable-debug --enable-optimize --enable-address-sanitizer --disable-jemalloc + fi + +Getting Symbols in Address Sanitizer Traces +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By default, ASan traces are unsymbolized and only print the +binary/library and a memory offset instead. In order to get more useful +traces, containing symbols, there are two approaches. + +Using the LLVM Symbolizer (recommended) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +LLVM ships with a symbolizer binary that ASan will readily use to +immediately output symbolized traces. To use it, just set the +environment variable ``ASAN_SYMBOLIZER_PATH`` to reflect the location of +your ``llvm-symbolizer`` binary, before running the process. This +program is usually included in an LLVM distribution. Stacks without +symbols can also be post-processed, see below. + +.. warning:: + + .. note:: + + **Warning:** On OS X, the content sandbox prevents the symbolizer + from running. To use llvm-symbolizer on ASan output from a + content process, the content sandbox must be disabled. This can be + done by setting ``MOZ_DISABLE_CONTENT_SANDBOX=1`` in your run + environment. Setting this in .mozconfig has no effect. + + +Post-Processing Traces with asan_symbolize.py +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Instead of using the llvm-symbolizer binary, you can also pipe the +output through the ``asan_symbolize.py`` script, shipped with LLVM +(``$LLVM_HOME/projects/compiler-rt/lib/asan/scripts/asan_symbolize.py``), +often included in LLVM distributions. The disadvantage is that the +script will need to use ``addr2line`` to get the symbols, which means +that every library will have to be loaded into memory +(including``libxul``, which takes a bit). + +However, in certain situations it makes sense to use this script. For +example, if you have/received an unsymbolized trace, then you can still +use the script to turn it into a symbolized trace, given that you can +get the original binaries that produced the unsymbolized trace. In order +for the script to work in such cases, you need to ensure that the paths +in the trace point to the actual binaries, or change the paths +accordingly. + +Since the output of the ``asan_symbolize.py`` script is still mangled, +you might want to pipe the output also through ``c++filt`` afterwards. + +Troubleshooting / Known problems +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Cannot specify -o when generating multiple output files +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you get the error +"``cannot specify -o when generating multiple output files"`` from +clang, disable ``elf-hack`` in your ``mozconfig`` to work around the +issue: + +:: + + ac_add_options --disable-elf-hack + +Optimized build +^^^^^^^^^^^^^^^ + +Since `an issue with -O2/-Os and +ASan <https://github.com/google/sanitizers/issues/20>`__ +has been resolved, the regular optimizations used by Firefox should work +without any problems. The optimized build has only a barely noticeable +speed penalty and seems to be even faster than regular debug builds. + +No "AddressSanitizer: **libc** interceptors initialized" shows after running ./mach run +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:: + + $ ASAN_OPTIONS=verbosity=2 ./mach run + +Use the above command instead + +"An admin user name and password" is required to enter Developer Mode +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Please enable **Developer** **mode** by: + +:: + + $ /usr/sbin/DevToolsSecurity -enable + Developer mode is now enabled. + +Debugging issues that ASan finds +-------------------------------- + +When ASan discovers an issue it will simply print an error message and +exit the app. To stop the app in a debugger before ASan exits it, set a +breakpoint on ``__asan::ReportGenericError``. For more info on using +ASan and debugging issues that it uncovers, see the page `Address +sanitizer and a +debugger <https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger>`__ +page on the upstream wiki. + +``__asan_describe_address(pointer)`` issued at the debugger prompt or +even directly in the code allows outputting lots of information about +this memory address (thread and stack of allocation, of deallocation, +whether or not it is a bit outside a known buffer, thread and stack of +allocation of this buffer, etc.). This can be useful to understand where +some buffer that is not aligned was allocated, when doing SIMD work, for +example. + +`rr <https://rr-project.org/>`__ (Linux x86 only) works great with ASan +and combined, this combo allows doing some very powerful debugging +strategies. + +LeakSanitizer +------------- + +LeakSanitizer (LSan) is a special execution mode for regular ASan. It +takes advantage of how ASan tracks the set of live blocks at any given +point to print out the allocation stack of any block that is still alive +at shutdown, but is not reachable from the stack, according to a +conservative scan. This is very useful for detecting leaks of things +such as ``char*`` that do not participate in the usual Gecko shutdown +leak detection. LSan is supported on x86_64 Linux and OS X. + +LSan is enabled by default in ASan builds, as of more recent versions of +Clang. To make an ASan build not run LSan, set the environment variable +``ASAN_OPTIONS`` to ``detect_leaks=0`` (or add it as an entry to a +``:``-separated list if it is already set to something). If you want to +enable it when it is not for some reason, set it to 1 instead of 0. If +LSan is enabled and you are using a non-debug build, you will also want +to set the environment variable ``MOZ_CC_RUN_DURING_SHUTDOWN=1``, to +ensure that we run shutdown GCs and CCs to avoid spurious leaks. + +If an object that is reported by LSan is intentionally never freed, a +symbol can be added to ``build/sanitizers/lsan_suppressions.txt`` to get +LSan to ignore it. + +For some more information on LSan, see the `Leak Sanitizer wiki +page <https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer>`__. + + +A `meta bug called lsan <https://bugzilla.mozilla.org/show_bug.cgi?id=lsan>`__ +is maintained to keep track of all the bugs found with LSan. + + + +Frequently Asked Questions about ASan +------------------------------------- + +How does ASan work exactly? +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +More information on how ASan works can be found on `the Address Sanitizer wiki <https://github.com/google/sanitizers/wiki/AddressSanitizer>`__. diff --git a/tools/sanitizer/docs/asan_nightly.rst b/tools/sanitizer/docs/asan_nightly.rst new file mode 100644 index 0000000000..790def3931 --- /dev/null +++ b/tools/sanitizer/docs/asan_nightly.rst @@ -0,0 +1,204 @@ +ASan Nightly +============ + +The **ASan Nightly Project** involves building a Firefox Nightly browser +with the popular +`AddressSanitizer <https://github.com/google/sanitizers/wiki/AddressSanitizer>`__ +tool and enhancing it with remote crash reporting capabilities for any +errors detected. + +The purpose of the project is to find subtle memory corruptions +occurring during regular browsing that would either not crash at all or +crash in a way that we cannot figure out what the exact problem is just +from the crash dump. We have a lot of inactionable crash reports and +AddressSanitizer traces are usually a lot more actionable on their own +(especially use-after-free traces). Part of this project is to figure +out if and how many actionable crash reports ASan can give us just by +surfing around. The success of the project of course also depends on the +number of participants. + +You can download the latest build using one of the links below. The +builds are self-updating daily like regular nightly builds (like with +regular builds, you can go to *"Help"* → *"About Nightly"* to force an +update check or confirm that you run the latest version). + +.. note:: + + If you came here looking for regular ASan builds (e.g. for fuzzing or + as a developer to reproduce a crash), you should probably go to the + :ref:`Address Sanitizer` doc instead. + +.. _Requirements: + +Requirements +~~~~~~~~~~~~ + +Current requirements are: + +- Windows or Linux-based Operating System +- 16 GB of RAM recommended +- Special ASan Nightly Firefox Build + + - `Linux + Download <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.shippable.latest.firefox.linux64-asan-reporter-opt/artifacts/public/build/target.tar.bz2>`__ + - `Windows + Download <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.shippable.latest.firefox.win64-asan-reporter-shippable-repackage-signing/artifacts/public/build/target.installer.exe>`__ + +If you are already using regular Nightly, it should be safe to share the +profile with the regular Nightly instance. If you normally use a beta or +release build (and you would like to be able to switch back to these), +you should consider using a second profile. + +.. warning:: + + **Windows Users:** Please note that the Windows builds currently show + an error during setup (see "*Known Issues*" section below), but + installation works nonetheless. We are working on the problem. + +.. note:: + + If you run in an environment with any sorts of additional security + restrictions (e.g. custom process sandboxing), please make sure that + your /tmp directory is writable and the shipped ``llvm-symbolizer`` + binary is executable from within the Firefox process. + +Preferences +~~~~~~~~~~~ + +If you wish for your crash report to be identifiable, you can go to +``about:config`` and set the **``asanreporter.clientid``** to your +**valid email address**. This isn't mandatory, you can of course report +crash traces anonymously. If you decide to send reports with your email +address and you have a Bugzilla account, consider using the same email +as your Bugzilla account uses. We will then Cc you on any bugs filed +from your crash reports. If your email does not belong to a Bugzilla +account, then we will not publish it but only use it to resolve +questions about your crash reports. + +.. note:: + + Setting this preference helps us to get back to you in case we have + questions about your setup/OS. Please consider using it so we can get + back to you if necessary. + +Bug Bounty Program +~~~~~~~~~~~~~~~~~~ + +As a special reward for participating in the program, we decided to +treat all submitted reports as if they were filed directly in Bugzilla. +This means that reports that + +- indicate a security issue of critical or high rating +- **and** that can be fixed by our developers + +are eligible for a bug bounty according to our `client bug bounty +program +rules <https://www.mozilla.org/security/client-bug-bounty/>`__. As +the report will usually not include any steps to reproduce or a test +case, it will most likely receive a lower-end bounty. Like with regular +bug reports, we would typically reward the first (identifable) report of +an issue. + +.. warning:: + + If you would like to participate in the bounty program, make sure you + set your **``asanreporter.clientid``** preference as specified above. + We cannot reward any reports that are submitted with no email + address. + + +Known Issues +~~~~~~~~~~~~ + +This section lists all currently known limitations of the ASan Nightly +builds that are considered bugs. + +- [STRIKEOUT:Flash is currently not working] +- `Bug + 1477490 <https://bugzilla.mozilla.org/show_bug.cgi?id=1477490>`__\ [STRIKEOUT:- + Windows: Stack instrumentation disabled due to false positives] +- `Bug + 1478096 <https://bugzilla.mozilla.org/show_bug.cgi?id=1478096>`__ - + **Windows:** Error during install with maintenanceservice_tmp.exe +- It has been reported that ASan Nightly performance is particularly + bad if you run on a screen with 120hz refresh rate. Switching to 60hz + should improve performance drastically. + +Note that these bugs are **specific** to ASan Nightly as listed in the +`tracking bug dependency +list <https://bugzilla.mozilla.org/showdependencytree.cgi?id=1386297&hide_resolved=0>`__. +For the full list of bugs found by this project, see `this +list <https://bugzilla.mozilla.org/showdependencytree.cgi?id=1479399&hide_resolved=0>`__ +instead and note that some bugs might not be shown because they are +security bugs. + +If you encounter a bug not listed here, please file a bug at +`bugzilla.mozilla.org <https://bugzilla.mozilla.org/>`__ or send an +email to +`choller@mozilla.com <mailto:choller@mozilla.com?subject=%5BASan%20Nightly%20Project%5D%5BBug%20Report%5D>`__. +When filing a bug, it greatly helps if you Cc that email address and +make the bug block `bug +1386297 <https://bugzilla.mozilla.org/show_bug.cgi?id=1386297>`__. + +FAQ +~~~ + +What additional data is collected? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The project only collects ASan traces and (if you set it in the +preferences) your email address. We don't collect any other browser +data, in particular not the sites you were visiting or page contents. It +is really just crash traces submitted to a remote location. + +.. note:: + + The ASan Nightly browser also still has all the data collection + capabilities of a regular Nightly browser. The answer above only + refers to what this project collects **in addition** to what the + regular Nightly browser can collect. + +What's the performance impact? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ASan Nightly build only comes with a slight slowdown at startup and +browsing, sometimes it is not even noticeable. The RAM consumption +however is much higher than with a regular build. Be prepared to restart +your browser sometimes, especially if you use a lot of tabs at once. +Also, the updates are larger than the regular ones, so download times +for updates will be higher, especially if you have a slower internet +connection. + +.. warning:: + + If you experience performance issues, see also the *"Known Issues"* + section above, in particular the problem about screen refresh rate + slowing down Firefox. + +What about stability? +^^^^^^^^^^^^^^^^^^^^^ + +The browser is as stable as a regular Nightly build. Various people have +been surfing around with it for their daily work for weeks now and we +have barely received any crash reports. + +How do I confirm that I'm running the correct build? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you open ``about:config`` and type *"asanreporter"* into the search +field, you should see an entry called ``asanreporter.apiurl`` associated +with a URL. Do not modify this value. + +.. warning:: + + Since Firefox 64, the *"ASan Crash Reporter"* feature is no longer + listed in ``about:support`` + +Will there be support for Mac? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We are working on support for Mac, but it might take longer because we +have no ASan CI coverage on Mac due to hardware constraints. If you work +on Release Engineering and would like to help make e.g. Mac happen +earlier, feel free to `contact +me <mailto:choller@mozilla.com?subject=%5BASan%20Nightly%20Project%5D%20>`__. diff --git a/tools/sanitizer/docs/index.rst b/tools/sanitizer/docs/index.rst new file mode 100644 index 0000000000..3a1dc0806e --- /dev/null +++ b/tools/sanitizer/docs/index.rst @@ -0,0 +1,33 @@ +Sanitizer +========= + +.. toctree:: + :maxdepth: 1 + :hidden: + :glob: + + * + +**Address Sanitizer** + +Address Sanitizer (ASan) is a fast memory error detector that detects use-after-free and out-of-bound bugs in C/C++ programs. It uses a compile-time instrumentation to check all reads and writes during the execution. In addition, the runtime part replaces the malloc and free functions to check dynamically allocated memory. + +:ref:`More information <Address Sanitizer>` + +**Thread Sanitizer** + +Thread Sanitizer (TSan) is a fast data race detector for C/C++ programs. It uses a compile-time instrumentation to check all non-race-free memory access at runtime. Unlike other tools, it understands compiler-builtin atomics and synchronization and therefore provides very accurate results with no false positives (except if unsupported synchronization primitives like inline assembly or memory fences are used). + +:ref:`More information <Thread Sanitizer>` + +**Memory Sanitizer** + +Memory Sanitizer (MSan) is a fast detector used for uninitialized memory in C/C++ programs. It uses a compile-time instrumentation to ensure that all memory access at runtime uses only memory that has been initialized. + +:ref:`More information <Memory Sanitizer>` + +**ASan Nightly Project** + +The ASan Nightly Project involves building a Firefox Nightly browser with the popular AddressSanitizer tool and enhancing it with remote crash reporting capabilities for any errors detected. + +:ref:`More information <ASan Nightly>` diff --git a/tools/sanitizer/docs/memory_sanitizer.rst b/tools/sanitizer/docs/memory_sanitizer.rst new file mode 100644 index 0000000000..161826fbc2 --- /dev/null +++ b/tools/sanitizer/docs/memory_sanitizer.rst @@ -0,0 +1,173 @@ +Memory Sanitizer +================ + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +What is Memory Sanitizer? +------------------------- + +Memory Sanitizer (MSan) is a fast detector used for uninitialized memory +in C/C++ programs. It uses a compile-time instrumentation to ensure that +all memory access at runtime uses only memory that has been initialized. +Unlike most other sanitizers, MSan can easily cause false positives if +not all libraries are instrumented. This happens because MSan is +not able to observe memory initialization in uninstrumented libraries. +More information on MSan can be found on `the Memory Sanitizer +wiki <https://github.com/google/sanitizers/wiki/MemorySanitizer>`__. + +Public Builds +------------- + +**Note:** No public builds are available at this time yet. + +Manual Build +------------ + +Build prerequisites +~~~~~~~~~~~~~~~~~~~ + +**Note:** MemorySanitizer requires **64-bit Linux** to work. Other +platforms/operating systems are not supported. + +LLVM/Clang +^^^^^^^^^^ + +The MSan instrumentation is implemented as an LLVM pass and integrated +into Clang. As MSan is one of the newer sanitizers, we recommend using a +recent Clang version, such as Clang 3.7+. + +You can find precompiled binaries for LLVM/Clang on `the LLVM releases +page <https://releases.llvm.org/download.html>`__. + +Building Firefox +~~~~~~~~~~~~~~~~ + +.. warning:: + + **Warning: Running Firefox with MemorySanitizer would require all + external dependencies to be built with MemorySanitizer as well. To + our knowledge, this has never been attempted yet, so the build + configuration provided here is untested and without an appropriately + instrumented userland, it will cause false positives.** + +Getting the source +^^^^^^^^^^^^^^^^^^ + +If you don't have a source code repository clone yet, you need to :ref:`get +yourself a clone of Mozilla-central <Mercurial Overview>`. + +Adjusting the build configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Create the build configuration file ``.mozconfig`` with the following +content in your Mozilla-central directory: + +.. code:: + + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-msan + + # Enable LLVM specific code and build workarounds + ac_add_options --enable-memory-sanitizer + # If clang is already in your $PATH, then these can simply be: + # export CC=clang + # export CXX=clang++ + export CC="/path/to/clang" + export CXX="/path/to/clang++" + + # llvm-symbolizer displays much more complete backtraces when data races are detected. + # If it's not already in your $PATH, then uncomment this next line: + #export LLVM_SYMBOLIZER="/path/to/llvm-symbolizer" + + # Add MSan to our compiler flags + export CFLAGS="-fsanitize=memory" + export CXXFLAGS="-fsanitize=memory" + + # Additionally, we need the MSan flag during linking. Normally, our C/CXXFLAGS would + # be used during linking as well but there is at least one place in our build where + # our CFLAGS are not added during linking. + # Note: The use of this flag causes Clang to automatically link the MSan runtime :) + export LDFLAGS="-fsanitize=memory" + + # These three are required by MSan + ac_add_options --disable-jemalloc + ac_add_options --disable-crashreporter + ac_add_options --disable-elf-hack + + # Keep symbols to symbolize MSan traces + export MOZ_DEBUG_SYMBOLS=1 + ac_add_options --enable-debug-symbols + ac_add_options --disable-install-strip + + # Settings for an opt build + ac_add_options --enable-optimize="-O2 -gline-tables-only" + ac_add_options --disable-debug + +Starting the build process +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now you start the build process using the regular ``make -f client.mk`` +command. + +Starting Firefox +^^^^^^^^^^^^^^^^ + +After the build has completed, you can start Firefox from the ``objdir`` +as usual. + +Building the JavaScript shell +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Note:** Unlike Firefox itself, the JavaScript shell does **not** +require an instrumented userland. Calls to external libraries like +zlib are handled with special annotations inside the engine. + +.. warning:: + + **Warning: Certain technologies used inside the JavaScript engine are + incompatible with MSan and must be disabled at runtime to prevent + false positives. This includes the JITs and asm.js. Therefore always + make sure to run with + ``--no-ion --no-baseline --no-asmjs --no-native-regexp``.** + +If you want to build only the JavaScript shell instead of doing a full +Firefox build, the build script below will probably help you to do so. +Before using it, you must, of course, adjust the path name for +``LLVM_ROOT`` to match your setup. Once you have adjusted everything, +execute this script in the ``js/src/`` subdirectory and pass a directory +name as the first parameter. The build will then be created in a new +subdirectory with that name. + +.. code:: + + #! /bin/sh + + if [ -z $1 ] ; then + echo "usage: $0 <dirname>" + elif [ -d $1 ] ; then + echo "directory $1 already exists" + else + autoconf2.13 + mkdir $1 + cd $1 + LLVM_ROOT="/path/to/llvm" + CC="$LLVM_ROOT/build/bin/clang" \ + CXX="$LLVM_ROOT/build/bin/clang++" \ + CFLAGS="-fsanitize=memory" \ + CXXFLAGS="-fsanitize=memory" \ + LDFLAGS=""-fsanitize=memory" \ + ../configure --enable-debug --enable-optimize --enable-memory-sanitizer --disable-jemalloc --enable-posix-nspr-emulation + make -j 8 + fi + +Using LLVM Symbolizer for faster/better traces +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By default, MSan traces are not symbolized. + +LLVM ships with the symbolizer binary ``llvm-symbolize`` that MSan will +readily use to immediately output symbolized traces if the program is +found on the ``PATH``. If your ``llvm-symbolizer`` lives outside the +``PATH``, you can set the ``MSAN_SYMBOLIZER_PATH`` environment variable +to point to your symbolizer binary. diff --git a/tools/sanitizer/docs/tsan.rst b/tools/sanitizer/docs/tsan.rst new file mode 100644 index 0000000000..77fb6c89d7 --- /dev/null +++ b/tools/sanitizer/docs/tsan.rst @@ -0,0 +1,327 @@ +Thread Sanitizer +================ + +What is Thread Sanitizer? +-------------------------- + +Thread Sanitizer (TSan) is a fast data race detector for C/C++ and Rust +programs. It uses a compile-time instrumentation to check all non-race-free +memory access at runtime. Unlike other tools, it understands compiler-builtin +atomics and synchronization and therefore provides very accurate results +with no false positives (except if unsupported synchronization primitives +like inline assembly or memory fences are used). More information on how +TSan works can be found on `the Thread Sanitizer wiki <https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm>`__. + +A `meta bug called tsan <https://bugzilla.mozilla.org/show_bug.cgi?id=tsan>`__ +is maintained to keep track of all the bugs found with TSan. + +A `blog post on hacks.mozilla.org <https://hacks.mozilla.org/2021/04/eliminating-data-races-in-firefox-a-technical-report/>`__ describes this project. + +Note that unlike other sanitizers, TSan is currently **only supported on Linux**. + +Downloading artifact builds +--------------------------- + +The easiest way to get Firefox builds with Thread Sanitizer is to download a +continuous integration TSan build of mozilla-central (updated at least daily): + +- mozilla-central optimized builds: + `linux <https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-tsan-opt/artifacts/public/build/target.tar.bz2>`__ + +The fuzzing team also offers a tool called ``fuzzfetch`` to download this and many +other CI builds. It makes downloading and unpacking these builds much easier and +can be used not just for fuzzing but for all purposes that require a CI build download. + +You can install ``fuzzfetch`` from +`Github <https://github.com/MozillaSecurity/fuzzfetch>`__ or +`via pip <https://pypi.org/project/fuzzfetch/>`__. + +Afterwards, you can run + +:: + + $ python -m fuzzfetch --tsan -n firefox-tsan + +to get the build mentioned above unpacked into a directory called ``firefox-tsan``. + +Creating Try builds +------------------- + +If for some reason you can't use the pre-built binaries mentioned in the +previous section (e.g. you need to test a patch), you can either build +Firefox yourself (see the following section) or use the :ref:`try server <Pushing to Try>` +to create the customized build for you. Pushing to try requires L1 commit +access. If you don't have this access yet you can request access (see +`Becoming A Mozilla +Committer <https://www.mozilla.org/about/governance/policies/commit/>`__ +and `Mozilla Commit Access +Policy <https://www.mozilla.org/about/governance/policies/commit/access-policy/>`__ +for the requirements). + +Using ``mach try fuzzy --full`` you can select the ``build-linux64-tsan/opt`` job +and related tests (if required). + +Creating local builds on Linux +------------------------------ + +Build prerequisites +~~~~~~~~~~~~~~~~~~~ + +LLVM/Clang/Rust +^^^^^^^^^^^^^^^ + +The TSan instrumentation is implemented as an LLVM pass and integrated +into Clang. We strongly recommend that you use the Clang version supplied +as part of the ``mach bootstrap`` process, as we backported several required +fixes for TSan on Firefox. + +Sanitizer support in Rust is genuinely experimental, +so our build system only works with a specially patched version of Rust +that we build in our CI. To install that specific version (or update to a newer +version), run the following in the root of your mozilla-central checkout: + +:: + + ./mach artifact toolchain --from-build linux64-rust-dev + rm -rf ~/.mozbuild/rustc-sanitizers + mv rustc ~/.mozbuild/rustc-sanitizers + rustup toolchain link gecko-sanitizers ~/.mozbuild/rustc-sanitizers + rustup override set gecko-sanitizers + +``mach artifact`` will always download the ``linux64-rust-dev`` toolchain associated +with the current mozilla central commit you have checked out. The toolchain should +mostly behave like a normal rust nightly but we don't recommend using it for anything +other than building gecko, just in case. Also note that +``~/.mozbuild/rustc-sanitizers`` is just a reasonable default location -- feel +free to "install" the toolchain wherever you please. + +Building Firefox +~~~~~~~~~~~~~~~~ + +Getting the source +^^^^^^^^^^^^^^^^^^ + +Using that or any later revision, all you need to do is to :ref:`get yourself +a clone of mozilla-central <Mercurial overview>`. + +Adjusting the build configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Create the build configuration file ``mozconfig`` with the following +content in your mozilla-central directory: + +:: + + # Combined .mozconfig file for TSan on Linux+Mac + + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-tsan + + # Enable ASan specific code and build workarounds + ac_add_options --enable-thread-sanitizer + + # This ensures that we also instrument Rust code. + export RUSTFLAGS="-Zsanitizer=thread" + + # rustfmt is currently missing in Rust nightly + unset RUSTFMT + + # Current Rust Nightly has warnings + ac_add_options --disable-warnings-as-errors + + # These are required by TSan + ac_add_options --disable-jemalloc + ac_add_options --disable-crashreporter + ac_add_options --disable-elf-hack + ac_add_options --disable-profiling + + # The Thread Sanitizer is not compatible with sandboxing + # (see bug 1182565) + ac_add_options --disable-sandbox + + # Keep symbols to symbolize TSan traces later + export MOZ_DEBUG_SYMBOLS=1 + ac_add_options --enable-debug-symbols + ac_add_options --disable-install-strip + + # Settings for an opt build (preferred) + # The -gline-tables-only ensures that all the necessary debug information for ASan + # is present, but the rest is stripped so the resulting binaries are smaller. + ac_add_options --enable-optimize="-O2 -gline-tables-only" + ac_add_options --disable-debug + + # Settings for a debug+opt build + #ac_add_options --enable-optimize + #ac_add_options --enable-debug + + +Starting the build process +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now you start the build process using the regular ``./mach build`` +command. + +Starting Firefox +^^^^^^^^^^^^^^^^ + +After the build has completed, ``./mach run`` with the usual options for +running in a debugger (``gdb``, ``lldb``, ``rr``, etc.) work fine, as do +the ``--disable-e10s`` and other options. + +Building only the JavaScript shell +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you want to build only the JavaScript shell instead of doing a full +Firefox build, the build script below will probably help you to do so. +Execute this script in the ``js/src/`` subdirectory and pass a directory +name as the first parameter. The build will then be created in a new +subdirectory with that name. + +:: + + #! /bin/sh + + if [ -z $1 ] ; then + echo "usage: $0 <dirname>" + elif [ -d $1 ] ; then + echo "directory $1 already exists" + else + autoconf2.13 + mkdir $1 + cd $1 + CC="/path/to/mozbuild/clang" \ + CXX="/path/to/mozbuild/clang++" \ + ../configure --disable-debug --enable-optimize="-O2 -gline-tables-only" --enable-thread-sanitizer --disable-jemalloc + fi + +Thread Sanitizer and Symbols +---------------------------- + +Unlike Address Sanitizer, TSan requires in-process symbolizing to work +properly in the first place, as any kind of runtime suppressions will +otherwise not work. + +Hence, it is required that you have a copy of ``llvm-symbolizer`` either +in your ``PATH`` or pointed to by the ``TSAN_SYMBOLIZER_PATH`` environment +variable. This binary is included in your local mozbuild directory, obtained +by ``./mach bootstrap``. + + +Runtime Suppressions +-------------------- + +TSan has the ability to suppress race reports at runtime. This can be used to +silence a race while a fix is developed as well as to permanently silence a +(benign) race that cannot be fixed. + +.. warning:: + **Warning**: Many races *look* benign but are indeed not. Please read + the :ref:`FAQ section <Frequently Asked Questions about TSan>` carefully + and think twice before attempting to suppress a race. + +The runtime Suppression list is directly baked into Firefox at compile-time and +located at `mozglue/build/TsanOptions.cpp <https://searchfox.org/mozilla-central/source/mozglue/build/TsanOptions.cpp>`__. + +.. warning:: + **Important**: When adding a suppression, always make sure to include + the bug number. If the suppression is supposed to be permanent, please + add the string ``permanent`` in the same line as the bug number. + +.. warning:: + **Important**: When adding a suppression for a *data race*, always make + sure to include a stack frame from **each** of the two race stacks. + Adding only one suppression for one stack can cause intermittent failures + that are later on hard to track. One exception to this rule is when suppressing + races on global variables. In that case, a single race entry with the name of + the variable is sufficient. + +Troubleshooting / Known Problems +-------------------------------- + +Known Sources of False Positives +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TSan has a number of things that can cause false positives, namely: + + * The use of memory fences (e.g. Rust Arc) + * The use of inline assembly for synchronization + * Uninstrumented code (e.g. external libraries) using compiler-builtins for synchronization + * A lock order inversion involving only a single thread can cause a false positive deadlock + report (see also https://github.com/google/sanitizers/issues/488). + +If none of these four items are involved, you should *never* assume that TSan is reporting +a false positive to you without consulting TSan peers. It is very easy to misjudge a race +to be a false positive because races can be highly complex and totally non-obvious due to +compiler optimizations and the nature of parallel code. + +Intermittent Broken Stacks +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you intermittently see race reports where one stack is missing with a ``failed to restore the stack`` +message, this can indicate that a suppression is partially covering the race you are seeing. + +Any race where only one of the two stacks is matched by a runtime suppression will show up +if that particular stack fails to symbolize for some reason. The usual solution is to search +the suppressions for potential candidates and disable them temporarily to check if your race +report now becomes mostly consistent. + +However, there are other reasons for broken TSan stacks, in particular if they are not intermittent. +See also the ``history_size`` parameter in the `TSan flags <https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags>`__. + +Intermittent Race Reports +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Unfortunately, the TSan algorithm does not guarantee, that a race is detected 100% of the +time. Intermittent failures with TSan are (to a certain degree) to be expected and the races +involved should be filed and fixed to solve the problem. + +.. _Frequently Asked Questions about TSan: + +Frequently Asked Questions about TSan +------------------------------------- + +Why fix data races? +~~~~~~~~~~~~~~~~~~~ + +Data races are undefined behavior and can cause crashes as well as correctness issues. +Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior. + +At Mozilla, we have already seen several dangerous races, causing random +`use-after-free crashes <https://bugzilla.mozilla.org/show_bug.cgi?id=1580288>`__, +`intermittent test failures <https://bugzilla.mozilla.org/show_bug.cgi?id=1602009>`__, +`hangs <https://bugzilla.mozilla.org/show_bug.cgi?id=1607008>`__, +`performance issues <https://bugzilla.mozilla.org/show_bug.cgi?id=1615045>`__ and +`intermittent asserts <https://bugzilla.mozilla.org/show_bug.cgi?id=1601940>`__. Such problems do +not only decrease the quality of our code and user experience, but they also waste countless hours +of developer time. + +Since it is very hard to judge if a particular race could cause such a situation, we +have decided to fix all data races wherever possible, since doing so is often cheaper +than analyzing a race. + +My race is benign, can we ignore it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While it is possible to add a runtime suppression to ignore the race, we *strongly* encourage +you to not do so, for two reasons: + + 1. Each suppressed race decreases the overall performance of the TSan build, as the race + has to be symbolized each time when it occurs. Since TSan is already in itself a slow + build, we need to keep the amount of suppressed races as low as possible. + + 2. Deciding if a race is truly benign is surprisingly hard. We recommend to read + `this blog post <http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong>`__ + and `this paper <https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf>` + on the effects of seemingly benign races. + +Valid reasons to suppress a confirmed benign race include performance problems arising from +fixing the race or cases where fixing the race would require an unreasonable amount of work. + +Note that the use of atomics usually does not have the bad performance impact that developers +tend to associate with it. If you assume that e.g. using atomics for synchronization will +cause performance regressions, we suggest to perform a benchmark to confirm this. In many +cases, the difference is not measurable. + +How does TSan work exactly? +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +More information on how TSan works can be found on `the Thread Sanitizer wiki <https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm>`__. |