diff options
Diffstat (limited to 'tools/fuzzing/docs')
| -rw-r--r-- | tools/fuzzing/docs/fuzzing_interface.rst | 496 | ||||
| -rw-r--r-- | tools/fuzzing/docs/index.rst | 438 |
2 files changed, 934 insertions, 0 deletions
diff --git a/tools/fuzzing/docs/fuzzing_interface.rst b/tools/fuzzing/docs/fuzzing_interface.rst new file mode 100644 index 0000000000..561727d540 --- /dev/null +++ b/tools/fuzzing/docs/fuzzing_interface.rst @@ -0,0 +1,496 @@ +Fuzzing Interface +================= + +The fuzzing interface is glue code living in mozilla-central in order to +make it easier for developers and security researchers to test C/C++ +code with either `libFuzzer <https://llvm.org/docs/LibFuzzer.html>`__ or +`afl-fuzz <http://lcamtuf.coredump.cx/afl/>`__. + +These fuzzing tools, are based on *compile-time instrumentation* to measure +things like branch coverage and more advanced heuristics per fuzzing test. +Doing so allows these tools to progress through code with little to no custom +logic/knowledge implemented in the fuzzer itself. Usually, the only thing +these tools need is a code "shim" that provides the entry point for the fuzzer +to the code to be tested. We call this additional code a *fuzzing target* and +the rest of this manual describes how to implement and work with these targets. + +As for the tools used with these targets, we currently recommend the use of +libFuzzer over afl-fuzz, as the latter is no longer maintained while libFuzzer +is being actively developed. Furthermore, libFuzzer has some advanced +instrumentation features (e.g. value profiling to deal with complicated +comparisons in code), making it overall more effective. + +What can be tested? +~~~~~~~~~~~~~~~~~~~ + +The interface can be used to test all C/C++ code that either ends up in +``libxul`` (more precisely, the gtest version of ``libxul``) **or** is +part of the JS engine. + +Note that this is not the right testing approach for testing the full +browser as a whole. It is rather meant for component-based testing +(especially as some components cannot be easily separated out of the +full build). + +.. note:: + + **Note:** If you are working on the JS engine (trying to reproduce a + bug or seeking to develop a new fuzzing target), then please also read + the :ref:`JS Engine Specifics Section <JS Engine Specifics>` at the end + of this documentation, as the JS engine offers additional options for + implementing and running fuzzing targets. + + +Reproducing bugs for existing fuzzing targets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are working on a bug that involves an existing fuzzing interface target, +you have two options for reproducing the issue: + + +Using existing builds +^^^^^^^^^^^^^^^^^^^^^ + +We have several fuzzing builds in CI that you can simply download. We recommend +using ``fuzzfetch`` for this purpose, as it makes downloading and unpacking +these builds much easier. + +You can install ``fuzzfetch`` from +`Github <https://github.com/MozillaSecurity/fuzzfetch>`__ or +`via pip <https://pypi.org/project/fuzzfetch/>`__. + +Afterwards, you can run + +:: + + $ python -m fuzzfetch -a --fuzzing --gtest -n firefox-fuzzing + +to fetch the latest optimized build. Alternatively, we offer non-ASan debug builds +which you can download using + +:: + + $ python -m fuzzfetch -d --fuzzing --gtest -n firefox-fuzzing + +In both commands, ``firefox-fuzzing`` indicates the name of the directory that +will be created for the download. + +Afterwards, you can reproduce the bug using + +:: + + $ FUZZER=TargetName firefox-fuzzing/firefox test.bin + +assuming that ``TargetName`` is the name of the fuzzing target specified in the +bug you are working on and ``test.bin`` is the attached testcase. + +.. note:: + + **Note:** You should not export the ``FUZZER`` variable permanently + in your shell, especially if you plan to do local builds. If the ``FUZZER`` + variable is exported, it will affect the build process. + +If the CI builds don't meet your requirements and you need a local build instead, +you can follow the steps below to create one: + +.. _Local build requirements and flags: + +Local build requirements and flags +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You will need a Linux environment with a recent Clang. Using the Clang downloaded +by ``./mach bootstrap`` or a newer version is recommended. + +The only build flag required to enable the fuzzing targets is ``--enable-fuzzing``, +so adding + +:: + + ac_add_options --enable-fuzzing + +to your ``.mozconfig`` is already sufficient for producing a fuzzing build. +However, for improved crash handling capabilities and to detect additional errors, +it is strongly recommended to combine libFuzzer with :ref:`AddressSanitizer <Address Sanitizer>` +at least for optimized builds and bugs requiring ASan to reproduce at all +(e.g. you are working on a bug where ASan reports a memory safety violation +of some sort). + +Once your build is complete, if you want to run gtests, you **must** additionally run + +:: + + $ ./mach gtest dontruntests + +to force the gtest libxul to be built. + +.. note:: + + **Note:** If you modify any code, please ensure that you run **both** build + commands to ensure that the gtest libxul is also rebuilt. It is a common mistake + to only run ``./mach build`` and miss the second command. + +Once these steps are complete, you can reproduce the bug locally using the same +steps as described above for the downloaded builds. + + +Developing new fuzzing targets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Developing a new fuzzing target using the fuzzing interface only requires a few steps. + + +Determine if the fuzzing interface is the right tool +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The fuzzing interface is not suitable for every kind of testing. In particular +if your testing requires the full browser to be running, then you might want to +look into other testing methods. + +The interface uses the ``ScopedXPCOM`` implementation to provide an environment +in which XPCOM is available and initialized. You can initialize further subsystems +that you might require, but you are responsible yourself for any kind of +initialization steps. + +There is (in theory) no limit as to how far you can take browser initialization. +However, the more subsystems are involved, the more problems might occur due to +non-determinism and loss of performance. + +If you are unsure if the fuzzing interface is the right approach for you or you +require help in evaluating what could be done for your particular task, please +don't hestitate to :ref:`contact us <Fuzzing#contact-us>`. + + +Develop the fuzzing code +^^^^^^^^^^^^^^^^^^^^^^^^ + +Where to put your fuzzing code +'''''''''''''''''''''''''''''' + +The code using the fuzzing interface usually lives in a separate directory +called ``fuzztest`` that is on the same level as gtests. If your component +has no gtests, then a subdirectory either in tests or in your main directory +will work. If such a directory does not exist yet in your component, then you +need to create one with a suitable ``moz.build``. See `the transport target +for an example <https://searchfox.org/mozilla-central/source/dom/media/webrtc/transport/fuzztest/moz.build>`__ + +In order to include the new subdirectory into the build process, you will +also have to modify the toplevel ``moz.build`` file accordingly. For this +purpose, you should add your directory to ``TEST_DIRS`` only if ``FUZZING_INTERFACES`` +is set. See again `the transport target for an example +<https://searchfox.org/mozilla-central/rev/de7676288a78b70d2b9927c79493adbf294faad5/media/mtransport/moz.build#18-24>`__. + +How your code should look like +'''''''''''''''''''''''''''''' + +In order to define your fuzzing target ``MyTarget``, you only need to implement 2 functions: + +1. A one-time initialization function. + + At startup, the fuzzing interface calls this function **once**, so this can + be used to perform one-time operations like initializing subsystems or parsing + extra fuzzing options. + + This function is the equivalent of the `LLVMFuzzerInitialize <https://llvm.org/docs/LibFuzzer.html#startup-initialization>`__ + function and has the same signature. However, with our fuzzing interface, + it won't be resolved by its name, so it can be defined ``static`` and called + whatever you prefer. Note that the function should always ``return 0`` and + can (except for the return), remain empty. + + For the sake of this documentation, we assume that you have ``static int FuzzingInitMyTarget(int* argc, char*** argv);`` + +2. The fuzzing iteration function. + + This is where the actual fuzzing happens, and this function is the equivalent + of `LLVMFuzzerTestOneInput <https://llvm.org/docs/LibFuzzer.html#fuzz-target>`__. + Again, the difference to the fuzzing interface is that the function won't be + resolved by its name. In addition, we offer two different possible signatures + for this function, either + + ``static int FuzzingRunMyTarget(const uint8_t* data, size_t size);`` + + or + + ``static int FuzzingRunMyTarget(nsCOMPtr<nsIInputStream> inputStream);`` + + The latter is just a wrapper around the first one for implementations that + usually work with streams. No matter which of the two signatures you choose + to work with, the only thing you need to implement inside the function + is the use of the provided data with your target implementation. This can + mean to simply feed the data to your target, using the data to drive operations + on the target API, or a mix of both. + + While doing so, you should avoid altering global state in a permanent way, + using additional sources of data/randomness or having code run beyond the + lifetime of the iteration function (e.g. on another thread), for one simple + reason: Coverage-guided fuzzing tools depend on the **deterministic** nature + of the iteration function. If the same input to this function does not lead + to the same execution when run twice (e.g. because the resulting state depends + on multiple successive calls or because of additional external influences), + then the tool will not be able to reproduce its fuzzing progress and perform + badly. Dealing with this restriction can be challenging e.g. when dealing + with asynchronous targets that run multi-threaded, but can usually be managed + by synchronizing execution on all threads at the end of the iteration function. + For implementations accumulating global state, it might be necessary to + (re)initialize this global state in each iteration, rather than doing it once + in the initialization function, even if this costs additional performance. + + Note that unlike the vanilla libFuzzer approach, you are allowed to ``return 1`` + in this function to indicate that an input is "bad". Doing so will cause + libFuzzer to discard the input, no matter if it generated new coverage or not. + This is particularly useful if you have means to internally detect and catch + bad testcase behavior such as timeouts/excessive resource usage etc. to avoid + these tests to end up in your corpus. + + +Once you have implemented the two functions, the only thing remaining is to +register them with the fuzzing interface. For this purpose, we offer two +macros, depending on which iteration function signature you used. If you +sticked to the classic signature using buffer and size, you can simply use + +:: + + #include "FuzzingInterface.h" + + // Your includes and code + + MOZ_FUZZING_INTERFACE_RAW(FuzzingInitMyTarget, FuzzingRunMyTarget, MyTarget); + +where ``MyTarget`` is the name of the target and will be used later to decide +at runtime which target should be used. + +If instead you went for the streaming interface, you need a different include, +but the macro invocation is quite similar: + +:: + + #include "FuzzingInterfaceStream.h" + + // Your includes and code + + MOZ_FUZZING_INTERFACE_STREAM(FuzzingInitMyTarget, FuzzingRunMyTarget, MyTarget); + +For a live example, see also the `implementation of the STUN fuzzing target +<https://searchfox.org/mozilla-central/source/dom/media/webrtc/transport/fuzztest/stun_parser_libfuzz.cpp>`__. + +Add instrumentation to the code being tested +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +libFuzzer requires that the code you are trying to test is instrumented +with special compiler flags. Fortunately, adding these on a per-directory basis +can be done just by including the following directive in each ``moz.build`` +file that builds code under test: + +:: + + # Add libFuzzer configuration directives + include('/tools/fuzzing/libfuzzer-config.mozbuild') + + +The include already does the appropriate configuration checks to be only +active in fuzzing builds, so you don't have to guard this in any way. + +.. note:: + + **Note:** This include modifies `CFLAGS` and `CXXFLAGS` accordingly + but this only works for source files defined in this particular + directory. The flags are **not** propagated to subdirectories automatically + and you have to ensure that each directory that builds source files + for your target has the include added to its ``moz.build`` file. + +By keeping the instrumentation limited to the parts that are actually being +tested using this tool, you not only increase the performance but also potentially +reduce the amount of noise that libFuzzer sees. + + +Build your code +^^^^^^^^^^^^^^^ + +See the :ref:`Build instructions above <Local build requirements and flags>` for instructions +how to modify your ``.mozconfig`` to create the appropriate build. + + +Running your code and building a corpus +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You need to set the following environment variable to enable running the +fuzzing code inside Firefox instead of the regular browser. + +- ``FUZZER=name`` + +Where ``name`` is the name of your fuzzing module that you specified +when calling the ``MOZ_FUZZING_INTERFACE_RAW`` macro. For the example +above, this would be ``MyTarget`` or ``StunParser`` for the live example. + +Now when you invoke the firefox binary in your build directory with the +``-help=1`` parameter, you should see the regular libFuzzer help. On +Linux for example: + +:: + + $ FUZZER=StunParser obj-asan/dist/bin/firefox -help=1 + +You should see an output similar to this: + +:: + + Running Fuzzer tests... + Usage: + + To run fuzzing pass 0 or more directories. + obj-asan/dist/bin/firefox [-flag1=val1 [-flag2=val2 ...] ] [dir1 [dir2 ...] ] + + To run individual tests without fuzzing pass 1 or more files: + obj-asan/dist/bin/firefox [-flag1=val1 [-flag2=val2 ...] ] file1 [file2 ...] + + Flags: (strictly in form -flag=value) + verbosity 1 Verbosity level. + seed 0 Random seed. If 0, seed is generated. + runs -1 Number of individual test runs (-1 for infinite runs). + max_len 0 Maximum length of the test input. If 0, libFuzzer tries to guess a good value based on the corpus and reports it. + ... + + +Reproducing a Crash +''''''''''''''''''' + +In order to reproduce a crash from a given test file, simply put the +file as the only argument on the command line, e.g. + +:: + + $ FUZZER=StunParser obj-asan/dist/bin/firefox test.bin + +This should reproduce the given problem. + + +FuzzManager and libFuzzer +''''''''''''''''''''''''' + +Our FuzzManager project comes with a harness for running libFuzzer with +an optional connection to a FuzzManager server instance. Note that this +connection is not mandatory, even without a server you can make use of +the local harness. + +You can find the harness +`here <https://github.com/MozillaSecurity/FuzzManager/tree/master/misc/afl-libfuzzer>`__. + +An example invocation for the harness to use with StunParser could look +like this: + +:: + + FUZZER=StunParser python /path/to/afl-libfuzzer-daemon.py --fuzzmanager \ + --stats libfuzzer-stunparser.stats --libfuzzer-auto-reduce-min 500 --libfuzzer-auto-reduce 30 \ + --tool libfuzzer-stunparser --libfuzzer --libfuzzer-instances 6 obj-asan/dist/bin/firefox \ + -max_len=256 -use_value_profile=1 -rss_limit_mb=3000 corpus-stunparser + +What this does is + +- run libFuzzer on the ``StunParser`` target with 6 parallel instances + using the corpus in the ``corpus-stunparser`` directory (with the + specified libFuzzer options such as ``-max_len`` and + ``-use_value_profile``) +- automatically reduce the corpus and restart if it grew by 30% (and + has at least 500 files) +- use FuzzManager (need a local ``.fuzzmanagerconf`` and a + ``firefox.fuzzmanagerconf`` binary configuration as described in the + FuzzManager manual) and submit crashes as ``libfuzzer-stunparser`` + tool +- write statistics to the ``libfuzzer-stunparser.stats`` file + +.. _JS Engine Specifics: + +JS Engine Specifics +~~~~~~~~~~~~~~~~~~~ + +The fuzzing interface can also be used for testing the JS engine, in fact there +are two separate options to implement and run fuzzing targets: + +Implementing in C++ +^^^^^^^^^^^^^^^^^^^ + +Similar to the fuzzing interface in Firefox, you can implement your target in +entirely C++ with very similar interfaces compared to what was described before. + +There are a few minor differences though: + +1. All of the fuzzing targets live in `js/src/fuzz-tests`. + +2. All of the code is linked into a separate binary called `fuzz-tests`, + similar to how all JSAPI tests end up in `jsapi-tests`. In order for this + binary to be built, you must build a JS shell with ``--enable-fuzzing`` + **and** ``--enable-tests``. Again, this can and should be combined with + AddressSanitizer for maximum effectiveness. This also means that there is no + need to (re)build gtests when dealing with a JS fuzzing target and using + a shell as part of a full browser build. + +3. The harness around the JS implementation already provides you with an + initialized ``JSContext`` and global object. You can access these in + your target by declaring + + ``extern JS::PersistentRootedObject gGlobal;`` + + and + + ``extern JSContext* gCx;`` + + but there is no obligation for you to use these. + +For a live example, see also the `implementation of the StructuredCloneReader target +<https://searchfox.org/mozilla-central/source/js/src/fuzz-tests/testStructuredCloneReader.cpp>`__. + + +Implementing in JS +^^^^^^^^^^^^^^^^^^ + +In addition to the C++ targets, you can also implement targets in JavaScript +using the JavaScript Runtime (JSRT) fuzzing approach. Using this approach is +not only much simpler (since you don't need to know anything about the +JSAPI or engine internals), but it also gives you full access to everything +defined in the JS shell, including handy functions such as ``timeout()``. + +Of course, this approach also comes with disadvantages: Calling into JS and +performing the fuzzing operations there costs performance. Also, there is more +chance for causing global side-effects or non-determinism compared to a +fairly isolated C++ target. + +As a rule of thumb, you should implement the target in JS if + +* you don't know C++ and/or how to use the JSAPI (after all, a JS fuzzing target is better than none), +* your target is expected to have lots of hangs/timeouts (you can catch these internally), +* or your target is not isolated enough for a C++ target and/or you need specific JS shell functions. + + +There is an `example target <https://searchfox.org/mozilla-central/source/js/src/shell/jsrtfuzzing/jsrtfuzzing-example.js>`__ +in-tree that shows roughly how to implement such a fuzzing target. + +To run such a target, you must run the ``js`` (shell) binary instead of the +``fuzz-tests`` binary and point the ``FUZZER`` variable to the file containing +your fuzzing target, e.g. + +:: + + $ FUZZER=/path/to/jsrtfuzzing-example.js obj-asan/dist/bin/js --fuzzing-safe --no-threads -- <libFuzzer options here> + +More elaborate targets can be found in `js/src/fuzz-tests/ <https://searchfox.org/mozilla-central/source/js/src/fuzz-tests/>`__. + +Troubleshooting +~~~~~~~~~~~~~~~ + + +Fuzzing Interface: Error: No testing callback found +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This error means that the fuzzing callback with the name you specified +using the ``FUZZER`` environment variable could not be found. Reasons +for are typically either a misspelled name or that your code wasn't +built (check your ``moz.build`` file and build log). + + +``mach build`` doesn't seem to update my fuzzing code +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Keep in mind you always need to run both the ``mach build`` and +``mach gtest dontruntests`` commands in order to update your fuzzing +code. The latter rebuilds the gtest version of ``libxul``, containing +your code. diff --git a/tools/fuzzing/docs/index.rst b/tools/fuzzing/docs/index.rst new file mode 100644 index 0000000000..9a4e2d01c4 --- /dev/null +++ b/tools/fuzzing/docs/index.rst @@ -0,0 +1,438 @@ +Fuzzing +======= + +.. toctree:: + :maxdepth: 1 + :hidden: + :glob: + :reversed: + + * + +This section focuses on explaining the software testing technique called +“Fuzzing” or “Fuzz Testing” and its application to the Mozilla codebase. +The overall goal is to educate developers about the capabilities and +usefulness of fuzzing and also allow them to write their own fuzzing +targets. Note that not all fuzzing tools used at Mozilla are open +source. Some tools are for internal use only because they can easily +find critical security vulnerabilities. + +What is Fuzzing? +---------------- + +Fuzzing (or Fuzz Testing) is a technique to randomly use a program or +parts of it with the goal to uncover bugs. Random usage can have a wide +variety of forms, a few common ones are + +- random input data (e.g. file formats, network data, source code, etc.) + +- random API usage + +- random UI interaction + +with the first two being the most practical methods used in the field. +Of course, these methods are not entirely separate, combinations are +possible. Fuzzing is a great way to find quality issues, some of them +being also security issues. + +Random input data +~~~~~~~~~~~~~~~~~ + +This is probably the most obvious fuzzing method: You have code that +processes data and you provide it with random or mutated data, hoping +that it will uncover bugs in your implementation. Examples are media +formats like JPEG or H.264, but basically anything that involves +processing a “blob” of data can be a valuable target. Countless security +vulnerabilities in a variety of libraries and programs have been found +using this method (the AFLFuzz +`bug-o-rama <http://lcamtuf.coredump.cx/afl/#bugs>`__ gives a good +impression). + +Common tools for this task are e.g. +`libFuzzer <https://llvm.org/docs/LibFuzzer.html>`__ and +`AFLFuzz <http://lcamtuf.coredump.cx/afl/>`__, but also specialized +tools with custom logic like +`LangFuzz <https://www.usenix.org/system/files/conference/usenixsecurity12/sec12-final73.pdf>`__ +and `Avalanche <https://github.com/MozillaSecurity/avalanche>`__. + +Random API Usage +~~~~~~~~~~~~~~~~ + +Randomly testing APIs is especially helpful with parts of software that +expose a well-defined interface (see also :ref:`Well-defined +behavior and Safety <Well defined behaviour and safety>`). If this interface is additionally exposed to +untrusted parties/content, then this is a strong sign that random API +testing would be worthwhile here, also for security reasons. APIs can be +anything from C++ layer code to APIs offered in the browser. + +A good example for a fuzzing target here is the DOM (Document Object +Model) and various other browser APIs. The browser exposes a variety of +different APIs for working with documents, media, communication, +storage, etc. with a growing complexity. Each of these APIs has +potential bugs that can be uncovered with fuzzing. At Mozilla, we +currently use domino (internal tool) for this purpose. + +Random UI Interaction +~~~~~~~~~~~~~~~~~~~~~ + +A third way to test programs and in particular user interfaces is by +directly interacting with the UI in a random way, typically in +combination with other actions the program has to perform. Imagine for +example an automated browser that surfs through the web and randomly +performs actions such as scrolling, zooming and clicking links. The nice +thing about this approach is that you likely find many issues that the +end-user also experiences. However, this approach typically suffers from +bad reproducibility (see also :ref:`Reproducibility <Reproducibility>`) and is therefore +often of limited use. + +An example for a fuzzing tool using this technique is `Android +Monkey <https://developer.android.com/studio/test/monkey>`__. At +Mozilla however, we currently don’t make much use of this approach. + +Why Fuzzing Helps You +--------------------- + +Understanding the value of fuzzing for you as a developer and software +quality in general is important to justify the support this testing +method might need from you. When your component is fuzzed for the first +time there are two common things you will be confronted with: + +**Bug reports that don’t seem real bugs or not important:** Fuzzers +find all sorts of bugs in various corners of your component, even +obscure ones. This automatically leads to a larger number of bugs that +either don’t seem to be bugs (see also the :ref:`Well-defined behavior and +safety <Well defined behaviour and safety>` section below) or that don’t seem to be important bugs. + +Fixing these bugs is still important for the fuzzers because ignoring them +in fuzzing costs resources (performance, human resources) and might even +prevent the fuzzer from hitting other bugs. For example certain fuzzing tools +like libFuzzer run in-process and have to restart on every crash, involving a +costly re-read of the fuzzing samples. + +Also, as some of our code evolves quickly, a corner case might become a +hot code path in a few months. + +**New steps to reproduce:** Fuzzing tools are very likely to exercise +your component using different methods than an average end-user. A +common technique is modify existing parts of a program or write entirely +new code to yield a fuzzing "target". This target is specifically +designed to work with the fuzzing tools in use. Reproducing the reported +bugs might require you to learn these new steps to reproduce, including +building/acquiring that target and having the right environment. + +Both of these issues might seem like a waste of time in some cases, +however, realizing that both steps are a one-time investment for a +constant stream of valuable bug reports is paramount here. Helping your +security engineers to overcome these issues will ensure that future +regressions in your code can be detected at an earlier stage and in a +form that is more easily actionable. Especially if you are dealing with +regressions in your code already, fuzzing has the potential to make your +job as a developer easier. + +One of the best examples at Mozilla is the JavaScript engine. The JS +team has put great quite some effort into getting fuzzing started and +supporting our work. Here’s what Jan de Mooij, a senior platform +engineer for the JavaScript engine, has to say about it: + +*“Bugs in the engine can cause mysterious browser crashes and bugs that +are incredibly hard to track down. Fortunately, we don't have to deal +with these time consuming browser issues very often: usually the fuzzers +find a reliable shell test long before the bug makes it into a release. +Fuzzing is invaluable to us and I cannot imagine working on this project +without it.”* + +Levels of Fuzzing in Firefox/Gecko +---------------------------------- + +Applying fuzzing to e.g. Firefox happens at different "levels", similar +to the different types of automated tests we have: + +Full Browser Fuzzing +~~~~~~~~~~~~~~~~~~~~ + +The most obvious method of testing would be to test the full browser and +doing so is required for certain features like the DOM and other APIs. +The advantage here is that we have all the features of the browser +available and testing happens closely to what we actually ship. The +downside here though is that browser testing is by far the slowest of +all testing methods. In addition, it has the most amount of +non-determinism involved (resulting e.g. in intermittent testcases). +Browser fuzzing at Mozilla is largely done with the `Grizzly +framework <https://blog.mozilla.org/security/2019/07/10/grizzly/>`__ +(`meta bug <https://bugzilla.mozilla.org/show_bug.cgi?id=grizzly>`__) +and one of the most successful fuzzers is the Domino tool (`meta +bug <https://bugzilla.mozilla.org/show_bug.cgi?id=domino>`__). + +Summarizing, full browser fuzzing is the right technique to investigate +if your feature really requires it. Consider using other methods (see +below) if your code can be exercised in this way. + +The Fuzzing Interface +~~~~~~~~~~~~~~~~~~~~~ + +**Fuzzing Interface** + +The fuzzing interface is glue code living in mozilla-central in order to make it +easier for developers and security researchers to test C/C++ code with either libFuzzer or afl-fuzz. + +This interface offers a gtest (C++ unit test) level component based +fuzzing approach and is suitable for anything that could also be +tested/exercised using a gtest. This method is by far the fastest, but +usually limited to testing isolated components that can be instantiated +on this level. Utilizing this method requires you to write a fuzzing +target similar to writing a gtest. This target will automatically be +usable with libFuzzer and AFLFuzz. We offer a :ref:`comprehensive manual <Fuzzing Interface>` +that describes how to write and utilize your own target. + +A simple example here is the `SDP parser +target <https://searchfox.org/mozilla-central/rev/efdf9bb55789ea782ae3a431bda6be74a87b041e/media/webrtc/signaling/fuzztest/sdp_parser_libfuzz.cpp#30>`__, +which tests the SipccSdpParser in our codebase. + +Shell-based Fuzzing +~~~~~~~~~~~~~~~~~~~ + +Some of our fuzzing, e.g. JS Engine testing, happens in a separate shell +program. For JS, this is the JS shell also used for most of the JS tests +and development. In theory, xpcshell could also be used for testing but +so far, there has not been a use case for this (most things that can be +reached through xpcshell can also be tested on the gtest level). + +Identifying the right level of fuzzing is the first step towards +continuous fuzz testing of your code. + +Code/Process Requirements for Fuzzing +------------------------------------- + +In this section, we are going to discuss how code should be written in +order to yield optimal results with fuzzing. + +Defect Oracles +~~~~~~~~~~~~~~ + +Fuzzing is only effective if you are able to know when a problem has +been found. Crashes are typically problems if the unit being tested is +safe for fuzzing (see Well-defined behavior and Safety). But there are +many more problems that you would want to find, correctness issues, +corruptions that don’t necessarily crash etc. For this, you need an +*oracle* that tells you something is wrong. + +The simplest defect oracle is the assertion (ex: ``MOZ_ASSERT``). +Assertions are a very powerful instrument because they can be used to +determine if your program is performing correctly, even if the bug would +not lead to any sort of crash. They can encode arbitrarily complex +information about what is considered correct, information that might +otherwise only exist in the developers’ minds. + +External tools like the sanitizers (AddressSanitizer aka ASan, +ThreadSanitizer aka TSan, MemorySanitizer aka MSan and +UndefinedBehaviorSanitizer - UBSan) can also serve as oracles for +sometimes severe issues that would not necessarily crash. Making sure +that these tools can be used on your code is highly useful. + +Examples for bugs found with sanitizers are `bug +1419608 <https://bugzilla.mozilla.org/show_bug.cgi?id=1419608>`__, +`bug 1580288 <https://bugzilla.mozilla.org/show_bug.cgi?id=1580288>`__ +and `bug 922603 <https://bugzilla.mozilla.org/show_bug.cgi?id=922603>`__, +but since we started using sanitizers, we have found over 1000 bugs with +these tools. + +Another defect oracle can be a reference implementation. Comparing +program behavior (typically output) between two programs or two modes of +the same program that should produce the same outputs can find complex +correctness issues. This method is often called differential testing. + +One example where this is regularly used to find issues is the Mozilla +JavaScript engine: Running random programs with and without JIT +compilation enabled finds lots of problems with the JIT implementation. +One example for such a bug is `Bug +1404636 <https://bugzilla.mozilla.org/show_bug.cgi?id=1404636>`__. + +Component Decoupling +~~~~~~~~~~~~~~~~~~~~ + +Being able to test components in isolation can be an advantage for +fuzzing (both for performance and reproducibility). Clear boundaries +between different components and documentation that explains the +contracts usually help with this goal. Sometimes it might be useful to +mock a certain component that the target component is interacting with +and that is much harder if the components are tightly coupled and their +contracts unclear. Of course, this does not mean that one should only +test components in isolation. Sometimes, testing the interaction between +them is even desirable and does not hurt performance at all. + +Avoiding external I/O +~~~~~~~~~~~~~~~~~~~~~ + +External I/O like network or file interactions are bad for performance +and can introduce additional non-determinism. Providing interfaces to +process data directly from memory instead is usually much more helpful. + +.. _Well defined behaviour and safety: + +Well-defined Behavior and Safety +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This requirement mostly ties in where defect oracles ended and is one of +the most important problems seen in the wild nowadays with fuzzing. If a +part of your program’s behavior is unspecified, then this potentially +leads to bad times if the behavior is considered a defect by fuzzing. +For example, if your code has crashes that are not considered bugs, then +your code might be unsuitable for fuzzing. Your component should be +fuzzing safe, meaning that any defect oracle (e.g. assertion or crash) +triggered by the fuzzer is considered a bug. This important aspect is +often neglected. Be aware that any false positives cause both +performance degradation and additional manual work for your fuzzing +team. The Mozilla JS developers for example have implemented this +concept in a “--fuzzing-safe” switch which disables harmful functions. +Sometimes, crashes cannot be avoided for handling certain error +conditions. In such situations, it is important to mark these crashes in +a way the fuzzer can recognize and distinguish them from undesired +crashes. However, keep in mind that crashes in general can be disruptive +to the fuzzing process. Performance is an important aspect of fuzzing +and frequent crashes can severely degrade performance. + +.. _Reproducibility: + +Reproducibility +~~~~~~~~~~~~~~~ + +Being able to reproduce issues found with fuzzing is necessary for +several reasons: First, you as the developer probably want a test that +reproduces the issue so you can debug it better. Our feedback from most +developers is that traces without a reproducible test can help to find a +problem, but it makes the whole process very complicated. Some of these +non-reproducible bugs never get fixed. Second, having a reproducible +test also helps the triage process by allowing an automated bisection to +find the responsible developer. Last but not least, the test can be +added to a test suite, used for automated verification of fixes and even +serve as a basis for more fuzzing. + +Adding functionality to the program that improve reproducibility is +therefore a good idea in case non-reproducible issues are found. Some +examples are shown in the next section. + +While many problems with reproducibility are specific for the project +you are working on, there is one source of these problems that many +programs have in common: Threading. While some bugs only occur in the +first place due to concurrency, some other bugs would be perfectly +reproducible without threads, but are intermittent and hard to with +threading enabled. If the bug is indeed caused by a data race, then +tools like ThreadSanitizer will help and we are currently working on +making ThreadSanitizer usable on Firefox. For bugs that are not caused +by threading, it sometimes makes sense to be able to disable threading +or limit the amount of worker threads involved. + +Supporting Code +~~~~~~~~~~~~~~~ + +Some possibilities of what support implementations for fuzzing can do +have already been named in the previous sections: Additional defect +oracles and functionality to improve reproducibility and safety. In +fact, many features added specifically for fuzzing fit into one of these +categories. However, there’s room for more: Often, there are ways to +make it easier for fuzzers to exercise complex and hard to reach parts +of your code. For example, if a certain optimization feature is only +turned on under very specific conditions (that are not a requirement for +the optimization), then it makes sense to add a functionality to force +it on. Then, a fuzzer can hit the optimization code much more +frequently, increasing the chance to find issues. Some examples from +Firefox and SpiderMonkey: + +- The `FuzzingFunctions <https://searchfox.org/mozilla-central/rev/efdf9bb55789ea782ae3a431bda6be74a87b041e/dom/webidl/FuzzingFunctions.webidl#15>`__ + interface in the browser allows fuzzing tools to perform GC/CC, tune various + settings related to garbage collection or enable features like accessibility + mode. Being able to force a garbage collection at a specific time helped + identifying lots of problems in the past. + +- The --ion-eager and --baseline-eager flags for the JS shell force JIT + compilation at various stages, rather than using the builtin + heuristic to enable it only for hot functions. + +- The --no-threads flag disables all threading (if possible) in the JS shell. + This makes some bugs reproduce deterministically that would otherwise be + intermittent and harder to find. However, some bugs that only occur with + threading can’t be found with this option enabled. + +Another important feature that must be turned off for fuzzing is +checksums. Many file formats use checksums to validate a file before +processing it. If a checksum feature is still enabled, fuzzers are +likely never going to produce valid files. The same often holds for +cryptographic signatures. Being able to turn off the validation of these +features as part of a fuzzing switch is extremely helpful. + +An example for such a checksum can be found in the +`FlacDemuxer <https://searchfox.org/mozilla-central/rev/efdf9bb55789ea782ae3a431bda6be74a87b041e/dom/media/flac/FlacDemuxer.cpp#494>`__. + +Test Samples +~~~~~~~~~~~~ + +Some fuzzing strategies make use of existing data that is mutated to +produce the new random data. In fact, mutation-based strategies are +typically superior to others if the original samples are of good quality +because the originals carry a lot of semantics that the fuzzer does not +have to know about or implement. However, success here really stands and +falls with the quality of the samples. If the originals don’t cover +certain parts of the implementation, then the fuzzer will also have to +do more work to get there. + + +Fuzz Blockers +~~~~~~~~~~~~~ + +Fuzz blockers are issues that prevent fuzzers from being as +effective as possible. Depending on the fuzzer and its scope a fuzz blocker +in one area (or component) can impede performance in other areas and in +some cases block the fuzzer all together. Some examples are: + +- Frequent crashes - These can block code paths and waste compute + resources due to the need to relaunch the fuzzing target and handle + the results (regardless of whether it is ignored or reported). This can also + include assertions that are mostly benign in many cases are but easily + triggered by fuzzers. + +- Frequent hangs / timeouts - This includes any issue that slows down + or blocks execution of the fuzzer or the target. + +- Hard to bucket - This includes crashes such as stack overflows or any issue + that crashes in an inconsistent location. This also includes issues that + corrupt logs/debugger output or provide a broken/invalid crash report. + +- Broken builds - This is fairly straightforward, without up-to-date builds + fuzzers are unable to run or verify fixes. + +- Missing instrumentation - In some cases tools such as ASan are used as + defect oracles and are required by the fuzzing tools to allow for proper + automation. In other cases incomplete instrumentation can give a false sense + of stability or make investigating issues much more time consuming. Although + this is not necessarily blocking the fuzzers it should be prioritized + appropriately. + +Since these types of crashes harm the overall fuzzing progress, it is important +for them to be addressed in a timely manner. Even if the bug itself might seem +trivial and low priority for the product, it can still have devastating effects +on fuzzing and hence prevent finding other critical issues. + +Issues in Bugzilla are marked as fuzz blockers by adding “[fuzzblocker]” +to the “Whiteboard” field. A list of open issues marked as fuzz blockers +can be found on `Bugzilla <https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=fuzzblockers&sharer_id=486634>`__. + + +Documentation +~~~~~~~~~~~~~ + +It is important for the fuzzing team to know how your software, tests +and designs work. Even obvious tasks, like how a test program is +supposed to be invoked, which options are safe, etc. might be hard to +figure out for the person doing the testing, just as you are reading +this manual right now to find out what is important in fuzzing. + +Contact Us +~~~~~~~~~~ + +The fuzzing team can be reached at +`fuzzing@mozilla.com <mailto:fuzzing@mozilla.com>`__ or +`on Matrix <https://chat.mozilla.org/#/room/#fuzzing:mozilla.org>`__ +and will be happy to help you with any questions about fuzzing +you might have. We can help you find the right method of fuzzing for +your feature, collaborate on the implementation and provide the +infrastructure to run it and process the results accordingly. |