diff options
Diffstat (limited to 'docs/contributing')
67 files changed, 7287 insertions, 0 deletions
diff --git a/docs/contributing/Code_Review_FAQ.rst b/docs/contributing/Code_Review_FAQ.rst new file mode 100644 index 0000000000..18fe85a6e2 --- /dev/null +++ b/docs/contributing/Code_Review_FAQ.rst @@ -0,0 +1,93 @@ +Code Review FAQ +=============== + +What is the purpose of code review? +----------------------------------- + +Code review is our basic mechanism for validating the design and +implementation of patches. It also helps us maintain a level of +consistency in design and implementation practices across the many +hackers and among the various modules of Mozilla. + +Of course, code review doesn't happen instantaneously, and so there is +some latency built into the system. We're always looking for ways to +reduce the wait, while simultaneously allowing reviewers to do a good +chunk of hacking themselves. We don't have a perfect system, and we +never will. It's still evolving, so let us know if you have suggestions. + +Mozilla used to have the concept of "super-review", but `a consensus was +reached in +2018 <https://groups.google.com/forum/#!topic/mozilla.governance/HHU0h-44NDo>`__ +to retire this process. + +Who must review my code? +------------------------ + +You must have an approval ("r={{ mediawiki.external('name') }}") from +the module owner or designated "peer" of the module where the code will +be checked in. If your code affects several modules, then generally you +should have an "r={{ mediawiki.external('name') }}" from the owner or +designated peer of each affected module. We try to be reasonable here, +so we don't have an absolute rule on when every module owner must +approve. For example, tree-wide changes such as a change to a string +class or a change to text that is displayed in many modules generally +doesn't get reviewed by every module owner. + +You may wish to ask others as well. + + +What do reviewers look for? +--------------------------- + +A review is focused on a patch's design, implementation, usefulness in +fixing a stated problem, and fit within its module. A reviewer should be +someone with domain expertise in the problem area. A reviewer may also +utilize other areas of his or her expertise and comment on other +possible improvements. There are no inherent limitations on what +comments a reviewer might make about improving the code. + +Reviewers will probably look at the following areas of the code: + +- “goal” review: is the issue being fixed actually a bug? Does the + patch fix the fundamental problem? +- API/design review. Because APIs define the interactions between + modules, they need special care. Review is especially important to + keep APIs balanced and targeted, and not too specific or + overdesigned. There are a `WebIDL review + checklist <https://wiki.mozilla.org/WebAPI/WebIDL_Review_Checklist>`__. + There are also templates for emails that should be sent when APIs are + going to be exposed to the Web and general guidance around naming on + `this wiki + page <https://wiki.mozilla.org/WebAPI/ExposureGuidelines>`__. +- Maintainability review. Code which is unreadable is impossible to + maintain. If the reviewer has to ask questions about the purpose of a + piece of code, then it is probably not documented well enough. Does + the code follow the :ref:`Coding style` ? Be careful when + reviewing code using modern C++ features like auto. +- Security review. Does the design use security concepts such as input + sanitizers, wrappers, and other techniques? Does this code need + additional security testing such as fuzz-testing or static analysis? +- Integration review. Does this code work properly with other modules? + Is it localized properly? Does it have server dependencies? Does it + have user documentation? +- Testing review. Are there tests for correct function? Are there tests + for error conditions and incorrect inputs which could happen during + operation? +- Performance review. Has this code been profiled? Are you sure it's + not negatively affecting performance of other code? +- License review. Does the code follow the `code licensing + rules <http://www.mozilla.org/hacking/committer/committers-agreement.pdf>`__? + + +How can I tell the status of reviews? +------------------------------------- + +When a patch has passed review you'll see "Accepted" in green at the top +of a Phabricator revision, under the title. In Bugzilla (which is +deprecated in favour of Phabricator), this is indicated by "{{ +mediawiki.external('name') }}:review+" in the attachment table in the +bug report. If it has failed review then you'll see "Needs Revision" in +red at the top of the revision, or, in Bugzilla, "{{ +mediawiki.external('name') }}:review-". Most of the time that a reviewer +sets a review flag, they will also add a comment to the bug explaining +the review. diff --git a/docs/contributing/build/artifact_builds.rst b/docs/contributing/build/artifact_builds.rst new file mode 100644 index 0000000000..302331e4c0 --- /dev/null +++ b/docs/contributing/build/artifact_builds.rst @@ -0,0 +1,193 @@ +Understanding Artifact Builds +============================= + +Firefox for Desktop and Android supports a **fast build mode** called +*artifact mode*. The resulting builds are called *artifact builds*. +Artifact mode downloads pre-built C++ components rather than building them +locally, trading bandwidth for time. + +Artifact builds will be useful to many developers who are not working +with compiled code (see "Restrictions" below). Artifacts are typically +fetched from `mozilla-central <https://hg.mozilla.org/mozilla-central/>`__. + +To automatically download and use pre-built binary artifacts, add the +following lines into your :ref:`mozconfig <Configuring Build Options>` +file: + +.. code-block:: shell + + # Automatically download and use compiled C++ components: + ac_add_options --enable-artifact-builds + + # Write build artifacts to: + mk_add_options MOZ_OBJDIR=./objdir-frontend + +To automatically download and use the debug version of the pre-built +binary artifact (currently supported for Linux, OSX and Windows +artifacts), add ``ac_add_options --enable-debug`` to your mozconfig file +(with artifact builds option already enabled): + +.. code-block:: shell + + # Enable debug versions of the pre-built binary artifacts: + ac_add_options --enable-debug + + # Automatically download and use compiled C++ components: + ac_add_options --enable-artifact-builds + + # Download debug info so that stack traces refers to file and columns rather than library and Hex address + ac_add_options --enable-artifact-build-symbols + + # Write build artifacts to: + mk_add_options MOZ_OBJDIR=./objdir-frontend-debug-artifact + + +Prerequisites +------------- + +Artifact builds are supported for users of Mercurial and Git. Git +artifact builds require a mozilla-central clone made with the help of +`git-cinnabar <https://github.com/glandium/git-cinnabar>`__. Please +follow the instructions on the git-cinnabar project page to install +git-cinnabar. Further information about using git-cinnabar to interact +with Mozilla repositories can be found on `the project +wiki <https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development>`__. + +Building +-------- + +If you've added ``--enable-artifact-builds`` to your ``mozconfig``, each +time you run ``mach build`` and ``mach build path/to/subdirectory`` the +build system will determine what the best pre-built binary artifacts +available are, download them, and put them in place for you. The +computations are cached, so the additional calculations should be very +fast after the up-to-date artifacts are downloaded -- just a second or +two on modern hardware. Most Desktop developers should find that + +.. code-block:: shell + + ./mach build + ./mach run + +just works. + +To only rebuild local changes (to avoid re-checking for pushes and/or +unzipping the downloaded cached artifacts after local commits), you can +use: + +.. code-block:: shell + + ./mach build faster + +which only "builds" local JS, CSS and packaged (e.g. images and other +asset) files. + +Most Firefox for Android developers should find that + +.. code-block:: shell + + ./mach build + ./mach package + ./mach install + +just works. + +Pulling artifacts from a try build +---------------------------------- + +To only accept artifacts from a specific revision (such as a try build), +set ``MOZ_ARTIFACT_REVISION`` in your environment to the value of the +revision that is at the head of the desired push. Note that this will +override the default behavior of finding a recent candidate build with +the required artifacts, and will cause builds to fail if the specified +revision does not contain the required artifacts. + +Pulling artifacts from local build / remote URL +----------------------------------------------- + +If you need to do an artifact build against a local build or one hosted +somewhere, you need to make use of respectively ``MOZ_ARTIFACT_FILE`` or +``MOZ_ARTIFACT_URL``. In case of a local build, you will have to make sure you + +- produce a package using ``./mach package`` +- point to it via ``MOZ_ARTIFACT_FILE=path/to/firefox.tar.bz2`` on your + ``./mach build`` command line. The path needs to be absolute, and the package + is under your object directory within ``dist/``. + +Using ``MOZ_ARTIFACT_URL`` will download the package at the given URL and then +follow the same process as the local build case. + +``MOZ_ARTIFACT_FILE`` and ``MOZ_ARTIFACT_URL`` only provide the package, they +do not provide sibling artifacts including the test artifacts, extra archives +such as XPT data, etc. In general, prefer ``MOZ_ARTIFACT_REVISION``, which +will can provide these sibling artifacts. + +Restrictions +------------ + +Oh, so many. Artifact builds are rather delicate: any mismatch between +your local source directory and the downloaded binary artifacts can +result in difficult to diagnose incompatibilities, including unexplained +crashes and catastrophic XPCOM initialization and registration +failures. These are rare, but do happen. + +Things that are supported +------------------------- + +- Modifying JavaScript, (X)HTML, and CSS resources; and string + properties and FTL files. +- Modifying Android Java code, resources, and strings. +- Running mochitests and xpcshell tests. +- Modifying ``Scalars.yaml`` to add Scalar Telemetry (since {{ + Bug("1425909") }}, except artifact builds on try). +- Modifying ``Events.yaml`` to add Event Telemetry (since {{ + Bug("1448945") }}, except artifact builds on try). + +Essentially everything updated by ``mach build faster`` should work with +artifact builds. + +Things that are not supported +----------------------------- + +- Support for products other than Firefox for Desktop and + Android are not supported and are unlikely to ever be supported. + Other projects like Thunderbird may provide + `their own support <https://developer.thunderbird.net/thunderbird-development/building-thunderbird/artifact-builds>`__ + for artifact builds. +- You cannot modify C, C++, or Rust source code anywhere in the tree. + If it’s compiled to machine code, it can't be changed. +- You cannot modify ``histograms.json`` to add Telemetry histogram + definitions.(But see `Bug 1206117 <https://bugzilla.mozilla.org/show_bug.cgi?id=1206117>`__). +- Modifying build system configuration and definitions does not work in + all situations. + +Things that are not **yet** supported +------------------------------------- + +- Tests other than mochitests, xpcshell, and Marionette-based tests. + There aren’t inherent barriers here, but these are not known to work. +- Modifying WebIDL definitions, even ones implemented in JavaScript. + +Troubleshooting +--------------- + +There are two parts to artifact mode: +the ``--disable-compile-environment`` option, and the ``mach artifact`` +command that implements the downloading and caching. Start by running + +.. code-block:: shell + + ./mach artifact install --verbose + +to see what the build system is trying to do. There is some support for +querying and printing the cache; run ``mach artifact`` to see +information about those commands. + +Downloaded artifacts are stored in +``$MOZBUILD_STATE_PATH/package-frontend``, which is almost always +``~/.mozbuild/package-frontend``. + +Discussion is best started on the `dev-builds mailing +list <https://lists.mozilla.org/listinfo/dev-builds>`__. Questions are +best raised in `#build <https://chat.mozilla.org/#/room/#build:mozilla.org>`__ on `Matrix <https://chat.mozilla.org/>`__. Please +file bugs in *Firefox Build System :: General*, blocking `Bug 901840 <https://bugzilla.mozilla.org/show_bug.cgi?id=901840>`__ diff --git a/docs/contributing/build/building_mobile_firefox.rst b/docs/contributing/build/building_mobile_firefox.rst new file mode 100644 index 0000000000..90e922af8a --- /dev/null +++ b/docs/contributing/build/building_mobile_firefox.rst @@ -0,0 +1,34 @@ +Firefox for Mobile Devices +-------------------------- + +We have several different mobile products aimed at different tasks, +devices, and audiences: + +- Building **Firefox for Android** (codename: fenix). Our general-purpose + mobile browser is split into several different artifact layers: + + - `The fenix Android application </mobile/android/fenix.html>`_ + - `The android-components Android library <https://github.com/mozilla-mobile/firefox-android/tree/main/android-components>`_ + - `The GeckoView platform </mobile/android/geckoview>`_ + +- `Firefox for iOS <https://github.com/mozilla-mobile/firefox-ios>`_, + our general-purpose browser for iOS with desktop sync built-in. +- Building **Firefox Focus**, our privacy-focused browser for + + - `iOS <https://github.com/mozilla-mobile/focus-ios>`_ + - `Android <https://github.com/mozilla-mobile/firefox-android/tree/main/focus-android>`_. This browser + also uses the android-components library and GeckoView platform, like Firefox for Android + +For both Desktop and Mobile development, please bear the following in +mind: + +- While you can build Firefox on older hardware it can take quite a bit + of time to compile on slower machines. Having at least 8GB of RAM is + recommended, and more is always better. The build process is both CPU + and I/O intensive, so building on a machine with an SSD is also + strongly preferred. +- Fast broadband internet is strongly recommended as well. Both the + development environment and the source code repository are quite + large. +- Though you can build Firefox to run on 32-bit machines, the build + process for almost all of our products requires a 64-bit OS. diff --git a/docs/contributing/build/supported.rst b/docs/contributing/build/supported.rst new file mode 100644 index 0000000000..4223f812ff --- /dev/null +++ b/docs/contributing/build/supported.rst @@ -0,0 +1 @@ +.. include:: ../../build/buildsystem/supported-configurations.rst diff --git a/docs/contributing/contributing_to_mozilla.rst b/docs/contributing/contributing_to_mozilla.rst new file mode 100644 index 0000000000..347a6ecfb9 --- /dev/null +++ b/docs/contributing/contributing_to_mozilla.rst @@ -0,0 +1,63 @@ +Contributing to Mozilla projects +================================ + +This page is here to help you get from "I want to build Firefox" +to "I'm building my own Firefox" to "I can contribute to Firefox". +So if you'd like to help Mozilla build the best web browsers in the +world, you're in the right place. + +.. rubric:: Need help? + :name: Need_help + +The Mozilla community prides itself on being an open, accessible, and +friendly community for new participants. If you have any difficulties +getting involved or finding answers to your questions, please `come and +ask your questions in our +chatroom <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`_, +where we can help you get started. + +We know even before you start contributing that getting set up to work +on Firefox and finding a bug that's a good fit for your skills can be a +challenge, and we're always looking for ways to improve this process: making +Mozilla more open, accessible, and easier to participate with. If you're +having any trouble following this documentation, or hit a barrier you +can't get around, please join us in the the Introduction room on Matrix. + +What skills do I need? +---------------------- + +Mozilla maintains small and large projects and we are thrilled to have contributors with +very diverse skills: + +- If you know **C++,** **Rust,** **JavaScript,** **HTML** or **CSS**, + you can :ref:`contribute to the core layers <Firefox Contributors' Quick Reference>` of + Firefox and many other Mozilla projects. +- If you know **Rust**, you can also contribute to the `Rust programming + language <https://github.com/rust-lang/rust>`_ itself, numerous crates like `grcov <https://github.com/mozilla/grcov/>`_ + or `Servo <https://servo.org/>`_, the web browser engine designed for parallelism and safety. +- If you know **Kotlin**, you can contribute to `Firefox + for Android <https://github.com/mozilla-mobile/fenix>`_ (code name: + "Fenix"). +- If you know **Swift**, you can contribute to `Firefox for + iOS <https://github.com/mozilla-mobile/firefox-ios>`_ and `Firefox + Focus for iOS <https://github.com/mozilla-mobile/focus-ios>`_. +- If you know **C++**, you can contribute to our `VPN client <https://github.com/mozilla-mobile/mozilla-vpn-client>`_. +- If you know **Python**, you can contribute to our web services, + including Firefox Sync and Firefox Accounts. +- If you know **Make**, **shell**, **Perl**, or **Python**, you can + contribute to our build systems, release engineering, and automation. +- If you know **Go** or **JavaScript**, you can contribute to `TaskCluster + <https://github.com/taskcluster/taskcluster>`_ our CI infrastructure. +- If you know **C**, you can contribute to `NSS <https://developer.mozilla.org/docs/Mozilla/Projects/NSS>`_, + `Opus <https://opus-codec.org/>`_, and `Daala <https://wiki.xiph.org/Daala>`_. +- There are even many ways to contribute to the Mozilla mission without + programming. If getting involved in design, support, translation, + testing, or other types of contributions sparks your interest please + see the `Volunteer Opportunities + wiki <https://contribute.mozilla.org>`_ or the `Mozilla + community <https://community.mozilla.org/>`_ site. + +Perhaps you do not know programming yet, but you want to start learning? +There are `plenty of +resources <https://developer.mozilla.org/learn>`_ available on +the MDN Web Docs! diff --git a/docs/contributing/contribution_quickref.rst b/docs/contributing/contribution_quickref.rst new file mode 100644 index 0000000000..981198e566 --- /dev/null +++ b/docs/contributing/contribution_quickref.rst @@ -0,0 +1,408 @@ +Firefox Contributors' Quick Reference +===================================== + +Some parts of this process, including cloning and compiling, can take a long time even on modern hardware. +If at any point you get stuck, please don't hesitate to ask at `https://chat.mozilla.org <https://chat.mozilla.org>`__ +in the `#introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`__ channel. + +Don’t hesitate to look at the :ref:`Getting Set Up To Work On The Firefox Codebase<Getting Set Up To Work On The Firefox Codebase>` for a more detailed tutorial. + +Before you start +---------------- +Please register and create your account for + +`Bugzilla <https://bugzilla.mozilla.org/>`__ : web-based general-purpose bug tracking system. +To register with Phabricator, make sure you enable Two-Factor Authentication (My Profile >> Edit Profile & Preferences >> Two-Factor Authentication) in Bugzilla. + +`Phabricator <https://phabricator.services.mozilla.com/>`__: web-based software development collaboration tools, mainly for code review. +Please obtain an API Token (Settings >> Conduit API Tokens) + +Windows dependencies +-------------------- + +#. You need a :ref:`supported version of Windows<tier_1_hosts>`. +#. Download the `MozillaBuild Package. <https://ftp.mozilla.org/pub/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe>`__ Installation directory should be: + + .. code-block:: shell + + $ c:\mozilla-build\ + +#. Before moving on to the next steps, make sure to fulfill the :ref:`Windows prerequisites <Building Firefox On Windows>` + +.. note:: + + All the commands of this tutorial must be run in the shell provided with the MozillaBuild Package (start-shell.bat) + +:ref:`More information on building Firefox on Windows <Building Firefox On Windows>` + +Bootstrap a copy of the Firefox source code +------------------------------------------- + +You can download the source code and have Firefox automatically download and install the other dependencies it needs. The below command as per your Operating System, will download a lot of data (years of Firefox history!) then guide you through the interactive setup process. + +Downloading can take from 40 minutes to two hours (depending on your connection) and the repository should be less than 5GB (~ 20GB after the build). + +The default options are recommended. +If you're not planning to write C++ or Rust code, select :ref:`Artifact Mode <Understanding Artifact Builds>` +and follow the instructions at the end of the bootstrap for creating a mozconfig file. + +To Setup Firefox On Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: shell + + $ cd c:/ + $ mkdir mozilla-source + $ cd mozilla-source + $ wget https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py + $ python3 bootstrap.py + +More information on :ref:`building Firefox for Windows <Building Firefox On Windows>`. + +To Setup Firefox On macOS and Linux +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: shell + + $ curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O + $ python3 bootstrap.py + +More information on :ref:`building Firefox for Linux <Building Firefox On Linux>` and :ref:`building Firefox for MacOS <Building Firefox On MacOS>`. + +To set up your editor +--------------------- + +.. note:: + + Visual Studio Code is the recommended editor for Firefox development. + Not because it is better than the other editors but because we decided to + focus our energy on a single editor. + +Setting up your editor is an important part of the contributing process. Having +linting and other features integrated, saves you time and will help with reducing +build and reviews cycles. + +See our :ref:`editor page for more information about how to set up your favorite editor <Editor / IDE integration>`. + +To build & run +-------------- + +Once the System is bootstrapped, run: + +.. code-block:: shell + + $ cd mozilla-unified + $ ./mach build + +which will check for dependencies and start the build. +This will take a while; a few minutes to a few hours depending on your hardware. + +.. note:: + + If you build Firefox often, add `ac_add_options \-\-with-ccache=sccache` to .mozconfig. + sccache will significantly speed up your builds by caching compilation results. + The Firefox build system will download sccache automatically. + +.. note:: + + The default build is a compiled build with optimizations. Check out the + :ref:`mozconfig file documentation <Configuring Build Options>` + to see other build options. If you don't plan to change C++ or Rust code, + an :ref:`artifact build <Understanding Artifact Builds>` will be faster. + +To run it: + +.. code-block:: shell + + $ ./mach run + +This command will open your locally built Firefox in a new window. + +:ref:`More information about building Firefox on Linux <Building Firefox On Linux>` / :ref:`More information about building Firefox on MacOS <Building Firefox On MacOS>` + +If you encounter build errors, please reference the more detailed "Building Firefox" on your specific operating system document and specifically the "Troubleshooting" section. + +.. _write_a_patch: + +To write a patch +---------------- + +Make the changes you need in the codebase. You can look up UI text in `Searchfox <https://searchfox.org>`__ to find the right file. + +.. note:: + If you are unsure of what changes you need to make, or need help from the mentor of the bug, + please don't hesitate to use the needinfo feature ("Request information from") on `Bugzilla <https://bugzilla.mozilla.org/home>`__ to get the attention of your mentor. + + +After making your changes, visualize your changes to ensure you're including all the necessary work: + +.. code-block:: shell + + # Mercurial + # For files changed/added/removed + $ hg status + + # For detailed line changes + $ hg diff + + # Git + # For files changed/added/removed + $ git status + + # For detailed line changes + $ git diff + +Then commit your changes: + +.. code-block:: shell + + # Mercurial + $ hg commit + + # Git + $ git commit + +.. _Commit message: + +The commit message should look like: + +.. code-block:: text + + Bug xxxx - Short description of your change. r?reviewer + + Optionally, a longer description of the change. + +**Make sure you include the bug number and at least one reviewer (or reviewer group) in this format.** + +For example, here is an example of a good commit message: +"Bug 123456 - Null-check presentation shell so we don't crash when a button removes itself +during its own onclick handler. r=person" + +To :ref:`find a reviewer or a review group <Getting reviews>`, the easiest way is to run +``hg log <modified-file>`` (or ``git log <modified-file>``, if +you're using git) on the relevant files, and look who usually is +reviewing the actual changes (ie not reformat, renaming of variables, etc). + + +To visualize your patch in the repository, run: + +.. code-block:: shell + + # Mercurial + $ hg wip + + # Git + $ git show + +:ref:`More information on how to work with stack of patches <Working with stack of patches Quick Reference>` + +:ref:`More information on Mercurial <Mercurial Overview>` + +To make sure the change follows the coding style +------------------------------------------------ + +To detect coding style violations, use mach lint: + +.. code-block:: shell + + $ ./mach lint path/to/the/file/or/directory/you/changed + + # To get the autofix, add --fix: + $ ./mach lint path/to/the/file/or/directory/you/changed --fix + +:ref:`More information <Code quality>` + +To test a change locally +------------------------ + +To run the tests, use mach test with the path. However, it isn’t +always easy to parse the results. + +.. code-block:: shell + + $ ./mach test dom/serviceworkers + +To run tests based on :ref:`GTest` (C/C++ based unit tests), run: + +.. code-block:: shell + + $ ./mach gtest 'QuotaManager.*' + +To test a change remotely +------------------------- + +Running all the tests for Firefox takes a very long time and requires multiple +operating systems with various configurations. To build Firefox and run its +tests on continuous integration servers (CI), multiple :ref:`options to select tasks <Selectors>` +are available. + +To automatically select the tasks that are most likely to be affected by your changes, run: + +.. code-block:: shell + + $ ./mach try auto + +To select tasks manually using a fuzzy search interface, run: + +.. code-block:: shell + + $ ./mach try fuzzy + +To rerun the same tasks: + +.. code-block:: shell + + $ ./mach try again + +From `Treeherder <https://treeherder.mozilla.org/>`__ (our continuous integration system), it is also possible to attach new jobs. As every review has +a try CI run associated, it makes this work easier. See :ref:`attach-job-review` for +more information. + +.. note:: + + This requires `level 1 commit access <https://www.mozilla.org/about/governance/policies/commit/access-policy/>`__. + + You can ask your reviewer to submit the patch for you if you don't have that + level of access. + +:ref:`More information <Pushing to Try>` + + +To submit a patch +----------------- + +To submit a patch for review, we use a tool called `moz-phab <https://pypi.org/project/MozPhab/>`__. +To install it, run: + +.. code-block:: shell + + $ ./mach install-moz-phab + +Once you want to submit your patches (make sure you :ref:`use the right commit message <Commit message>`), run: + +.. code-block:: shell + + $ moz-phab + +It will publish all the currently applied patches to Phabricator and inform the reviewer. + +If you wrote several patches on top of each other: + +.. code-block:: shell + + $ moz-phab submit <first_revision>::<last_revision> + +`More +information on how to use Phabricator and MozPhab <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`__ + +To update the working directory +------------------------------- + +If you're finished with a patch and would like to return to the tip to make a new patch: + +.. code-block:: shell + + $ hg pull central + $ hg up central + +To update a submitted patch +--------------------------- + +It is rare that a reviewer will accept the first version of patch. Moreover, +as the code review bot might suggest some improvements, changes to your patch +may be required. + +If your patch is not loaded in your working directory, you first need to re-apply it: + +.. code-block:: shell + + $ moz-phab patch D<revision_id> + + # Or you can use the URL of the revision on Phabricator + $ moz-phab patch https://phabricator.services.mozilla.com/D<revision_id> + +Make your changes in the working folder and run: + +.. code-block:: shell + + # Or, if you need to pass arguments, e.g., changing the commit message: + $ hg commit --amend + + # Git + $ git commit --amend + +After amending the patch, you will need to submit it using moz-phab again. + +.. warning:: + + Don't use ``hg commit --amend -m`` or ``git commit --amend -m``. + + Phabricator tracks revision by editing the commit message when a + revision is created to add a special ``Differential Revision: + <url>`` line. + + When ``--amend -m`` is used, that line will be lost, leading to + the creation of a new revision when re-submitted, which isn't + the desired outcome. + +If you wrote many changes, you can squash or edit commits with the +command: + +.. code-block:: shell + + # Mercurial + $ hg histedit + + # Git + $ git rebase -i + +The submission step is the same as for the initial patch. + +:ref:`More information on how to work with stack of patches <Working with stack of patches Quick Reference>` + +Retrieve new changes from the repository +---------------------------------------- + +To pull changes from the repository, run: + +.. code-block:: shell + + # Mercurial + $ hg pull --rebase + + # Git + $ git pull --rebase + +.. _push_a_change: + +To push a change in the code base +--------------------------------- + +Once the change has been accepted and you've fixed any remaining issues +the reviewer identified, the reviewer should land the patch. + +If the patch has not landed on "autoland" (the integration branch) after a few days, +feel free to contact the reviewer and/or +@Aryx or @Sylvestre on the `#introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`__ +channel. + +The landing procedure will automatically close the review and the bug. + +:ref:`More information <How to submit a patch>` + +Contributing to GeckoView +------------------------- + +Note that the GeckoView setup and contribution processes are different from those of Firefox; +GeckoView setup and contribution docs live in `geckoview.dev <https://geckoview.dev>`__. + +More documentation about contribution +------------------------------------- + +:ref:`Contributing to Mozilla projects` + +https://mozilla-version-control-tools.readthedocs.io/en/latest/devguide/contributing.html + +https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html + +https://mikeconley.github.io/documents/How_mconley_uses_Mercurial_for_Mozilla_code diff --git a/docs/contributing/debugging/capturing_minidump.rst b/docs/contributing/debugging/capturing_minidump.rst new file mode 100644 index 0000000000..4c8f7591f5 --- /dev/null +++ b/docs/contributing/debugging/capturing_minidump.rst @@ -0,0 +1,259 @@ +Capturing a minidump +==================== + +*Minidumps* are files created by various Windows tools which record the +complete state of a program as it's running, or as it was at the moment +of a crash. Small minidumps are created by the Breakpad :ref:`crash +reporting <Crash Reporter>` tool, but sometimes that's not +sufficient to diagnose a problem. For example, if the application is +hanging (not responding to input, but hasn't crashed) then Breakpad is +not triggered, and it can be difficult to determine where the problem +lies. Sometimes a more complete form of minidump is needed to see +additional details about a crash, in which case manual capture of a +minidump is desired. + +This page describes how to capture these minidumps on Windows, to permit +better debugging. + + +Privacy and minidumps +--------------------- + +.. warning:: + + **Warning!** Unlike the minidumps submitted by Breakpad, these + minidumps contain the **complete** contents of program memory. They + are therefore much more likely to contain private information, if + there is any in the browser. For this reason, you may prefer to + generate minidumps against a `clean + profile <http://support.mozilla.com/en-US/kb/Managing%20profiles>`__ + where possible. + + +Capturing a minidump: application crash +--------------------------------------- + +To capture a full minidump for an application crash, you can use a tool called +**Debugging Tools for Windows**, which is provided by Microsoft for free. + + +Install Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please follow `these instructions +<https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/>`__. +You can install the Debugging Tools for Windows alone, but you cannot +download it alone. To install it, download Windows Software Development Kit +(SDK) first, start the installer, and then select only **Debugging Tools for +Windows** in the list of features to install. + +.. image:: img/sdk-installer.png + +The latest installer installs the tools for all CPU architectures (X86, +X64, ARM, and ARM64). You need to choose a tool of the architecture +matching Firefox you want to capture a minidump from. For example, if +you want to capture a minidump from 32-bit Firefox on 64-bit Windows, +use the X86 version of tools, not X64 tools. + +The default install path of SDK for 64-bit Windows is +``%ProgramFiles(x86)%\Windows Kits\10``. The debugging tools can be found in +the folder named **Debuggers** under your install path of SDK. + +The Debugging Tools for Windows contains both graphical and command line +debugging tools that can be used to capture a minidump. If you prefer +a graphical tool, please follow `Capture a minidump in a graphical way +<#capture-a-minidump-in-a-graphical-way>`__. If you prefer a command +line tool, please follow `Capture a minidump from the command line +<#capture-a-minidump-from-the-command-line>`__. + + +Capture a minidump in a graphical way +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#. Launch the graphical debugger named WinDbg from the Start Menu. WinDbg + icons are registered under "Windows Kits" as below. + + |WinDbg in Start Menu| + +#. Connect Firefox to the debugger. + + a. If Firefox is not already running, open the **"File"** menu on WinDbg + and choose **"Open Executable..."**. In the file chooser window that + appears, open the firefox.exe executable. If you're not sure about where + it is, please see `How to find the location of firefox.exe + <#how-to-find-the-location-of-firefox-exe>`__. + + b. If Firefox is already running and you know which process you want to + capture a minidump from, open the **"File"** menu on WinDbg and choose + **"Attach to a Process..."**. In the "Attach to Process" dialog that + appears, select the process. To identify a process, please see + `Identify a process to attach a debugger to + <#identify-a-process-to-attach-a-debugger-to>`__. + +#. You should now see a "Command" text window with debug output at the + top and an input box at the bottom. From the menu, select + ``Debug → Go``, and Firefox should start. If the debugger spits out + some text right away and Firefox doesn't come up, select + ``Debug → Go`` again. + +#. When the program is about to crash, WinDbg will spit out more data, + and the prompt at the bottom will change from saying "``*BUSY*``" to + having a number in it. At this point, you should type + "``.dump /ma c:\temp\firefoxcrash.dmp``" -- without the quotes, but + don't forget the dot at the beginning. Once it completes, which can + take a fair while, you will have a very large file at + ``c:\temp\firefoxcrash.dmp`` that can be used to help debug your + problem. File size will depend on this size of Firefox running in + your environment, which could several GB. + +#. Ask in the relevant bug or thread how best to share this very large + file! + + +Capture a minidump from the command line +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox is not already running, open the Command Prompt and run the following +command. This command launches all Firefox processes under a debugger. This +technique is useful when you have a startup crash or when you're not sure about +which process will crash. + +To find out where firefox.exe is, please see `How to find the location +of firefox.exe <#how-to-find-the-location-of-firefox-exe>`__. + +.. code:: + + <path to debugger>\cdb.exe -g -G -o <path to firefox>\firefox.exe + + +For example, if both the debugging tools and Firefox are installed in the +default folder and you want to capture a minidump of 64-bit Firefox, +the command will be like this. Please note that you need to add double +quotes when a path contains one or more whitespaces. + +.. code:: + + "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o "C:\Program Files\Mozilla Firefox\firefox.exe" + + +If a Firefox process you want to capture a minidump from is already running, +attach the debugger to it with the following command. To identify a process, +please see `Identify a process to attach a debugger to +<#identify-a-process-to-attach-a-debugger-to>`__. + +.. code:: + + <path to debugger>\cdb.exe -g -G -p <PID> + +When the process crashes, the debugger tool captures it and waits for your +command. At this point, you should type ``.dump /ma c:\temp\firefoxcrash.dmp`` +-- don't forget the dot at the beginning. Once it completes, which can take +a fair while, you will have a very large file at ``c:\temp\firefoxcrash.dmp`` +that can be used to help debug your problem. File size will depend on this +size of Firefox running in your environment, which could several GB. + +After a minidump is generated, type ``q`` and press Enter to quit the debugger. + + +Capturing a minidump: application hang +-------------------------------------- + +On Windows Vista and Windows 7, you can follow `these +instructions <http://support.microsoft.com/kb/931673>`__ to capture a +dump file and locate it after it's been saved. + + +Identify a process to attach a debugger to +------------------------------------------ + +When you're running Firefox, even if you have only a single tab, you may have +noticed a bunch of firefox.exe instances in Task Manager. This means Firefox +consists of multiple processes. Since an application crash happens per process +and a minidump is generated per process, you need to identify which process will +crash before starting a debugger. + +Identify a process type +~~~~~~~~~~~~~~~~~~~~~~~ + +Each instance of firefox.exe has a type. To identify a process to attach +a debugger to, you need to know its process type first. + +When a crash happens, if all Firefox windows are suddenly gone and Mozilla +Crash Reporter window is opend, a process that crashed is the main process. + +.. image:: img/crashreporter.png + +When a crash happens, if you see the following page, a process that crashed +is a tab (content) process. + +.. image:: img/tabcrashed.png + +There are more process types, but there isn't an easy way to detect a crash in +a process of those types because the symptom varies. If you cannot be sure +about the type of a crashing process, terminate Firefox and launch a new +instance of Firefox under a debugger in the way described above. + +If a GPU process crashes, you may see a window is not rendered correctly as +below. Since the main process relaunches a GPU process, this symptom will be +transient and the window will be rendered correctly again. + +.. image:: img/crash-gpu.png + +If a GMP (= Gecko Media Plugin) process crashes, you will see an information +bar will be displayed below the address bar. + +.. image:: img/crash-gmp.png + +If an RDD (= Remote Data Decoder) process crashes, Firefox may stop playing +a video as below, but not limited to this symptom. + +.. image:: img/crash-rdd.png + + +Identify a process ID (PID) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you understand what type of process crashes, the next step is to get a +process ID (PID), which is a value to specify in the debugger command we +discussed above. We present two ways to get a PID here. + +The first way is to use Firefox itself. Open a new tab and go to the +**about:processes** page. This page shows the list of all processes and their +PIDs. In the example below, the PID of the main process is **6308** and the +PID of the tab process hosting a page of mozilla.org is **6748**. + +.. image:: img/about-processes.png + +The second way is to use `Process Explorer +<https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer>`__, +which is provided by Microsoft for free. You may need this technique to attach +a debugger to a hung process or when you cannot open a new tab in the existing +Firefox for some reason. + +Process Explorer is basically an advanced version of Task Manager. Since it +displays processes in a hierarchical tree view, you can easily locate the main +process, which is the parent of all child processes. In the example below, the +PID of the main process is **6308** and all processes, including +plugin-container.exe, under the main process is child processes. + +Another helpful feature of Process Explorer is that when you hover the mouse +cursor on a process, it displays a tooltip window showing the process's command +line string. For a child process of firefox.exe, the command line's last +keyword shows the process type, so you can tell which process is which process +type. In the example below, the tooltip window displays a command line string +of a GPU process. + +.. image:: img/process-explorer.png + + +How to find the location of firefox.exe +--------------------------------------- + +If you're not sure about the location of the executable file (firefox.exe) of +Firefox you run, you can find it in the **about:support** page. In the +"Application Basics" section, the path to firefox.exe is displayed in the row +of "Application Binary". + +.. image:: img/about-support.png + +.. |WinDbg in Start Menu| image:: img/windbg-in-startmenu.png + :width: 50% diff --git a/docs/contributing/debugging/debugging_a_hang_on_macos.rst b/docs/contributing/debugging/debugging_a_hang_on_macos.rst new file mode 100644 index 0000000000..ee2bed16b2 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_hang_on_macos.rst @@ -0,0 +1,10 @@ +Debugging A Hang On macOS +========================= + +See `How to Report a Hung +Firefox <https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox>`_. + +See also +~~~~~~~~ + +`Debugging on macOS <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_macOS>`__ diff --git a/docs/contributing/debugging/debugging_a_minidump.rst b/docs/contributing/debugging/debugging_a_minidump.rst new file mode 100644 index 0000000000..5638c29d85 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_minidump.rst @@ -0,0 +1,202 @@ +Debugging A Minidump +==================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +The +`minidump <http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx>`__ +file format contains data about a crash on Windows. It is used by +`rust-minidump <https://github.com/luser/rust-minidump>`__, +`Breakpad <https://wiki.mozilla.org/Breakpad>`__, and also by various +Windows debugging tools. Each minidump includes the following data. + +- Details about the exception which led to the crash. +- Information about each thread in the process: the address which was + executing and the register state at the time the process stopped. +- A list of shared libraries loaded into the process at the time of the + crash. +- The stack memory of each thread. +- The memory right around the crashing address. +- (Optional) Other memory regions, if requested by the application. +- (Optional) Other platform-specific data. + +Accessing minidumps from crash reports +-------------------------------------- + +Minidumps are not available to everyone. For details on how to gain +access and where to find minidump files for crash reports, consult the +:ref:`crash report documentation <Understanding Crash Reports>` + +Using rust-minidump's tooling +----------------------------------- + +Most of our crash-reporting infrastructure is based on rust-minidump. +The primary tool for this is the +`minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ +CLI application, which includes extensive user documentation. + +That documentation includes a +`dedicated section <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk#analyzing-firefox-minidumps>`__ +on locally analyzing Firefox crashreports and minidumps. + +If you're looking for minidump_dump, it's included as part of +minidump-stackwalk. + +Using the MS Visual Studio debugger +----------------------------------- + +#. Set up the debugger to :ref:`use the Mozilla symbol + server <Using The Mozilla Symbol Server>` and + :ref:`source server <Using The Mozilla Source Server>`. +#. Double-click on the minidump file to open it in the debugger. +#. When it loads, click the green icon in the visual studio debugger + toolbar that looks like a play button. + +For Firefox releases older than Firefox 41, you will also need to +install the relevant release of Firefox (for example from +`here <https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/>`__), +and add the directory it is in (e.g., "C:\Program Files\Mozilla +Firefox 3.6 Beta 1\") to the same dialog in which you set up the +symbol server (in case the binary location in the minidump is not the +same as the one on your machine). Note that you can install the +relevant release anywhere. Just make sure to configure the symbol +server to the directory where you installed it. For releases from 41 +onward, the binaries are available on the symbol server. + +If this doesn't work, downloading the exact build and crashreporter +symbols full files. These can be found in treeherder / build folder. +Load Visual Studio, and go to file -> open -> minidump location. Click +on "Run Native", and Visual Studio will ask for the corresponding symbol +files. For each .dll you wish to have symbols for, you must go to a +console and go to the corresponding directory. E.g. (xul.dll should go +to xul.pdf in the crashreporter symbols directory). Each directory will +have a .pd\_ file. In a command shell run: "expand /r foo.pd\_". Then +point Visual Studio to this directory. + +Then you'll be able to examine: + ++------------------+-------------------------------------------------------------------------+ +| stack trace | The debugger shows the stack trace. You can right-click on any frame | +| | in the stack, and then choose "go to disassembly" or "go to source". | +| | (Choosing "go to disassembly" from the source view might not get you | +| | to the right place due to optimizations.) When looking at the | +| | source, beware that the debugging information will associate all | +| | inlined functions as part of the line into which they were inlined, | +| | and compiler (with PGO) inlines *very* aggressively (including | +| | inlining virtual functions). You can often figure out where you | +| | *really* are by reading the disassembly. | ++------------------+-------------------------------------------------------------------------+ +| registers | In the Registers tab (Debug->Windows->Registers if you don't have | +| | it open), you can look at the registers associated with each stack | +| | frame, but only at the current state (i.e., the time of the crash). | +| | Registers that Visual Studio can't figure out will be grayed-out and | +| | have the value 00000000. | ++------------------+-------------------------------------------------------------------------+ +| stack memory | You open a window (Memory 1, etc.) that shows contiguous segments of | +| | memory using the Debug->Windows->Memory menu item. You can then | +| | enter the address of the stack pointer (ESP register) in this window | +| | and look at the memory on the stack. (The minidump doesn't have the | +| | memory on the heap.) It's a good idea to change the "width" dropdown | +| | in the top right corner of the window from its default "Auto" to | +| | either "8" or "16" so that the memory display is word-aligned. If | +| | you're interested in pointers, which is usually the case, you can | +| | right click in this window and change the display to show 4-byte | +| | words (so that you don't have to reverse the order due to | +| | little-endianness). This view, combined with the disassembly, can | +| | often be used to reconstruct information beyond what in shown the | +| | function parameters. | ++------------------+-------------------------------------------------------------------------+ +| local variables | In the Watch 1 (etc.) window (which, if you don't have open, you can | +| | iget from Debug->Windows->Watch), you can type an expression | +| | (e.g., the name of a local variable) and the debugger will show you | +| | its value (although it sometimes gets confused). If you're looking | +| | at a pointer to a variable that happens to be on the stack, you can | +| | even examine member variables by just typing expressions. If Visual | +| | Studio can't figure something out from the minidump, it might show | +| | you 00000000 (is this true?). | ++------------------+-------------------------------------------------------------------------+ + +Using minidump-2-core on Linux +------------------------------ + +The `Breakpad +source <https://chromium.googlesource.com/breakpad/breakpad/+/master/>`__ +contains a tool called +`minidump-2-core <https://chromium.googlesource.com/breakpad/breakpad/+/master/src/tools/linux/md2core/>`__, +which converts Linux minidumps into core files. If you checkout and +build Breakpad, the binary will be at +``src/tools/linux/md2core/minidump-2-core``. Running the binary with the +path to a Linux minidump will generate a core file on stdout which can +then be loaded in gdb as usual. You will need to manually download the +matching Firefox binaries, but then you can use the :ref:`GDB Python +script <Downloading symbols on Linux / Mac OS X>` to download symbols. + +The ``minidump-2-core`` source does not currently handle processing +minidumps from a different CPU architecture than the system it was +built for. If you want to use it on an ARM dump, for example, you may +need to build the tool for ARM and run it under QEMU. + +Using other tools to inspect minidump data +------------------------------------------ + +Ted has a few tools that can be built against an already-built copy of +Breakpad to do more targeted inspection. All of these tools assume you +have checked out their source in a directory next to the breakpad +checkout, and that you have built Breakpad in an objdir named +``obj-breakpad`` at the same level. + +- `stackwalk-http <https://hg.mozilla.org/users/tmielczarek_mozilla.com/stackwalk-http/>`__ + is a version of the breakpad's minidump_stackwalk that can fetch symbols + over HTTP, and also has the Mozilla symbol server URL baked in. If you + run it like ``stackwalk /path/to/dmp /tmp/syms`` it will print the stack + trace and save the symbols it downloaded in ``/tmp/syms``. Note that + symbols are only uploaded to the symbol server for nightly and + release builds, not per-change builds. +- `dump-lookup <https://hg.mozilla.org/users/tmielczarek_mozilla.com/dump-lookup/>`__ + takes a minidump and prints values on the stack that are potential + return addresses. This is useful when a stack trace looks truncated + or otherwise wrong. It needs symbol files to produce useful output, + so you will generally want to have run ``stackwalk-http`` to download + them first. +- `get-minidump-instructions <https://hg.mozilla.org/users/tmielczarek_mozilla.com/get-minidump-instructions/>`__ + retrieves and displays the memory range surrounding the faulting + instruction pointer from a minidump. You will almost always want to + run it with the ``--disassemble`` option, which will make it send the + bytes through ``objdump`` to display the disassembled instructions. + If you also give it a path to symbols (see ``stackwalk-http`` above) + it can download the matching source files from hg.mozilla.org and + display source interleaved with the disassembly. +- `minidump-modules <http://hg.mozilla.org/users/tmielczarek_mozilla.com/minidump-modules>`__ + takes a minidump and prints the list of modules from the crashed + process. It will print the full path to each module, whereas the + Socorro UI only prints the filename for each module for privacy + reasons. It also accepts a -v option to print the debug ID for each + module, and a -d option to print relative paths to the symbol files + that would be used instead of the module filenames. + +Getting a stack trace from a crashed B2G process +------------------------------------------------ + +#. Get the minidump file in the phone at + /data/b2g/mozilla/\*.default/minidump/. You can use `adb + pull <http://developer.android.com/tools/help/adb.html>`__ for that. +#. Build the debug symbols using the command ./build.sh buildsymbols + inside the B2G tree. The symbol files will be generated in + $OBJDIR/dist/crashreporter-symbols. +#. Build and install + `google-breakpad <https://code.google.com/p/google-breakpad/>`__. +#. Use the + `minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ + tool to get the stack trace. + +.. code:: bash + + Example: + + $ cd B2G + $ adb pull /data/b2g/mozilla/*.default/minidump/*.dmp . + $ls *.dmp + 71788789-197e-d769-67167423-4e7aef32.dmp + $ minidump-stackwalk 71788789-197e-d769-67167423-4e7aef32.dmp objdir-debug/dist/crashreporter-symbols/ diff --git a/docs/contributing/debugging/debugging_firefox_with_gdb.rst b/docs/contributing/debugging/debugging_firefox_with_gdb.rst new file mode 100644 index 0000000000..9572ce8e18 --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_gdb.rst @@ -0,0 +1,303 @@ +Debugging Firefox with GDB +========================== + +This page details how you can more easily debug Firefox with gdb. :ref:`rr +<Debugging Firefox with rr>` is most often a better choice to debug a problem, +but sometimes it isn't possible to use it, such as attempting to reproduce a +race condition or when performance is important to reproduce an issue. ``rr`` +chaos mode allows reproducing a lot of issues though, and should be tried. + +Where can I find general gdb documentation? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using GDB is beyond the scope of this document. Documentation is likely +available on your system if you have GDB installed, in the form of +**info,** **man** pages, or the gnome help browser. Additionally, you +can use a graphical front-end to GDB like +`ddd <https://www.gnu.org/software/ddd/>`__ or +`insight <https://sourceware.org/insight/>`__. For more information see +https://sourceware.org/gdb/current/onlinedocs/gdb/ + +How to debug Firefox with gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Firefox is a multiprocess application, with a parent process, and several child +processes, each specialized and sandboxed differently. + +.. code:: bash + + $ ./mach run --debugger=gdb + +allows running Firefox in gdb, and debug the parent process. You can substitute +``run`` with another action, such as ``test``, ``mochitest``, ``xpcshell-test``, +etc. + +Debugging child processes can be done by attaching the debugger. + +.. code:: bash + + $ gdb --pid <pid> + +There's a number of ways to find the PID of a process: hovering over a tab for a +content process, opening ``about:processes``, using ``ps ... | grep firefox`` on +the command line, etc. + +Sometimes, it is desirable to attach to a child process at startup, to diagnose +something very close to the start of the process. Setting the environment +variable ``MOZ_DEBUG_CHILD_PROCESS=10`` will make each new process print an few +informative lines, including the process type and its PID. The process with then +sleep for a number of seconds equal to the value of the environment variable, +allowing to attach a debugger. + +.. code:: bash + + $ MOZ_DEBUG_CHILD_PROCESS=10 ./mach run + ... + ... + ... + + CHILDCHILDCHILDCHILD (process type tab) + debug me @ 65230 + + ... + ... + ... + +Attaching gdb to Firefox might fail on Linux distributions that enable common +kernel hardening features such as the Yama security module. If you encounter the +following error when attaching: + +.. code:: bash + + $ gdb --pid <pid> + ... + ... + + Attaching to process <pid> + ptrace: Operation not permitted. + +Check the contents of `/proc/sys/kernel/yama/ptrace_scope`. If it set to `1` +you won't be able to attach to processes, set it to `0` from a root shell: + +.. code:: bash + + \# echo 0 > /proc/sys/kernel/yama/ptrace_scope + +If you still can't attach check your setup carefully. Do not, under any +circumstances, run gdb as root. Since gdb can execute arbitrary code and spawn +shells it can be extremely dangerous to use it with root permissions. + + +Advanced gdb configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The preferred method, is using the +:ref:`mach` command-line tool to run the +debugger, which can bypass several optional defaults. Use "mach help +run" to get more details. If inside the source directory, you would use +"./mach". Please note that +:ref:`mach is aware of mozconfigs <Configuring Build Options>`. + +.. code:: bash + + $ ./mach run --debug [arguments to pass to firefox] + +If you need to direct arguments to gdb, you can use '--debugger-args' +options via the command line parser, taking care to adhere to shell +splitting rules. For example, if you wanted to run the command 'show +args' when gdb starts, you would use: + +.. code:: bash + + $ ./mach run --debug --debugger-args "-ex 'show args'" + +Alternatively, you can run gdb directly against Firefox. However, you +won't get some of the more useful capabilities this way. For example, +mach sets an environment variable (see below) to stop the JS engine from +generating synthetic segfaults to support the slower script dialoging +mechanism. + +.. code:: + + (gdb) $OBJDIR/dist/bin/firefox + +How to debug a Firefox in the field (not compiled on the host) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you need to attach to a Firefox process live on a machine, and this Firefox +was built by Mozilla, or by certain Linux distros, it's possible to get symbols +and sources using the Mozilla symbol server, see :ref:`this section <Downloading +symbols on Linux / Mac OS X>` for setup instructions, it's just a matter of +sourcing a python script in ``.gdbinit``. + +Debugging then works as usual, except the build probably has a very high +optimization level. + +How do I pass arguments in prun? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the arguments in GDB before calling prun. Here's an example on how +to do that: + +.. code:: + + (gdb) set args https://www.mozilla.org + (gdb) prun + +Why breakpoints seem to not be hit? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most likely cause is that `gdb` hasn't been attached to the process in which +the code to diagnose is ran. Enabling the relevant `MOZ_LOG` modules can help, +since by default it prints the process type and pid of all logging statements. + +`break list` will display a list of breakpoints, and whether or not they're +enabled. C++ namespaces need to be specified entirely, and it's sometimes hard +to break in lambda. Breaking by line number is an alternative strategy that +often works in this case. + +How do I display an nsString? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + (gdb) p ToNewCString(string); + +This leaks a bit of memory but it doesn't really matter. + +How do I determine the concrete type of an object pointed to by an interface pointer? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can determine the concrete type of any object pointed to, by an +XPCOM interface pointer, by looking at the mangled name of the symbol +for the object's vtable: + +.. code:: + + (gdb) p aKidFrame + $1 = (nsIFrame *) 0x85058d4 + (gdb) x/wa *(void**)aKidFrame + 0x4210d380 <__vt_14nsRootBoxFrame>: 0x0 + (gdb) p *(nsRootBoxFrame*)aKidFrame + [ all the member variables of aKidFrame ] + +Or use the gdb command ``set print object on``. + +How can I debug JavaScript from gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have JavaScript Engine code on the stack, you'll probably want a +JS stack in addition to the C++ stack. + +.. code:: + + (gdb) call DumpJSStack() + +Please note that if `gdb` has been attached to a process, the stack might be +printed in the terminal window in which Firefox was started. + +See +`this MDN page +<https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_JavaScript>`__ +for more JS debugging tricks. + +How can I debug race conditions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Try :ref:`rr <Debugging Firefox with rr>` first. If this doesn't work, good +luck, maybe try :ref:`logging <Gecko Logging>` or sprinkling assertions. + +I keep getting a SIGSYS, or SIGSEGV in JS/JIT code under gdb even though there is no crash when gdb is not attached. How do I fix it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Allow gdb to read mozilla-central's .gdbinit, located at `build/.gdbinit`. In +your own `.gdbinit`, add the line: + + .. code:: + + add-auto-load-safe-path /path/to/mozilla-central + +How do I get useful stack traces inside system libraries? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many Linux distributions provide separate packages with debugging +information for system libraries, such as gdb, Valgrind, profiling +tools, etc., to give useful stack traces via system libraries. + +The modern way to do this is to enable ``debuginfod``. This can be done by adding: + + .. code:: + + set debuginfod enabled on + +in your ``.gdbinit``, but there might be distro-specific instructions. +Alternatively, you can install the packages that contain the debug symbols for +the libraries you want to debug. + +When using ``debuginfod``, the correct information will be downloaded +automatically when needed (and subsequently cached). + +If you're not sure what to use, there's a federated debuginfod server that +provides debug information for most mainstream distributions. You can use it +by adding the following line to your ``.gdbinit`` file: + + .. code:: + + set debuginfod urls "https://debuginfod.elfutils.org/" + +Keep in mind that it might take a while to download debug information the +very first time. This queries all the servers of multiple distributions +sequentially and debug information tends to be large. It will be cached for the +next run though. + +Fedora +^^^^^^ + +On Fedora, you need to enable the debuginfo repositories, as the +packages are in separate repositories. Enable them permanently, so when +you get updates you also get security updates for these packages. A way +to do this is edit ``/etc/yum.repos.d/fedora.repo`` and +``fedora-updates.repo`` to change the ``enabled=0`` line in the +debuginfo section to ``enabled=1``. This may then flag a conflict when +upgrading to a new distribution version. You would the need to perform +this edit again. + +You can finally install debuginfo packages with yum or other package +management tools. The best way is install the ``yum-utils`` package, and +then use the ``debuginfo-install`` command to install all the debuginfo: + +.. code:: bash + + $ yum install yum-utils + $ debuginfo-install firefox + +This can be done manually using: + +.. code:: bash + + $ yum install GConf2-debuginfo ORBit2-debuginfo atk-debuginfo \ + cairo-debuginfo dbus-debuginfo expat-debuginfo \ + fontconfig-debuginfo freetype-debuginfo gcc-debuginfo glib2-debuginfo \ + glibc-debuginfo gnome-vfs2-debuginfo gtk2-debuginfo gtk2-engines-debuginfo \ + hal-debuginfo libX11-debuginfo libXcursor-debuginfo libXext-debuginfo \ + libXfixes-debuginfo libXft-debuginfo libXi-debuginfo libXinerama-debuginfo \ + libXrender-debuginfo libbonobo-debuginfo libgnome-debuginfo \ + libselinux-debuginfo pango-debuginfo popt-debuginfo scim-bridge-debuginfo + +Disabling multiprocess +~~~~~~~~~~~~~~~~~~~~~~ + +``mach run`` and ``mach test`` both accept a ``--disable-e10s`` argument. Some +debuggers can't catch child-process crashes without it. This is sometimes a +viable alternative to attaching, but these days it changes enough thing that +it's not always a usable option. + +See also +~~~~~~~~~ + + +- `Mike Conley's blog post <https://mikeconley.ca/blog/2014/04/25/electrolysis-debugging-child-processes-of-content-for-make-benefit-glorious-browser-of-firefox>`__ +- `Performance tools <https://wiki.mozilla.org/Performance:Tools>`__ +- `Fun with + gdb <https://blog.mozilla.com/sfink/2011/02/22/fun-with-gdb/>`__ by + Steve Fink diff --git a/docs/contributing/debugging/debugging_firefox_with_lldb.rst b/docs/contributing/debugging/debugging_firefox_with_lldb.rst new file mode 100644 index 0000000000..88378bcc4a --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_lldb.rst @@ -0,0 +1,80 @@ +Debugging Firefox with LLDB +=========================== + +See http://lldb.llvm.org/index.html. + +Mozilla-specific lldb settings +------------------------------ + +There's an +``.lldbinit`` `file <https://searchfox.org/mozilla-central/source/.lldbinit>`_ +in the Mozilla source tree, which applies recommended settings and +includes a few type summaries and Mozilla-specific debugging commands +via the lldbutils module (see +`python/lldbutils/README.txt <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__). +For information about available features see the links above and the `Using +LLDB to debug Gecko <http://mcc.id.au/blog/2014/01/lldb-gecko>`__ blog +post. + +The in-tree ``.lldbinit`` should be loaded automatically in most cases +when running lldb from the command line (e.g. using +:ref:`mach`), but **not** +when using Xcode. See :ref:`Debugging on macOS` for information on setting up +Xcode. + +.. warning:: + + LLDB warning: Xcode 5 only comes with lldb (gdb is gone). The + introduction and use of UNIFIED_SOURCES in the source starting around + November 2013 has broken the default LLDB configuration so that it + will not manage to resolve breakpoints in files that are build using + UNIFIED_SOURCES (the breakpoints will be listed as "pending", and + lldb will not stop at them). To fix this add the following to your + $HOME/.lldbinit file: + + .. code:: + + # Mozilla's use of UNIFIED_SOURCES to include multiple source files into a + # single compiled file breaks lldb breakpoint setting. This works around that. + # See http://lldb.llvm.org/troubleshooting.html for more. + settings set target.inline-breakpoint-strategy always + + Restart Xcode/lldb and restart your debugging session. If that still + doesn't fix things then try closing Xcode/lldb, doing a clobber + build, reopening Xcode/lldb, and restarting your debugging session. + +Starting a debugging session +---------------------------- + +Attaching to an existing process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can attach to Firefox with following command: + +.. code:: + + (lldb) process attach --name firefox + +Some versions of lldb causes crashes after attaching to Firefox. + +Running a new process +~~~~~~~~~~~~~~~~~~~~~ + +To start Firefox under the debugger, run ``lldb`` followed by "--", +followed by the command line you'd like to run, like this: + +.. code:: bash + + $ lldb -- obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox -no-remote -profile /path/to/profile + +Then set breakpoints you need and start the process: + +.. code:: + + (lldb) breakpoint set --name nsInProcessTabChildGlobal::InitTabChildGlobal + Breakpoint created: 1: name = 'nsInProcessTabChildGlobal::InitTabChildGlobal', locations = 0 (pending) + WARNING: Unable to resolve breakpoint to any actual locations. + + (lldb) r + Process 7602 launched: '/.../obj-ff-opt/dist/Nightly.app/Contents/MacOS/firefox' (x86_64) + 1 location added to breakpoint 1 diff --git a/docs/contributing/debugging/debugging_firefox_with_rr.rst b/docs/contributing/debugging/debugging_firefox_with_rr.rst new file mode 100644 index 0000000000..ef68d9c6ea --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_rr.rst @@ -0,0 +1,98 @@ +Debugging Firefox with rr +========================= + +This page is intended to help Firefox/Gecko developers get started using rr to debug Firefox. + +Prerequisites +------------- + +You must have Linux installed with a recent kernel. If you're not running Linux already, an option is set up a virtual machine in which to record Firefox. Be forewarned though that + + * rr requires a VM hypervisor that virtualizes CPU performance counters. VMWare Workstation supports that. + * there's a 20% or so performance hit from running in a VM; generally speaking recorder overhead increases from ~1.2x to ~1.4x. (It's a feather in the cap of the hypervisor authors that the hit is that small, though!) + * Some features (reverse execution) `may not work well in VMWare <https://robert.ocallahan.org/2014/09/vmware-cpuid-conditional-branch.html>`__ due to a VMWare optimization that can be disabled `this way <http://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html>`__. + +When using VSCode, consider adding the `Midas <https://github.com/farre/midas>`__ addon to debug rr traces from the editor. You can use Midas to install / build rr and to configure the :code:`auto-load safe path` via the setup commands. + +Ensure that you've `installed <http://rr-project.org/>`__ or `built <https://github.com/mozilla/rr/wiki/Building-And-Installing>`__ rr and have `used it successfully <https://github.com/mozilla/rr/wiki/Usage>`__. Check that you have the latest version. + +You likely need to configure your :code:`auto-load safe path` for rr / gdb to work correctly. If you are on Linux and your builds are in ~/moz, then add the following to ~/.gdbinit + +.. code:: + + add-auto-load-safe-path ~/moz + +Firefox developers are strongly encouraged to build rr from source. If your Firefox patch triggers a bug in rr, rr developers will fix that bug with high priority. You might be able to pull a fix within a few hours or days instead of waiting for the next release. + +Recording Firefox +----------------- + + +To record Firefox running normally, simply launch it under rr as you would if running it under valgrind or gdb + +.. code:: bash + + $ rr $ff-objdir/dist/bin/firefox ... + +or use mach + +.. code:: bash + + $ ./mach run --debugger=rr + +This will save a trace to your working directory as described in the `usage instructions <https://github.com/mozilla/rr/wiki/Usage>`__. Please refer to `those instructions <https://github.com/mozilla/rr/wiki/Usage>`__ for details on how to debug the recording, which isn't covered in this document. + +Sandboxing reduces recording performance because of the SIGSYS signals and extra syscall. Disabling it can help. + +The background hang monitor might also be making things worse by causing a lot of extra syscalls. It can be disabled by setting +toolkit.content-background-hang-monitor.disabled=true. + +SIGSYS +------ + +When recording and replaying Firefox running with the Linux sandbox, you will get SIGSYS signals frequently. This is expected behavior caused by the sandbox. In gdb, use handle SIGSYS noprint nostop to suppress the signals. + +Recording test suites +--------------------- + +You can use the test runners' --debugger feature to punch rr down through the layers of python script to where Firefox is launched. This is used in the same way you would use --debugger to run valgrind or gdb, for example: + +.. code:: bash + + $ ./mach mochitest --debugger=rr ... + +The test harnesses disable the slow-script timeout when the --debugger argument is passed. That's usually sensible, because you don't want those warnings being generated while Firefox is stopped in gdb. However, this has been `observed to change Gecko behavior <https://bugzilla.mozilla.org/show_bug.cgi?id=986673>`__. rr doesn't need to have the slow-script timeout disabled, so to avoid those kinds of pitfalls, pass the --slowscript argument to the test harness. + +To run rr in chaos mode: + +.. code:: bash + + $ ./mach mochitest --debugger=rr --debugger-args="record --chaos" + +You can also run the entire test harness in rr: + +.. code:: bash + + $ rr ./mach mochitest ... + +The trace will contain many processes, so to debug the correct one, you'll want to use rr ps or rr replay -p firefox etc. + +Working with multiple processes +------------------------------ + +rr should work out of the box with multi-process Firefox. Once you have a recording you can use rr ps to show all the process that were recorded and rr replay -p <pid> to attach to a particular process. + +If you want to debug a particular part of code, you can use the :code:`MOZ_DBG` macro and :code:`getpid()` function to write the process id to stderr. `MOZ_LOG <https://firefox-source-docs.mozilla.org/xpcom/logging.html>`__ will include the pid in log messages by default. + +You can combine that with the -M and -g flags to jump to a particular point in a particular process's lifetime. + +Get help! +--------- + +If you encounter a problem with rr, please `file an issue <https://github.com/mozilla/rr/issues>`__. Firefox bugs are high priority, so usually your issue can be fixed very quickly. + +If you want to chat with rr developers, because you need more help or want to contribute or want to complain, we hang out in the `#rr channel <https://chat.mozilla.org/#/room/#rr:mozilla.org>`__. There is also a channel for `#midas <https://chat.mozilla.org/#/room/#midas:mozilla.org>`__. + +You also may find `these debugging protips <https://github.com/mozilla/rr/wiki/Debugging-protips>`__ helpful, though many are for rr developers, not users. + +Happy debugging! diff --git a/docs/contributing/debugging/debugging_firefox_with_valgrind.rst b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst new file mode 100644 index 0000000000..33aab3638a --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst @@ -0,0 +1,177 @@ +Debugging Firefox with Valgrind +=============================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This page describes how to use Valgrind (specifically, its Memcheck +tool) to find memory errors. + +Supported platforms +------------------- + +Valgrind runs desktop Firefox fine on Linux, especially on x86 and +x86-64. Firefox for Android and Firefox OS on ARMv7 should also run, +though perhaps not as smoothly. The other architectures supported by +Valgrind on Linux (AARCH64, PPC{32,64}, MIPS{32,64}, S390X) should also +work, in theory. + +MacOS X 10.10 (Yosemite), 64-bit only, works, although it can be a bit +of a rough ride. + +- Expect lower performance and a somewhat higher false positive error + rate than on Linux. +- Valgrind's handling of malloc zones on Yosemite is imperfect. Regard + leak reports with caution. +- Valgrind has been known to cause kernel panics, for unknown reasons. + +Where to get Valgrind +--------------------- + +Linux: Download `Valgrind <https://valgrind.org/>`__ directly, or use +your distribution's package manager (if it has a recent enough version). + +MacOSX: `Get Valgrind trunk from +SVN <https://valgrind.org/downloads/repository.html>`__ and build it. +Don't use 3.10.x or any other tarball. + +Make sure you have Valgrind 3.14 or later, version 3.16.1 is known to work, +3.13.0 did not. Newer versions tend to have better compatibility with both +Firefox's JITs and newer toolchain components (compiler, libc and linker +versions). + +Basics +------ + +Build +~~~~~ + +Build Firefox with the following options, which maximize speed and +accuracy. + +.. code:: + + ac_add_options --disable-jemalloc + ac_add_options --disable-strip + ac_add_options --enable-valgrind + ac_add_options --enable-optimize="-g -O2" + ac_add_options --disable-sandbox + +Run +~~~ + +Note that programs run *much* more slowly under Valgrind than they do +natively. Slow-downs of 20x or 30x aren't unexpected, and it's slower on +Mac than on Linux. Don't try this on an underpowered machine. + +Linux +^^^^^ + +On Linux, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --read-inline-info=yes + +The ``--smc-check`` and ``--vex-iropt-register-updates`` options are +necessary to avoid crashes in JIT-generated code. + +The ``--show-mismatched-frees`` option is necessary due to inconsistent +inlining of ``new`` and ``delete`` -- i.e. one gets inlined but the +other doesn't -- which lead to false-positive mismatched-free errors. + +The ``--read-inline-info`` option improves stack trace readability in +the presence of inlining. + +Also, run with the following environment variable set. + +.. code:: + + G_SLICE=always-malloc + +This is necessary to get the Gnome system libraries to use plain +``malloc`` instead of pool allocators. + +Mac +^^^ + +On Mac, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --dsymutil=yes + +The ``--dsymutil`` option ensures line number information is present in +stack traces. + +Advanced usage +-------------- + +Shared suppression files +~~~~~~~~~~~~~~~~~~~~~~~~ + +`/build/valgrind/ <https://searchfox.org/mozilla-central/source/build/valgrind/>`__ +contains the suppression files used by the periodic Valgrind jobs on +Treeherder. Some of these files are platform-specific. + +Running mochitests under Valgrind? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To run a mochitest under Valgrind, use the following command. + +.. code:: bash + + $ ./mach mochitest-plain --debugger="valgrind" --debugger-args="$VALGRIND_OPTIONS" relative/path/to/tests + +Where ``$VALGRIND_OPTIONS`` are the options described +:ref:`above <Debugging Firefox With Valgrind>`. You might also +need ``--trace-children=yes`` to trace into child processes. + +As of December 2014 it is possible to do a complete run of +mochitests-plain on Valgrind in about 8 CPU hours on a Core i4910 +(Haswell) machine. Maximum process size is 5.4G, of which about 80% is +in memory. Runs of small subsets of mochitests take far less memory. + +Bits and pieces +~~~~~~~~~~~~~~~ + +For un-released Linux distros (Fedora Rawhide, etc.) you'll need to use +a version of Valgrind trunk build, because fixes for the latest gcc and +glibc versions appear there first. Without them you'll be flooded with +false errors from Memcheck, and have debuginfo reading problems. + +On Linux, code compiled by LLVM at high optimisation levels can cause +Memcheck to report false uninitialised value errors. See +`here <https://bugs.kde.org/show_bug.cgi?id=242137#c3>`__ for an easy +workaround. On Mac, Valgrind has this workaround built in. + +You can make stack traces easier to read by asking for source file names +to be given relative to the root of your source tree. Do this by using +``--fullpath-after=`` to specify the rightmost part of the absolute path +that you don't want to see. For example, if your source tree is rooted +at ``/home/sewardj/MC-20-12-2014``, use ``--fullpath-after=2014/`` to +get path names relative to the source directory. + +The ``--track-origins=yes`` slows down Valgrind greatly, so don't use it +unless you are hunting down a specific uninitialised value error. But if +you are hunting down such an error, it's extremely helpful and worth +waiting for. + +Additional help +--------------- + +The `Valgrind Quick Start +Guide <http://www.valgrind.org/docs/manual/quick-start.html>`__ is short +and worth reading. The `User +Manual <https://valgrind.org/docs/manual/manual.html>`__ is also useful. + +If Valgrind asserts, crashes, doesn't do what you expect, or otherwise +acts up, first of all read this page and make sure you have both Firefox +and Valgrind correctly configured. If that's all OK, try using the +`Valgrind trunk from +SVN <http://www.valgrind.org/downloads/repository.html>`__. Oftentimes +bugs are fixed in the trunk before most users fall across them. If that +doesn't help, consider `filing a bug +report <http://www.valgrind.org/support/bug_reports.html>`__, and/or +mailing Julian Seward or Nick Nethercote. diff --git a/docs/contributing/debugging/debugging_on_macos.rst b/docs/contributing/debugging/debugging_on_macos.rst new file mode 100644 index 0000000000..036d5df550 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_macos.rst @@ -0,0 +1,356 @@ +Debugging On macOS +================== + +This document explains how to debug Gecko-based applications such as +Firefox, Thunderbird, and SeaMonkey on macOS using Xcode. If you want to +debug from the terminal see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. For specific +information on a way to debug hangs, see :ref:`Debugging a hang on macOS <Debugging A Hang On macOS>`. + +Creating a debuggable build +--------------------------- + +First, you need to build the application you're going to debug using +this in your .mozconfig + +.. code:: + + ac_add_options --disable-optimize + ac_add_options --enable-debug-symbols + +you can also add this flag if you want assertions etc. compiled in + +.. code:: + + ac_add_options --enable-debug + +See :ref:`Building Firefox for macOS <Building Firefox On MacOS>` +if you need help creating your own build. + +Debugging Firefox on macOS 10.14+ +--------------------------------- + +macOS 10.14 introduced Notarization and Hardened Runtime features for +improved application security. macOS 10.15 went further, requiring +applications to be Notarized with Hardened Runtime enabled in order to +launch (ignoring workarounds). When run on earlier macOS versions, +Notarization and Hardened Runtime settings have no effect. + +Official Builds +~~~~~~~~~~~~~~~ + +At this time, official builds of Firefox 69 and later are Notarized. +**As a result, it is not possible to attach a debugger to these official +Firefox releases on macOS 10.14+ without disabling System Integrity +Protection (SIP).** This is due to Notarization requiring Hardened +Runtime to be enabled with the ``com.apple.security.get-task-allow`` +entitlement disallowed. **Rather than disabling SIP (which has security +implications), it is recommended to debug with try builds or local +builds. The differences are explained below.** + +try Server Builds +~~~~~~~~~~~~~~~~~ + +In most cases, developers needing to debug a build as close as possible +to the production environment should use a :ref:`try +build <Pushing to Try>`. These +builds enable Hardened Runtime and only differ from production builds in +that they are not Notarized which should not otherwise affect +functionality, (other than the ability to easily launch the browser on +macOS 10.15+ -- see quarantine note below). At this time, developers can +obtain a Hardened Runtime build with the +``com.apple.security.get-task-allow`` entitlement allowed by submitting +a try build and downloading the dmg generated by the "Rpk" shippable +build job. A debugger can be attached to Firefox processes of these +builds. try builds use the developer entitlement files from the +source tree (allowing debugger attach) while production builds use +the production versions (which must meet notarization requirements). +**On macOS 10.15+, downloaded try builds will not launch by default +because Notarization is required. To workaround this problem, remove the +quarantine extended attribute from the downloaded Nightly:** + + ``$ xattr -r -d com.apple.quarantine /Path/to/Nightly.app`` + +Local Builds +~~~~~~~~~~~~ + +Local builds of mozilla-central do not enable Hardened Runtime and hence +do not have debugging restrictions. As a result, some functionality will +be permitted on local builds, but blocked on production builds which +have Hardened Runtime enabled. `Bug +1522409 <https://bugzilla.mozilla.org/show_bug.cgi?id=1522409>`__ was +filed to automate codesigning local builds to enable Hardened Runtime by +default and eliminate this discrepancy. + +Disabling System Integrity Protection (SIP) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If debugging a production build is required, follow Apple's documented +steps for disabling System Integrity Protection (SIP). Note that +disabling SIP bypasses Hardened Runtime restrictions which can mask some +bugs that only occur with Hardened Runtime so it is recommended to test +fixes with SIP enabled. **Disabling SIP has system security implications +that should be understood before taking this step.** + +Creating an Xcode project +------------------------- + +If you try to create a new Xcode project in an existing directory +then Xcode will delete its existing contents (Xcode will warn you +beforehand). To work around that, the steps below have you initialize +the project outside the Mozilla source tree, close the project, copy +the .xcodeproj project "file" into the source tree, and then reopen +the project to finish setting it up. + +Note also that since Xcode 7.3.1 it doesn't seem to be possible to +have the Xcode project live outside the source tree. If you try to do +that then Xcode will simply **copy** the source files under the +project directory rather than link to them which breaks debugging and the +possibility to modify-rebuild-relaunch from inside Xcode. + +These steps were last updated for Xcode 15: + +#. Open Xcode, and create a new Project with File > New > Project. + Select the "Other" tab then select the "Empty" project type, then + click Next. Name the project and click Next again. Xcode will refuse + to create a new project in a non-empty directory, so create/select + a temporary directory to contain the project files, untick "Create + Git repository" and then click Create. +#. Before going any further, close the project (File > Close Project). + Now open Finder, find the \*.xcodejproj directory in the temporary + directory, move it into the root directory of your Mozilla source + tree, then double-click on it to reopen it. (You can now delete the + temporary directory.) +#. In the left-hand pane in Xcode you should see a tree item where the + root item has the project name. Now, right click on the root item, + select 'Add files to "<project-name>"', select all the files and + directories in your source directory, untick "Copy items if needed", + then click Add. (These will then be progressively added under the + root item <project-name> in the left-hand pane. Note that + subdirectories may initially appear to be empty, but they too will + progressively be populated as Xcode processes the sourse files. + Once done, you should be able to open any file quickly by hitting + Cmd-Shift-O and typing in the name of a file.) +#. In the Product menu, select Scheme > New Scheme and name your scheme + (for example, "Debug"). After you click OK, Xcode should open the + settings window for the new scheme. (If not, then open its settings + from the Product > Edit Scheme menu.) +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on "None" and + selecting "Other...". A new dialog titled "Choose an executable to + launch" will pop up. Browse to the ``.app`` file that you want to + debug (``Firefox.app``, ``Nightly``\ ``Debug.app`` etc). The ``.app`` + file is typically found inside the ``dist`` folder in your build + directory. +#. If you are debugging Firefox, Thunderbird, or some other application + that supports multiple profiles, using a separate profile for + debugging purposes is recommended. See "Having a profile for + debugging purposes" below. Select the "Arguments" tab in the scheme + editor, and click the '+' below the "Arguments passed on launch" + field. Add "-P *profilename*", where *profilename* is the name of a + profile you created previously. Repeat that to also add the argument + "-no-remote". +#. Also in the "Arguments" panel, you may want to add an environment + variable MOZ_DEBUG_CHILD_PROCESS set to the value 1 to help with + debugging e10s. +#. Select "Build" from the left of the scheme editor window, and check + that there is nothing listed under Targets (otherwise it may cause + problems when you try to run the executable for debugging since you + will get build errors). +#. Click "Close" to close the scheme editor. + +At this point you can run and debug the Mozilla application from Xcode +(if the application doesn't appear to open, check whether it opened in +the background). When it hits breakpoints, Xcode should open the +correct source file at the correct line. + +Setting up lldb +--------------- + +``lldb`` is the debugger Xcode provides/uses. + +.. warning:: + + One important issue that the Mozilla .lldbinit file fixes is that by + default some breakpoints will be listed as "pending", and Xcode will + not stop at them. If you don't include the Mozilla's .lldbinit, you + must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Firefox with + lldb <Debugging Firefox with LLDB>`. + +The +`.lldbinit <http://searchfox.org/mozilla-central/source/.lldbinit>`__ +file in the source tree imports many useful `Mozilla specific lldb +settings, commands and +formatters <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__ +into ``lldb``, but you may need to take one of the following steps to +make sure this file is used. + +If you are using ``lldb`` on the command line (independently of Xcode) +and you will always run it from either the top source directory, the +object directory or else the dist/bin subdirectory of the object +directory, then adding the following setting to your ``$HOME/.lldbinit`` +is sufficient: + +:: + + settings set target.load-cwd-lldbinit true + +*However*, if you will run lldb from a different directory, or if you +will be running it indirectly by debugging in Xcode (Xcode always runs +lldb from "/"), then this setting will not help you. Instead, add the +following to your ``$HOME/.lldbinit``: + +:: + + # This automatically sources the Mozilla project's .lldbinit as soon as lldb + # starts or attaches to a Mozilla app (that's in an object directory). + # + # This is mainly a workaround for Xcode not providing a way to specify that + # lldb should be run from a given directory. (Xcode always runs lldb from "/", + # regardless of what directory Xcode was started from, and regardless of the + # value of the "Custom working directory" field in the Scheme's Run options. + # Therefore setting `settings set target.load-cwd-lldbinit true` can't help us + # without Xcode providing that functionality.) + # + # The following works by setting a one-shot breakpoint to break on a function + # that we know will both run early (which we want when we start first start the + # app) and run frequently (which we want so that it will trigger ASAP if we + # attach to an already running app). The breakpoint runs some commands to + # figure out the object directory path from the attached target and then + # sources the .lldbinit from there. + # + # NOTE: This scripts actions take a few seconds to complete, so the custom + # formatters, commands etc. that are added may not be immediately available. + # + breakpoint set --name nsThread::ProcessNextEvent --thread-index 1 --auto-continue true --one-shot true + breakpoint command add -s python + # This script that we run does not work if we try to use the global 'lldb' + # object, since it is out of date at the time that the script runs (for + # example, `lldb.target.executable.fullpath` is empty). Therefore we must + # get the following objects from the 'frame' object. + target = frame.GetThread().GetProcess().GetTarget() + debugger = target.GetDebugger() + + # Delete our breakpoint (not actually necessary with `--one-shot true`): + target.BreakpointDelete(bp_loc.GetBreakpoint().GetID()) + + # For completeness, find and delete the dummy breakpoint (the breakpoint + # lldb creates when it can't initially find the method to set the + # breakpoint on): + # BUG WORKAROUND! GetID() on the *dummy* breakpoint appears to be returning + # the breakpoint index instead of its ID. We have to add 1 to correct for + # that! :-( + dummy_bp_list = lldb.SBBreakpointList(target) + debugger.GetDummyTarget().FindBreakpointsByName("nsThread::ProcessNextEvent", dummy_bp_list) + dummy_bp_id = dummy_bp_list.GetBreakpointAtIndex(0).GetID() + 1 + debugger.GetDummyTarget().BreakpointDelete(dummy_bp_id) + + # "source" the Mozilla project .lldbinit: + os.chdir(target.executable.fullpath.split("/dist/")[0]) + debugger.HandleCommand("command source -s true " + os.path.join(os.getcwd(), ".lldbinit")) + DONE + +see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. for more information. + +Having a profile for debugging purposes +--------------------------------------- + +It is recommended to create a separate profile to debug with, whatever +your task, so that you don't lose precious data like Bookmarks, saved +passwords, etc. So that you're not bothered with the profile manager +every time you start to debug, expand the "Executables" branch of the +"Groups & Files" list and double click on the Executable you added for +Mozilla. Click the plus icon under the "Arguments" list and type "-P +<profile name>" (e.g. "-P MozillaDebug"). Close the window when you're +done. + +Running a debug session +----------------------- + +Make sure breakpoints are active (which implies running under the +debugger) by opening the Product menu and selecting "Debug / Activate +Breakpoints" (also shown by the "Breakpoints" button in the top right +section of the main window). Then click the "Run" button or select "Run" +from the Product menu. + +Setting breakpoints +~~~~~~~~~~~~~~~~~~~ + +Setting a breakpoint is easy. Just open the source file you want to +debug in Xcode, and click in the margin to the left of the line of code +where you want to break. + +During the debugging session, each time that line is executed, the +debugger will break there, and you will be able to debug it. + +.. warning:: + + Note that with the default configuration, some breakpoints will be + listed as "pending", and Xcode will not stop at them. If you don't + include the Mozilla's .lldbinit, you must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with + lldb <Debugging Firefox with LLDB>`. + +Using Firefox-specific lldb commands +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you included the .lldbinit when `Setting up +lldb <#setting-up-lldb>`__, you can use Mozilla-specific lldb commands +in the console, located in the Debug area of Xcode. For example, type +``js`` to see the JavaScript stack. For more information, see :ref:`Debugging +Mozilla with lldb <Debugging Firefox with LLDB>`. + +Debugging e10s child processes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using Xcode to debug child processes created by an e10s-enabled browser +is a little trickier than debugging a single-process browser, but it can +be done. These directions were written using Xcode 6.3.1 + +#. Complete all the steps above under "Creating the Project" +#. From the "Product" menu, ensure the scheme you created is selected + under "Scheme", then choose "Scheme > Edit Scheme" +#. In the resulting popup, click "Duplicate Scheme" +#. Give the resulting scheme a more descriptive name than "Copy of + Scheme" +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on the + "Executable" drop-down, and selecting the ``plugin-container.app`` + that is inside the app bundle of the copy of Firefox you want to + debug. +#. On the same tab, under "Launch" select "Wait for executable to be + launched" +#. On the "Arguments" tab, remove all arguments passed on launch. + +Now you're ready to start debugging: + +#. From the "Product" menu, ensure the scheme you created above is + selected under "Scheme" +#. Click the "Run" button. The information area at the top of the window + will show "Waiting for plugin-container to launch" +#. From a command line, run your build of Firefox. When that launches a + child process (for example, when you start to load a webpage), Xcode + will notice and attach to that child process. You can then debug the + child process like you would any other process. +#. When you are done debugging, click the "Stop" button and quit the + instance of Firefox that you were debugging in the normal way. + +For some help on using lldb see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. + +Other resources +--------------- + +Apple has an extensive list of `debugging tips and +techniques <https://developer.apple.com/library/mac/#technotes/tn2124/_index.html>`__. + +Questions? Problems? +~~~~~~~~~~~~~~~~~~~~ + +Try asking in our Element channels +`#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ or +`#macdev <https://chat.mozilla.org/#/room/#macdev:mozilla.org>`__. diff --git a/docs/contributing/debugging/debugging_on_windows.rst b/docs/contributing/debugging/debugging_on_windows.rst new file mode 100644 index 0000000000..3213299199 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_windows.rst @@ -0,0 +1,330 @@ +Debugging On Windows +==================== + +This document explains how to debug Gecko based applications such as +Firefox, Thunderbird, and SeaMonkey on Windows using the Visual Studio IDE. + +If VS and your Gecko application hang shortly after you launch the +application under the debugger, see `Problems Loading Debug +Symbols <#problems-loading-debug-symbols>`__. + +Ways to start the debugger +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First of all, it's necessary to install a Visual Studio extension to be +able to follow child processes as they are created. Firefox, in general, +and even in non-e10s mode, does not start the main process directly, it +starts it via a Launcher Process. This means that Visual Studio will +only attach to the first process it finds, and will not hit any +break-point (and even notifies you that it cannot find their location). +`Microsoft Child Process Debugging Power +Tool <https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool>`__ +allows automatically attaching to child processes, such as Web Content +process, GPU process, etc. Enable it by going its configuration menu in +"Debug > Other debugging targets > Child process debugging settings", +and ticking the box. + +If you have followed the steps in :ref:`Building Firefox for +Windows <Building Firefox On Windows>` +and have a local debug build, you can **execute this command from same command line.** + +.. code:: + + ./mach run --debug + +It would open Visual Studio with Firefox's +run options configured. You can **click "Start" button** to run Firefox +then, already attached in the debugger. + +Alternatively, if you have generated the Visual Studio solution, via +``./mach build-backend -b VisualStudio``, opening this solution allows +you to run ``firefox.exe`` directly in the debugger. To make it the +startup project, right click on the project and select ``Set As Startup +Project``. It appears bold when it's the case. Breakpoints are kept +across runs, this can be a good way to debug startup issues. + +**Run the program until you hit an assertion.** You will get a dialog +box asking if you would like to debug. Hit "Cancel". The MSDEV IDE will +launch and load the file where the assertion happened. This will also +create a Visual Studio Mozilla project in the directory of the executable +by default. + +**Attach the debugger to an existing Mozilla process**. In the Visual +Studio, select Debug > Attach to Process. If you want to debug a content +process, you can **hover on the tab** of page you want to debug, which +would show the pid. You can then select the process from dialog opened +from "Attach to Process". You can open ``about:processes`` to see the pid +for all subprocesses, including tabs but also GPU, networking etc. +For more information, see `Attach to Running Processes with the Visual Studio +Debugger <http://msdn.microsoft.com/en-us/library/vstudio/3s68z0b3.aspx>`__. + +**Starting an MSIX installed Firefox with the debugger**. In Visual +Studio, select Debug -> Other Debug Targets -> Debug Installed App Package. +In the dialog, select the installed Firefox package you wish to debug +and click "Start". + +Debugging Release and Nightly Builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the steps to :ref:`use the Mozilla symbol +server <Using The Mozilla Symbol Server>` and :ref:`source +server <Using The Mozilla Source Server>` + +Creating a Visual Studio project for Firefox +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please refer to :ref:`this <Visual Studio Projects>`. + +Changing/setting the executable to debug +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To change or set the executable to debug, go to Project > Properties > +Debugging > Command. (As of Visual Studio 2022.) + +It should show the executable you are debugging. If it is empty or +incorrect, manually add the correct path to the executable. + +Command line parameters and environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To change or set the command line options, go to Project > Properties > +Debugging > Command Arguments. + +Some common options would be the URL of the file you want the browser to +open as soon as it starts, starting the Profile Manager, or selecting a +profile. You can also redirect the console output to a file (by adding +"``> filename.txt``" for example, without the quotes). + +Customizing the debugger's variable value view +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can customize how Visual Studio displays classes in the variable view. +By default VS displays "{...}" and you need to click the small + icon +to expand the members. You can change this behaviour, and make Visual +Studio display whatever data member you want in whatever order, formatted +however you like instead of just "{...}". + +You need to locate a file called "gecko.natvis" under toolkit/library. +The file contains a list of types and how they should be displayed in +the debugger. It is XML and after a little practice you should be well +on your way. + +To understand the file in detail refer to `Create custom views of C++ +objects in the debugger using the Natvis framework +<https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects>`__ + +The file already comes with a number of entries that will make your life +easier, like support for several string types. If you need to add a custom +type, or want to change an existing entry for debugging purposes, you can +easily edit the file. For your convenience it is included in all generated +Visual Studio projects, and if you edit and save it within Visual Studio, it +will pick up the changes immediately. + +Handling multiple processes in Visual Studio +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Turn off "Break all processes when one process breaks" to single step a single +process. + +Turning off "Break all processes when one process breaks" adds "Step Into +Current Process", "Step Over Current Process" and "Step Out Current Process" to +the "Debug" menu. + +To single step a single process with the other processes paused: + +- Turn on "Break all processes when one process breaks" +- Hit a breakpoint which stops all processes +- Turn off "Break all processes when one process breaks" +- Now using "Step Into Current Process" will leave the other processes stopped + and just advance the current one. + +Obtaining ``stdout`` and other ``FILE`` handles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Running the following command in the Command Window in Visual Studio +returns the value of ``stdout``, which can be used with various +debugging methods (such as ``nsGenericElement::List``) that take a +``FILE*`` param: + +.. code:: + + Debug.EvaluateStatement {,,msvcr80d}(&__iob_func()[1]) + +(Alternatively you can evaluate ``{,,msvcr80d}(&__iob_func()[1])`` in +the Immediate window) + +Similarly, you can open a file on the disk using ``fopen``: + +.. code:: + + >Debug.EvaluateStatement {,,msvcr80d}fopen("c:\\123", "w") + 0x10311dc0 { ..snip.. } + >Debug.EvaluateStatement ((nsGenericElement*)0x03f0e710)->List((FILE*)0x10311dc0, 1) + <void> + >Debug.EvaluateStatement {,,msvcr80d}fclose((FILE*)0x10311dc0) + 0x00000000 + +Note that you may not see the debugging output until you flush or close +the file handle. + +Disabling ASSERTIONS +~~~~~~~~~~~~~~~~~~~~ + +There are basically two ways to disable assertions. One requires setting +an environment variable, while the other affects only the currently +running program instance in memory. + +Environment variable +^^^^^^^^^^^^^^^^^^^^ + +There is an environment variable that can disable breaking for +assertions. This is how you would normally set it: + +.. code:: + + set XPCOM_DEBUG_BREAK=warn + +The environment variable takes also other values besides ``warn``, see +``XPCOM_DEBUG_BREAK`` for more details. + +Note that unlike Unix, the default for Windows is not warn, it's to pop +up a dialog. To set the environment variable for Visual Studio, use +Project > Properties > Debugging > Environment and click the little box. +Then use + +.. code:: + + XPCOM_DEBUG_BREAK=warn + +Changing running code +^^^^^^^^^^^^^^^^^^^^^ + +You normally shouldn't need to do this (just quit the application, set +the environment variable described above, and run it again). And this +can be **dangerous** (like **trashing your hard disc and corrupting your +system**). So unless you feel comfortable with this, don't do it. **You +have been warned!** + +It is possible to change the interrupt code in memory (which causes you +to break into debugger) to be a NOP (no operation). + +You do this by running the program in the debugger until you hit an +assertion. You should see some assembly code. One assembly code +instruction reads "int 3". Check the memory address for that line. Now +open memory view. Type/copy/drag the memory address of "int 3" into the +memory view to get it to update on that part of the memory. Change the +value of the memory to "90", close the memory view and hit "F5" to +continue. + +Automatically handling ASSERTIONS without a debugger attached +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When an assertion happens and there is not a debugger attached, a small +helper application +(```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__) +is run. That application can automatically select a response to the "Do +you want to debug" dialog instead of prompting if you configure it, for +more info, see +```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__. + +Debugging optimized builds +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To effectively debug optimized builds, you should enable debugging +information which effectively leaves the debug symbols in optimized code +so you can still set breakpoints etc. Because the code is optimized, +stepping through the code may occasionally provide small surprises when +the debugger jumps over something. + +You need to make sure this configure parameter is set: + +.. code:: + + ac_add_options --enable-debug + +You can also choose to include or exclude specific modules. + +Console debugging +~~~~~~~~~~~~~~~~~ + +When printing to STDOUT from a content process, the console message will +not appear on Windows. One way to view it is simply to disable e10s +(``./mach run --disable-e10s``) but in order to debug with e10s enabled +one can run + +:: + + ./mach run ... 2>&1 | tee + +It may also be necessary to disable the content sandbox +(``MOZ_DISABLE_CONTENT_SANDBOX=1 ./mach run ...``). + +Running two instances of Mozilla simultaneously +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can run two instances of Mozilla (e.g. debug and optimized) +simultaneously by setting the environment variable ``MOZ_NO_REMOTE``: + +.. code:: + + set MOZ_NO_REMOTE=1 + +Or, starting with Firefox 2 and other Gecko 1.8.1-based applications, +you can use the ``-no-remote`` command-line switch instead (implemented +in +`bug 325509 <https://bugzilla.mozilla.org/show_bug.cgi?id=325509>`__). + +You can also specify the profile to use with the ``-P profile_name`` +command-line argument. + +Debugging JavaScript +~~~~~~~~~~~~~~~~~~~~ + +You can use helper functions from +`nsXPConnect.cpp <https://searchfox.org/mozilla-central/source/js/xpconnect/src/nsXPConnect.cpp>`__ +to inspect and modify the state of JavaScript code from the MSVS +debugger. + +For example, to print current JavaScript stack to stdout, evaluate this +in Immediate window: + +.. code:: + + {,,xul}DumpJSStack() + +Visual Studio will show you something in the quick watch window, but +not the stack, you have to look in the OS console for the output. + +Also this magical command only works when you have JS on the VS stack. + +Debugging minidumps +~~~~~~~~~~~~~~~~~~~ + +See :ref:`debugging a minidump <Debugging A Minidump>`. + +Problems post-mortem debugging on Windows 7 SP1 x64? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you attempt to use ``NS_DebugBreak`` etc to perform post-mortem +debugging on a 64bit Windows 7, but as soon as you try and continue +debugging the program crashes with an Access Violation, you may be +hitting a Windows bug relating to AVX support. For more details, +including a work-around see `this blog +post <http://www.os2museum.com/wp/?p=960>`__ or `this social.msdn +thread <http://social.msdn.microsoft.com/Forums/vstudio/en-US/392ca62c-e502-42d9-adbc-b4e22d5da0c3/jit-debugging-32bit-app-crashing-with-access-violation>`__. +(And just in-case those links die, the work-around is to execute + +:: + + bcdedit /set xsavedisable 1 + +from an elevated command-prompt to disable AVX support.) + +Got a tip? +~~~~~~~~~~ + +If you think you know a cool Mozilla debugging trick, feel free to +discuss it with `#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ and +then post it here. + +.. |Screenshot of disabling assertions| image:: https://developer.mozilla.org/@api/deki/files/420/=Win32-debug-nop.png + :class: internal diff --git a/docs/contributing/debugging/debugging_webrtc_calls.rst b/docs/contributing/debugging/debugging_webrtc_calls.rst new file mode 100644 index 0000000000..a08ab05999 --- /dev/null +++ b/docs/contributing/debugging/debugging_webrtc_calls.rst @@ -0,0 +1,1060 @@ +Debugging WebRTC Calls +====================== + +.. _reporting_webrtc_call_issues: + +Reporting WebRTC Call Issues +---------------------------- + +The best way to report an issue is through Bugzilla using this +`link <https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=WebRTC>`__. +Describe the issue you've run into, include a URL, along with the +details of the call setup. See the :ref:`adding-call-setup-information` +section for helpful templates. Here are some common examples of descriptive +WebRTC bug summaries: + +* caller is experiencing frozen video, screen capture, or desktop capture +* caller does not hear audio +* caller's voice sounds distorted or robotic +* video resolution is lower than expected +* caller's video appears rotated +* there is significant delay between a caller's video and audio +* camera, microphone, or screens are not appearing in the Firefox device access + permission prompts, etc. +* caller's video is garbled, partially missing, or the colors are incorrect +* caller can not share external display, but can share integrated display + +.. note:: + Not all web conferencing software makes extensive use of WebRTC. + + +For simple issues, the first place to look is to check the +`web developer console </devtools-user/web_console>`__ for error messages +related to media format issues. If you see messages here related to WebRTC, +getUserMedia, or getDisplayMedia, please add this information to your bug. + +.. _adding-call-setup-information: + +Adding Call Setup Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following template can help one provide all the call details needed to +diagnose many common issues. + +.. code:: md + + * Does this problem occur in Firefox for Desktop or Android? + + * Is this problem reproducible in Firefox Nightly? + + * Has this worked previously? + + * Have you tried using `about:profiles` to reproduce the problem in a + clean profile? + + * How many participants were in the call? + + * In which web conferencing or web calling services does the problem occur? + + * Does the problem present itself immediately upon starting the call? + + * If not how long does it take to occur? + + * If this is a problem with audio or video capture, what camera or microphone + are you using? (adding about:support text may be helpful) + + * If this is problem with screen capture, which screen was being captured, + and are there other screens attached to the same machine? + + * Would you be willing to help us find a regression range? + +If the issue is of specification compliance the template provided below may be +more useful. If one is unsure if this is a compliance issue, one can refer to +the :ref:`standards-documentation` section for links. + +.. code:: md + + * What unexpected behavior is being observed? What are the steps to reproduce + this issue? + + * What is the expected behavior? + + * Where is this behavior specified? + + * Is this problem reproducible in Nightly? + + * Have you tried using `about:profiles` to reproduce the problem in a clean + profile? + + * Has this worked previously? + + * If so, would you be willing to help us find a regression range? + +.. _share_your_aboutsupport_text: + +Adding about:support Text +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In your Bugzilla report, please include support information about the +current device on which you are experiencing an issue. + +#. Open a tab and visit about:support +#. Click 'Copy Text to Clipboard' +#. Paste this text in your Bugzilla bug comment and post. + +.. note:: + + To open :code:`about:*` links in Firefox one must do the following: + #. Right-click the link and select `Open Link in New Tab` + #. Select the new tab + #. Click inside the address bar which should contain the about URL + #. Press Enter + +.. _share_your_aboutwebrtc_contents: + +Adding about:webrtc RTCPeerConnection Stats +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Open about:webrtc. +#. Expand the RTCPeerConnection section. +#. Locate and expand the subsection RTCPeerConnection that one wishes to copy + stats from. +#. Press the :code:`Copy Stats Report` +#. In the Bugzilla bug, press the :code:`Attach New File` button. +#. Click inside the large text box labeled :code:`File`, and paste the copied + stats report. +#. Add a descriptive label to the the :code:`Description:`, e.g. + "PeerConnection statistics sample taken during period of frame loss". +#. In the drop down box, next to the :code:`select from list` radio option, + select "JSON source (application/json)". +#. If needed, add a descriptive comment to the :code:`comment` field. +#. Press the :code:`Submit` button. + +.. note:: + Hovering the mouse over some headings will reveal a clipboard icon. Clicking + this icon will copy the contents under that section to the clipboard as a + JSON document. This can be useful if one wants to submit a portion of the + available stats. Of particular note are the :code:`RTP Stats` heading whose + button will copy the latests RTP stats, and the :code:`SDP` section whose + button will copy the SDP offer; answer; and roles. + +Adding Your about:webrtc Contents +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For issues with call quality, please share web conferencing related +performance information by providing your about:webrtc information. Note +this information should be collected while the call in question is still +active. + +#. While your call is still ongoing, open an additional tab and visit + about:webrtc. +#. Click "Clear History" to clear the stats from other recent calls + which are no longer ongoing. +#. At the bottom of the page click 'Save Page', and save this file. +#. Add this file as an attachment to your bug. + +This data contains statistics about your call, the signalling that was +used to setup your call, and information about the network transports. + +.. _diagnosing_call_quality_issues: + +Diagnosing Call Quality Issues +------------------------------ + +.. _about_webrtc_overview: + +about:webrtc Overview +~~~~~~~~~~~~~~~~~~~~~ + +about:webrtc is an in-browser resource for debugging WebRTC calls. The +primary audience for about:webrtc is browser developers but it can also be +of use to anyone that needs to troubleshoot a WebRTC call. When there is no +call data to display, about:webrtc will appear as follows: + +.. figure:: img/about_webrtc_default_view.png + :alt: about:webrtc before any calls have been made + + about:webrtc before any calls have been made + +Note that there are several sections. During a call each section will contain +information pertinent to different aspects of the WebRTC browser implementation. + +RTCPeerConnection Statistics +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This section presents information useful for diagnosing active calls. It +contains RTCPeerConnection creation parameters, connection information, +negotiation details, RTP stream statistics, bandwidth statistics, and +output frame statistics. + +Connection Log +^^^^^^^^^^^^^^ + +When one needs to diagnose issues with the underlying transports, logs +can be found under Connection Log. + +User Modified WebRTC Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +This section will display any user modified preferences which effect the +performance or behavior of browser components which can impact WebRTC calls. +When hovering over a preference path displayed in this section a clipboard +icon will appear. Clicking that icon will copy the path to the clipboard. It +can then be pasted into `about:config <about:config>`__, to change or reset the +value to its default. + +.. warning:: + Unexpected values in this section may be caused by installed extensions. It + is best to test issues in a clean profile when possible using + `about:profiles <about:profiles>`__. + +Media Context +^^^^^^^^^^^^^ + +Information that is gathered to determine Codec availability and capability is +recorded under Media Context. + + +Bottom Control Bar +^^^^^^^^^^^^^^^^^^ + +At the bottom of about:webrtc is a row of buttons which allow the user to +perform miscellaneous actions. + +The `Save Page` button expands all the sections and presents a dialog to save +the contents of the page. This will produce an HTML file suitable for attaching +to bug reports. + +In the event of a WebRTC issue, the :code:`Enable WebRTC Log Preset` button is a very +quick way to start logging. Pressing that button will open a new tab with +`about:logging <about:logging>`__ with the :code:`webrtc` preset selected. This +preset contains all :ref:`standard-logging-modules`. Logging +will begin immediately. If one needs to change other log settings on that page +one can customize them, and then press :code:`Start Logging`. This may be +necessary if one wishes to log to the profiler. + +If experiencing echo cancellation issues, one may be asked to submit echo +cancellation logs. These logs are gathered by pressing the +:code:`Start AEC Logging` button. One should press the button to activate +the logging while actively experiencing an echo cancellation failure. + +.. note:: + + Producing echo cancellation logs is incompatible with the content sandbox. + The user will be prompted with further instructions if the sandbox is active + and the :code:`Start AEC Logging` button is pressed. + +.. _audiovideo_delay: + +Audio/Video Delay +^^^^^^^^^^^^^^^^^ + +Delayed media is commonly caused by long physical paths between +endpoints, though anything that slows down inter-hop delivery of packets +can be at fault. Note that this is different than the bandwidth of the +network path, and a high latency will not be fixed by reducing the video +resolution or audio sample rate. Round trip time, or RTT, is the time it +takes for a packet to get from the sender to the receiver and then for a +packet to get from the receiver back to the sender. If the path is +symmetric between the two endpoints one can assume that the one way +delay is half the delay of the round trip. + +The second common cause of A/V delay is jitter, the magnitude of +variability in packet inter-arrival times. In order to smoothly play out +of the incoming stream a receiver experiencing jitter will have to +buffer (delay) incoming packets. + +**Using** `about:webrtc <about:webrtc>`__ **to Diagnose Delay** + +The key metrics in `about:webrtc <about:webrtc>`__ are RTT (round-trip-time) and +jitter. They can be found in the RTP stats section of the +PeerConnection. The PeerConnection informational blocks start out in a +minimized state, and one will need to expand a block to find the RTP +stats section: + +.. figure:: img/about_webrtc_reveal_PeerConnection_stats.png + :alt: How to reveal the full statistics of a PeerConnection in about:webrtc + :width: 800px + + How to reveal the full statistics of a PeerConnection in about:webrtc + +Once expanded one can locate the RTP Stats section and find the key +diagnostic stats: + +.. figure:: img/about_webrtc_Jitter_and_RTT.png + :alt: Location in about:webrtc of jitter and RTT stats + :width: 800px + + Location in about:webrtc of jitter and RTT stats + +In this image we can see that there are 0 milliseconds of jitter, and 32 +milliseconds of round trip delay. This call should not be experiencing any +noticeable delay. See the `Delay Calculation <#delay-calculation>`__ +appendix section below for some more detail. + +If the perceived delay is larger than the estimated delay that could +indicate a problem within Firefox that requires debugging. Under these +circumstances it would be helpful to grab a JSON copy of the current +stats by pressing the "Copy Report" button, pasting those stats into +your Bugzilla bug report. + +.. figure:: img/about_webrtc_copy_report.png + :alt: Location in about:webrtc of Copy Report button + :width: 800px + + Location in about:webrtc of Copy Report button + +.. _performance-profiling-and-logging: + +Performance Profiling and Logging +--------------------------------- + +.. _capturing-a-firefox-performance-profile: + +Capturing a Firefox Performance Profile +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For basic performance issues, a performance profile can help engineers +diagnose issues with video formats, performance, and rendering. + +#. Visit https://profiler.firefox.com/ and enable the Profiler toolbar + button. +#. Click the toolbar button down arrow and select 'Media' in the + Settings drop down. +#. Open a tab and visit the page with the affected media content. +#. Click the Profiler toolbar main button to start recording. +#. Play media until the issue you are seeing manifests. +#. Click the Profiler toolbar button again to stop recording. +#. When a new Profile tab opens, click the upload profile button on the + upper right. +#. Copy the resulting profile URL and post this to your Bugzilla report. + +Additionally, detailed logging can be collected within performance +profiles to help aid in debugging complicated issues. To enable the +collection of a profile with low level debugging - + +#. Visit https://profiler.firefox.com/ and enable the Profiler toolbar + button. +#. In a new tab, visit about:webrtc. Click the 'Enable WebRTC Log + Preset' button, which will open a tab for about:logging with + pre-populated information. +#. In about:logging, click the "Start Logging" button. (You are now + recording a profile, the profiler toolbar toggle button should be + selected automatically.) +#. Open a new tab for testing and view the media you are having an issue + with. (After reproducing, DO NOT close this test tab yet.) +#. Switch to the about:logging tab, click 'Stop logging', and then close + the test tab. +#. Wait approximately 10 - 20 seconds for a new tab to automatically + open containing the generated performance profile. +#. Within the upper-right side of the profiler tab click the 'upload + local profile' button to initiate profile upload. Once the upload is + complete, a drop down text field will open displaying the URL of the + profile. Select this text and copy it. +#. Share the URL of the profile for analysis with the engineer you are + working with. + +Alternatively one can set the following environment variable: + +.. code:: sh + + MOZ_LOG="jsep:5,sdp:5,signaling:5,mtransport:5,RTCRtpReceiver:5,RTCRtpSender:5,RTCDMTFSender:5,VideoFrameConverter:5,WebrtcTCPSocket:5,CamerasChild:5,CamerasParent:5,VideoEngine:5,ShmemPool:5,TabShare:5,MediaChild:5,MediaParent:5,MediaManager:5,MediaTrackGraph:5,cubeb:5,MediaStream:5,MediaStreamTrack:5,DriftCompensator:5,ForwardInputTrack:5,MediaRecorder:5,MediaEncoder:5,TrackEncoder:5,VP8TrackEncoder:5,Muxer:5,GetUserMedia:5,MediaPipeline:5,PeerConnectionImpl:5,WebAudioAPI:5,webrtc_trace:5,RTCRtpTransceiver:5,ForwardedInputTrack:5,HTMLMediaElement:5,HTMLMediaElementEvents:5" + +Note that webrtc_trace will not be active until "Enable WebRTC Log +Preset" is pressed. + +.. _standard-logging-modules: + +Standard Logging Modules +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Standard Logging Modules + :header-rows: 1 + + * - Module + - Component + - Function + - Notes + * - jsep + - signalling + - JSEP state machine + - + * - sdp + - signalling + - SDP parsing + - + * - mtransport + - networking + - Network transports + - + * - RTCRtpReceiver + - JS API + - JS API related to receiving media and media control packets + - + * - RTCRtpSender + - JS API + - JS API related to sending media and media control packets + - + * - RTCDMTFSender + - JS API + - JS API related to sending DTMF messages + - + * - VideoFrameConverter + - + - + - + * - WebrtcTCPSocket + - networking + - + - + * - CamerasChild + - media capture + - Content process end of IPC channel for receiving frames from media + capture devices + - + * - CamerasParent + - media capture + - Parent process end of IPC channel for sending frames from media capture devices + - + * - VideoEngine + - media capture + - Orchestrates capture of frames from media capture devices in the parent process + - + * - ShmemPool + - media capture + - Object pool of shared memory frame buffers for transferring media capture frames from parent to child process + - + * - TabShare + - media capture + - Captures tab content for sharing + - + * - MediaChild + - media + - + - + * - MediaParent + - media + - + - + * - MediaManager + - media + - + - + * - MediaTrackGraph + - media + - + - + * - cubeb + - media + - + - + * - MediaStream + - media + - + - + * - MediaStreamTrack + - media + - + - + * - DriftCompensator + - media + - + - + * - ForwardInputTrack + - media + - + - + * - MediaRecorder + - media + - + - + * - MediaEncoder + - media + - + - + * - TrackEncoder + - media + - + - + * - VP8TrackEncoder + - media + - + - + * - Muxer + - media + - + - + * - MediaPipeline + - network + - Glue code between transport, media, and libwebrtc components + - + * - PeerConnectionImpl + - JS API + - implements the RTCPeerConnection object + - + * - WebAudioAPI + - + - + - + * - webrtc_trace + - webrtc + - libwebrtc logging + - Prior to Firefox v123 it must be enabled from + `about:webrtc <about:webrtc>`__ at runtime, or it must be set in the + :code:`MOZ_LOG` environment variable at launch. + * - RTCRtpTransceiver + - JS API + - implements the RTCRtpTransceiver object + - + * - HTMLMediaElement + - + - + - + * - ForwardedInputTrack + - + - + - + * - HTMLMediaElementEvents + - + - + - + +.. _non-standard-loggin-modules: + +Non-standard Logging Modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Standard Logging Modules + :header-rows: 1 + + * - Module + - Component + - Function + - Notes + * - RTPLogger + - network + - Logs RTP and RTCP packet fragments + - See `Debugging Encrypted Packets <#debugging-encrypted-packets>`__ + +.. _examining-call-performance-issues: + +Examining Call Performance Issues +--------------------------------- + +.. _enabling-call-stats-history: + +Enabling Call Stats History +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Call stats history is enabled by default in Nightly. To enable in +release builds open `about:config <about:config>`__, and change +"media.aboutwebrtc.hist.enabled" to true. This will keep a history +window of stats for a number of recent calls, allowing for inspection +in `about:webrtc <about:webrtc>`__ after a call has completed. + +.. _dumping-call-stats: + +Dumping Call Stats +~~~~~~~~~~~~~~~~~~ + +One can dump a JSON blob of call stats for an active call, or a recent +call if call stats history is enabled. There are two buttons in +`about:webrtc <about:webrtc>`__ to do this, "Copy Report" and "Copy Report +History". The former will create a copy of the most recent stats for the +PeerConnection. The later will copy all the history of stats reports +that `about:webrtc <about:webrtc>`__ has accumulated for that PeerConnection, this +can be up to several minutes of stats. + +.. _debugging-encrypted-packets: + +Debugging Encrypted Packets +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. warning:: + There is a `blog + post <https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-used-to-be/>`__ + covering dumping unencrypted partial RTP and RTCP packets in the logs. + While the information presented in that post is still relevant, + the command to extract the packet data in the blog is out of date. A + working method is presented below; + +Using the gecko logging system, unencrypted, mangled, partial, RTP-packets can +be written out. This may be a good avenue of investigation for packet loss and +recovery, simulcast, and feedback. Because the entirety of the packet is not +guaranteed to be logged, this is less suitable for debugging issues with +encoded media. These logged packets can be converted to PCAP files, which can +then be explored in `Wireshark <https://www.wireshark.org/>`__. The logs +produced by this module can be quite large, making it easy to identify by file +size which child process log files contain packet dumps. + +To start RTP logging, one must enable the :code:`RtpLogger` log module. The :code:`sync` +option should also be used, as it prevents undesirable interleaving of log +messages. Here are the minimal log settings needed: + +.. code:: sh + + MOZ_LOG='sync,RtpLogger:5' + +In order to interpret the packet contents, one needs to refer to the SDP. +Wireshark is unaware of the negotiated details, so it can not directly decode +the media, nor can it decode the header extensions. The SDP can also be logged, +and so the following is a more useful set of log settings: + +.. code:: sh + + MOZ_LOG='sync,RtpLogger:5,jsep:5' + +.. note:: + + On macOS it is simple to install Wireshark and text2pcap with Homebrew: + + .. code:: sh + + # Use only one of the following: + # ============================== + + # To install the Wireshark GUI application and the command line utilities: + brew install --cask wireshark + + # To install only the command line utilities: + brew install wireshark + +One can use :code:`tee` to capture log output +from a copy of Firefox launched from the command line, e.g. through +:code:`mach`. Alternatively, one can set a log file through the environment +variable :code:`MOZ_LOG_FILE` or through about:logging. + +.. warning:: + + If log files are not being created by child processes, this is likely due + to sandboxing of content processes. To work around this one must either + select a location within the sandbox, disable the content sandbox, or launch + Firefox from the command line, e.g. from a Firefox dev environment: + + .. code:: + + MOZ_LOG=sync,RtpLogger:5,jsep:5 MOZ_LOG_FILE= ./mach run 2>&1 | tee your.log + + +To produce a PCAP file one needs to filter the logs to include only the +RtpLogger log lines, reduce them down to the expected ingestion format for +text2pcap, and finally to invoke text2pcap. + +.. code:: sh + + cat your.log | rg 'RtpLogger.*RTC?P_PACKET|>>\s(?P<packet>.+$)' --only-matching --replace '$packet' | text2pcap -D -n -l 1 -i 17 -u 1234,1235 -t '%H:%M:%S.' - your.output.pcap + +.. note:: + + If :code:`rg`, a.k.a ripgrep, is not already available, one can install it via one of the following methods: + + .. code:: sh + + # Install through cargo on macOS, Linux, or Windows + cargo install ripgrep + + # Install via Homebrew on macOS + brew install ripgrep + + # ripgrep packages may be available through the package manager for your + # Linux distro + + +The resulting PCAP file can be explored with Wireshark. Currently, one must refer to the SDP in order to interpret the RTP packets. + +.. code:: sh + + # On most Linux distros + wireshark -d 'udp.port==1234,rtp' your.output.pcap + + # On macOS when installed via Homebrew + open /Applications/Wireshark.app --args -d 'udp.port==1234,rtp' your.output.pcap + +.. _examining-codec_availability-and-capabilities: + +Examining Codec Availability and Capabilities +--------------------------------------------- + +When codec negotiation doesn't happen as expected there are several helpful +locations where one can find information. The SDP offer and answer contain the +list of codecs that were in the initial offer, and the subset of those codecs +that were selected in the answer. + +The easiest way to get this information on a live call is through +about:webrtc. Each RTCPeerConnection has its own subsection, that when +expanded contains an SDP section. There are buttons to display the offer +and the answer. Depending on which party was the offerer and which was +the answerer one may have a local offer and a remote answer, or a remote offer +and a local answer. + +Firefox chooses which codecs to offer based on availability. Some codecs, +like Opus or VP8, are always available. Some codecs are available in software +and some codecs on some platforms are available in hardware. H264 support is +provided by a third-party, and is automatically downloaded the first time its +use is requested. This is a process which can take a variable amount of time +depending on network circumstances. + +.. note:: + A list of media codecs with playback support are available in the Media + section of `about:support#media <about:support#media>`__ . Not all media + codecs present and available to Firefox for playback are supported in WebRTC + calls. + +To check the current factors, including preferences, that are being used to +calculate availability beyond codec presence, one can check the `Media Context` +section of about:webrtc. + +.. figure:: img/about_webrtc_media_context.png + :alt: example about:webrtc media context values + +.. _running-webrtc-tests: + +For an in-depth reference covering the codecs available through WebRTC please +see the MDN Page: `Codecs Used by WebRTC <https://developer.mozilla.org/en-US/docs/Web/Media/Formats/WebRTC_codecs>`__. + +Running WebRTC Tests +-------------------- + +There are a variety of tests providing coverage over WebRTC related code. The +Web Platform Suite provides conformance tests for browsers. The +:code:`gtest` suite is composed of unit tests. Crashtests are a type of +regression test which are written to induce crashes. There are fuzzing tests +which exercise APIs in ways that the authors did not foresee. All of the WebRTC +tests can be run locally with :code:`mach` or in CI on `Try`. There is a +detailed overview of all available test types including those not exercised by +WebRTC code `here </testing/automated-testing/index.html#functional-testing>`__. + +.. note:: + Running :code:`./mach <verb> --help` is an indispensable tool for discovering + options that can streamline your testing process. + +.. note:: + A test suite on `Try` maybe an aggregate of multiple logical test suites. + For example, the `mochitest-media` suite on try includes both the WebRTC and + playback mochitests. + +.. warning:: + WebRTC calls make use of a number of internal timers. Amongst the + behaviors these timers control are transport selection, bandwidth estimation, + packet loss determination, media adaptation, lip sync, connection timeout, + and more. There are `Try` targets which are too slow to reliably run a + number of the tests. Before running a specific test on `Try` for the first + time, it may be best to check the relevant test suite manifest. This can be + done easily with Searchfox.org by searching for and viewing a test file. If + that test has been disabled on one or more platforms the details will appear + as shown below: + + .. figure:: img/searchfox_test_disabled_warning.png + :alt: Searchfox.org warning that the displayed test file has been disabled on Android + +.. _test-atlas: + +Test Atlas +---------- + +.. list-table:: WebRTC Test Locations + :widths: 10 10 20 10 10 + :header-rows: 1 + + * - Component + - Test type + - Test file location + - Try suite + - Treeherder Abbreviations + * - WebRTC + - Mochitest + - dom/media/webrtc/mochitests + - mochitest-media + - :code:`mda`, :code:`M(mda)` + * - + - Web Platform Test + - testing/web-platform/tests/webrtc + - wpt + - :code:`wpt`, :code:`W(wpt)` + * - + - Crashtest + - dom/media/webrtc/tests/crashtests + - crash + - :code:`R(C)` + * - WebRTC Signalling + - GTest + - media/webrtc/signaling/gtest + - gtest + - :code:`gtest` + * - WebRTC (gUM / gDDM) + - Browser Chrome Test (mochitest) + - browser/base/content/test/webrtc + - browser-chrome + - :code:`bc`, :code:`M(bc)` + * - WebRTC Transport + - CPPUnit + - dom/media/webrtc/transport/test + - cppunit + - + * - + - fuzztest + - dom/media/webrtc/transport/fuzztest + - fuzzing + - + * - SDP parser + - Fuzzing + - dom/media/webrtc/tests/fuzztests + - fuzzing + - + +.. _webrtc-web-platform-tests: + +Web Platform Tests +~~~~~~~~~~~~~~~~~~ + +The WPT suite comprises conformance tests for various W3C specs such as: CSS, +JS APIs, and HTML. WebRTC is a JS API and as such its tests are of the `testharness.js <https://web-platform-tests.org/writing-tests/testharness.html>`__ type. +There is detailed :code:`WPT` documentation available `here </web-platform/index.html#web-platform-tests>`__ +Web Platform Tests can be run locally from + +.. code:: bash + + # Run the entire WebRTC WPT test suite + ./mach wpt testing/web-platform/tests/webrtc + + # Run a single test, e.g. RTCPeerConnection-createAnswer.html + ./mach wpt testing/web-platform/tests/webrtc/RTCPeerConnection-createAnswer.html + + # Run all of the PeerConnection tests, i.e. RTCPeerConnection-*.html + # NOTE that the `mach` verb in use is `test` not `wpt` + ./mach test testing/web-platform/tests/webrtc/RTCPeerConnection + +.. warning:: + Running the :code:`WPT` tests locally can be very disruptive to one's working + desktop environment, as windows will frequently appear and grab focus. Using + :code:`mach`'s :code:`--headless` flag will prevent this, and can be a great + way to run them if one's problem can be evaluated from command line output. + +These tests are synced from the main `Web Platform Test +repository <https://github.com/web-platform-tests/wpt>`__, and likewise +our changes are synced from our `in-tree +copy <https://searchfox.org/mozilla-central/search?q=&path=testing%2Fweb-platform%2Ftests%2Fwebrtc&case=false®exp=false>`__ +back to that repository. + +.. warning:: + Running the WebRTC mochitests in `Try` is done using the entire Web Platform Test suite, :code:`wpt`. + As such, this can be slow. + +.. code:: bash + + ./mach try fuzzy --query 'wpt' + +One can `run those same tests in +Chromium <https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/testing/web_platform_tests_wptrunner.md>`__, +Safari, or Servo if one needs to compare behavior between browsers. This can be +done directly through :code:`mach`, see `running tests in other browsers </web-platform/index.html#running-tests-in-other-browsers>`__ +for more details. + +.. _mochitests: + +Mochitests +~~~~~~~~~~ + +The WebRTC mochitests are integration tests, regression tests, and sanity +tests. The needs of these tests did not align with specification conformance +testing in the WPT, Web Platform Test, suite. Before writing a new mochitest, +one should consider if a test would be better expressed as a WPT, which all browsers +can test against. + +Locally running the WebRTC mochitests should be done in a Firefox dev +environment using :code:`mach` as follows: + +.. code:: bash + + # Run the whole suite + ./mach mochitest dom/media/webrtc/tests/mochitests + + # Run a single test, e.g. test_peerConnection_basicAudioVideo.html + ./mach mochitest dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioVideo.html + # Or + ./mach mochitest test_peerConnection_basicAudioVideo.html + + # Run all of the PeerConnection tests, i.e. test_peerConnection_*.html + ./mach mochitest test_peerConnection + + +On :code:`try`, WebRTC mochitests are part of the larger media test suite. +This suite can be easily selected with the following fuzzy query: + +.. code:: bash + + # Run the media mochitest suite on all regular platforms + ./mach try fuzzy --query 'mochitest-media' + + # Run the media mochitest suite only on Linux which will resolve far faster + ./mach try fuzzy --query 'linux mochitest-media' + + +.. _gtests: + +GTests +~~~~~~ + +The gtests are all compiled into a single library target: :code:`xul-test`. +This makes running gtests from :code:`mach` slightly different than the other +test types. + +.. code:: bash + + # Run a single test by using Prefix.TestName, e.g. JsepSessionTest.FullCall + # https://searchfox.org/mozilla-central/rev/4d6a5b97428760d15bfcad13f8fc81439370a7ec/media/webrtc/signaling/gtest/jsep_session_unittest.cpp#1551 + ./mach gtest 'JsepSessionTest.FullCall' + + # Run all the tests in a single Prefix, e.g. JsepSessionTest + ./mach gtest 'JsepSessiontTest.*' + + # Run all tests which have a Prefix.TestName containing the substring 'Jsep' + # See the table of selectors below + ./mach gtest '*Jsep*' + + # Run all the gtests for Firefox + ./mach gtest + +Here is a list of helpful substring selectors for executing specific WebRTC gtests: + +.. list-table:: WebRTC GTest Selectors + :header-rows: 1 + + * - Selector + - Description + - Files + * - :code:`*Jsep*` + - JSEP (signalling) tests + - `jsep_session_unittest.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/jsep_session_unittest.cpp>`__ + `jsep_trak_unittest.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/jsep_track_unittest.cpp>`__ + * - :code:`*Sdp*` + - SDP parsing tests + - `sdp_unittests.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/sdp_unittests.cpp>`__ + * - :code:`*MediaPipeline*` + - MediaPipline and MediaPipeline filter tests for RTP media handling + - `mediapipeline_unittest.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/mediapipeline_unittest.cpp>`__ + * - :code:`*AudioConduit*` + - AudioConduit tests for libwebrtc glue for RTP audio media + - `audioconduit_unittests.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/audioconduit_unittests.cpp>`__ + * - :code:`*VideoConduit*` + - VideoConduit tests for libwebrtc glue for RTP video media + - `videoconduit_unittests.cpp <https://searchfox.org/mozilla-central/source/media/webrtc/signaling/gtest/videoconduit_unittests.cpp>`__ + +For more general information about gtests see the documentation `here </gtest/index.html>`__. + +Fuzz Testing +~~~~~~~~~~~~ + +It is not common to need to run fuzz testing as it is run on a semi-continuous +fashion in CI. It is more likely that one will need to respond to a bug filed +by a fuzzing bot. If one is interested in fuzzing one should consult the +excellent documentation available `here </tools/fuzzing/index.html>`__. + +.. _code-atlas: + +Code Atlas +---------- + +There are a number of components that work together to create a successful +WebRTC call. When debugging a call it can be difficult to see the larger puzzle +for all the pieces. A listing of the WebRTC related source code directories is +provided below to help one navigate. + +.. list-table:: WebRTC Code Atlas + :header-rows: 1 + + * - Directory + - Component + - Description + - Notes + * - `dom/media/webrtc <https://searchfox.org/mozilla-central/source/dom/media/webrtc>`__ + - WebRTC + - This is the primary directory for Firefox WebRTC code + - + * - `dom/media/webrtc/common <https://searchfox.org/mozilla-central/source/dom/media/webrtc/common>`__ + - WebRTC + - This contains WebRTC related utility code + - + * - `dom/media/webrtc/jsapi <https://searchfox.org/mozilla-central/source/dom/media/webrtc/jsapi>`__ + - JS API + - This contains the C++ implementations of the JavaScript WebRTC interfaces + - + * - `dom/media/webrtc/jsep <https://searchfox.org/mozilla-central/source/dom/media/webrtc/jsep>`__ + - Signalling + - This is the JSEP state engine implementation + - + * - `media/webrtc/libwebrtcglue <https://searchfox.org/mozilla-central/source/dom/media/webrtc/libwebrtcglue>`__ + - WebRTC (various) + - This is the glue code between libwebrtc and Firefox + - + * - `dom/media/webrtc/sdp <https://searchfox.org/mozilla-central/source/dom/media/webrtc/sdp>`__ + - Signalling + - This contains the SDP parsing interface + - + * - `dom/media/webrtc/tests <https://searchfox.org/mozilla-central/source/dom/media/webrtc/tests>`__ + - Tests + - This contains `some` of the WebRTC related tests + - + * - `dom/media/webrtc/third_party_build <https://searchfox.org/mozilla-central/source/dom/media/webrtc/third_party_build>`__ + - Build + - The scripting and configuration for vendoring new versions of libwebrtc are here + - This is unlikely to be of concern for debugging + * - `dom/media/webrtc/transport <https://searchfox.org/mozilla-central/source/dom/media/webrtc/transport>`__ + - Network + - This contains the ICE implementation, the MDNS implementation, and transport code + - + * - `dom/media/webrtc/transportbridge <https://searchfox.org/mozilla-central/source/dom/media/webrtc/transportbridge>`__ + - WebRTC + - This contains the MediaPipeline and MediaPipeline filter code which is glue between transport and the libwebrtc RTP stack + - + * - `third_party/libsrtp <https://searchfox.org/mozilla-central/source/third_party/libsrtp>`__ + - Network + - This is the SRTP implementation used by Firefox + - + * - `third_party/libwebrtc <https://searchfox.org/mozilla-central/source/third_party/libwebrtc>`__ + - WebRTC (various) + - libwebrtc handles many aspects of WebRTC calls above the transport layer and below the presentation layer + - + * - `third_party/rust/webrtc-sdp <https://searchfox.org/mozilla-central/source/third_party/rust/webrtc-sdp>`__ + - Signalling + - webrtc-sdp is a Rust implementation of a WebRTC-only SDP parser + - + * - `third_party/sipcc <https://searchfox.org/mozilla-central/source/third_party/sipcc>`__ + - Signalling + - sipcc is a C implementation of a general SDP parser + - this carries many local modifications + * - `dom/media <https://searchfox.org/mozilla-central/source/dom/media>`__ + - Media Capture + - GetUserMedia and related classes are here + - There are many other unrelated media source files here + * - `dom/webidl <https://searchfox.org/mozilla-central/source/dom/webidl>`__ + - WebIDL (JS API) + - This contains the WebIDL definitions for the WebRTC JS API amongst many other WebIDL definitions + - :code:`RTC*.webidl` + +.. _standards-documentation: + +Standards Documentation +----------------------- + +When debugging API behavior it may be necessary to consult the specifications +for WebRTC. The ECMAScript API is defined in several W3C standards, +`webrtc-pc <https://www.w3.org/TR/webrtc/>`__, and +`webrtc-stats <https://www.w3.org/TR/webrtc-stats/>`__. The number of IETF +standards that are incorporated into WebRTC are too numerous to list here. One +can find these standards in the +`Normative References <https://www.w3.org/TR/webrtc/#normative-references>`__ +section of the :code:`webrtc-pc` spec. + +.. _delay-calculation: + +Appendix: Delay Calculation +--------------------------- + +For all intents and purposes jitter and RTT are +additive in nature. If there was 25ms of jitter reported and a RTT of +272ms, one could estimate the expected delay from transmission at the +send side to play out on receive side to be + +:: + + 25ms + (272ms / 2) = 161ms diff --git a/docs/contributing/debugging/img/about-processes.png b/docs/contributing/debugging/img/about-processes.png Binary files differnew file mode 100644 index 0000000000..6f1563bcd1 --- /dev/null +++ b/docs/contributing/debugging/img/about-processes.png diff --git a/docs/contributing/debugging/img/about-support.png b/docs/contributing/debugging/img/about-support.png Binary files differnew file mode 100644 index 0000000000..2b12d1027d --- /dev/null +++ b/docs/contributing/debugging/img/about-support.png diff --git a/docs/contributing/debugging/img/about_webrtc_Jitter_and_RTT.png b/docs/contributing/debugging/img/about_webrtc_Jitter_and_RTT.png Binary files differnew file mode 100644 index 0000000000..0cb6ad329a --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_Jitter_and_RTT.png diff --git a/docs/contributing/debugging/img/about_webrtc_copy_report.png b/docs/contributing/debugging/img/about_webrtc_copy_report.png Binary files differnew file mode 100644 index 0000000000..79d8dcf9c3 --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_copy_report.png diff --git a/docs/contributing/debugging/img/about_webrtc_default_view.png b/docs/contributing/debugging/img/about_webrtc_default_view.png Binary files differnew file mode 100644 index 0000000000..4b65a666cc --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_default_view.png diff --git a/docs/contributing/debugging/img/about_webrtc_media_context.png b/docs/contributing/debugging/img/about_webrtc_media_context.png Binary files differnew file mode 100644 index 0000000000..a4a6ee11d4 --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_media_context.png diff --git a/docs/contributing/debugging/img/about_webrtc_reveal_PeerConnection_stats.png b/docs/contributing/debugging/img/about_webrtc_reveal_PeerConnection_stats.png Binary files differnew file mode 100644 index 0000000000..f877703b71 --- /dev/null +++ b/docs/contributing/debugging/img/about_webrtc_reveal_PeerConnection_stats.png diff --git a/docs/contributing/debugging/img/crash-gmp.png b/docs/contributing/debugging/img/crash-gmp.png Binary files differnew file mode 100644 index 0000000000..595e439307 --- /dev/null +++ b/docs/contributing/debugging/img/crash-gmp.png diff --git a/docs/contributing/debugging/img/crash-gpu.png b/docs/contributing/debugging/img/crash-gpu.png Binary files differnew file mode 100644 index 0000000000..9e40b63917 --- /dev/null +++ b/docs/contributing/debugging/img/crash-gpu.png diff --git a/docs/contributing/debugging/img/crash-rdd.png b/docs/contributing/debugging/img/crash-rdd.png Binary files differnew file mode 100644 index 0000000000..5548672d27 --- /dev/null +++ b/docs/contributing/debugging/img/crash-rdd.png diff --git a/docs/contributing/debugging/img/crashlist.jpg b/docs/contributing/debugging/img/crashlist.jpg Binary files differnew file mode 100644 index 0000000000..4cb376d0f2 --- /dev/null +++ b/docs/contributing/debugging/img/crashlist.jpg diff --git a/docs/contributing/debugging/img/crashreporter.png b/docs/contributing/debugging/img/crashreporter.png Binary files differnew file mode 100644 index 0000000000..3137c71c78 --- /dev/null +++ b/docs/contributing/debugging/img/crashreporter.png diff --git a/docs/contributing/debugging/img/process-explorer.png b/docs/contributing/debugging/img/process-explorer.png Binary files differnew file mode 100644 index 0000000000..26e7837a8d --- /dev/null +++ b/docs/contributing/debugging/img/process-explorer.png diff --git a/docs/contributing/debugging/img/sdk-installer.png b/docs/contributing/debugging/img/sdk-installer.png Binary files differnew file mode 100644 index 0000000000..c7dcc1f1a4 --- /dev/null +++ b/docs/contributing/debugging/img/sdk-installer.png diff --git a/docs/contributing/debugging/img/searchfox_test_disabled_warning.png b/docs/contributing/debugging/img/searchfox_test_disabled_warning.png Binary files differnew file mode 100644 index 0000000000..a1815262b8 --- /dev/null +++ b/docs/contributing/debugging/img/searchfox_test_disabled_warning.png diff --git a/docs/contributing/debugging/img/tabcrashed.png b/docs/contributing/debugging/img/tabcrashed.png Binary files differnew file mode 100644 index 0000000000..9982e527ed --- /dev/null +++ b/docs/contributing/debugging/img/tabcrashed.png diff --git a/docs/contributing/debugging/img/windbg-in-startmenu.png b/docs/contributing/debugging/img/windbg-in-startmenu.png Binary files differnew file mode 100644 index 0000000000..0819d4b424 --- /dev/null +++ b/docs/contributing/debugging/img/windbg-in-startmenu.png diff --git a/docs/contributing/debugging/local_symbols.rst b/docs/contributing/debugging/local_symbols.rst new file mode 100644 index 0000000000..875717e1a1 --- /dev/null +++ b/docs/contributing/debugging/local_symbols.rst @@ -0,0 +1,64 @@ +Symbolicating TreeHerder stacks locally +======================================= + +When using tools like the :ref:`Dark Matter Detector (DMD)` or +:ref:`refcount logging<Refcount Tracing and Balancing>` to +investigate issues occurring on TreeHerder that you can't reproduce locally, you +can often end up with unsymbolicated stacks. Fortunately, there is a way to +symbolicate these stacks on your own machine. + +These instructions are for a Linux TreeHerder build for MacOS, so they might +require some modifications for other combinations of platforms. + +Download ``target.tar.bz2`` and ``target.crashreporter-symbols.zip`` from the +Build job. **Note that these files are very large so you'll want to delete +them and the extracted files when you are done.** + +These files each contain a large number of files, so I'd recommend creating +a directory for each of them. Call these ``<TARGET_DIR>`` and ``<SYMB_DIR>``, +and move the prior two files into these two directories, respectively. + +Go to ``<TARGET_DIR>`` and run something like + +.. code-block:: shell + + tar xf target.tar.bz2 + +then go to ``<SYMB_DIR>`` and run something like + +.. code-block:: shell + + unzip target.crashreporter-symbols.zip + +You should be able to delete the two original files now. + +Next we need to ensure that the locations of binaries are rewritten from +where they are on TreeHerder to where we have them locally. We'll do this by +editing ``fix_stacks.py``. This file is located in the ``tools/rb/`` directory of +the Firefox source directory. You need to add these two lines to the function +``fixSymbols``, after ``line_str`` is defined and before it is written to +``fix_stacks.stdin``. I've done this right before the definition of +``is_missing_newline``. + +.. code-block:: python + + line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox", + "<TARGET_DIR>/firefox/firefox-bin") + line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so", + "<TARGET_DIR>/firefox/libxul.so") + +The initial locations should appear verbatim in the stack you are trying to +symbolicate, so double check that they match. Also, ``<TARGET_DIR>`` of course +needs to be replaced with the actual local directories where those files are +located. Note that the ``firefox`` executable is changed to ``firefox-bin``. +I don't know why that is necessary, but only the latter existed for me. + +Finally, we need to make it so that the stack fixer can find the location of +the breakpad symbols we downloaded. If you are running ``fix_stacks.py`` via +``dmd.py`` or directly (in a recent version), you can do this by running with the +environment variable ``BREAKPAD_SYMBOLS_PATH`` set to the ``<SYMB_DIR>`` from above. +If that doesn't work, you'll have to edit ``initFixStacks`` in ``fix_stacks.py`` to +set ``breakpadSymsDir`` to ``<SYMB_DIR>``. + +With all of that done, you should now be able to run ``dmd.py`` or ``fix_stacks.py`` +to fix the stacks. Note that the stack fixing process can take a minute or two. diff --git a/docs/contributing/debugging/process_dump_task_manager.rst b/docs/contributing/debugging/process_dump_task_manager.rst new file mode 100644 index 0000000000..d345171856 --- /dev/null +++ b/docs/contributing/debugging/process_dump_task_manager.rst @@ -0,0 +1,69 @@ +How to get a process dump with Windows Task Manager +=================================================== + +Introduction +------------ + +When tracking down the causes of process hangs, it is often helpful to +obtain a process dump while the process is experiencing a hang. This +article describes how to get a process dump with Task Manager on +Windows. (To get a process dump for Thunderbird or some other product, +substitute the product name where ever you see Firefox in these +instructions.) + + +Caution +------- + +The memory dump that will be created through this process is a complete +snapshot of the state of Firefox when you create the file, so it +contains URLs of active tabs, history information, and possibly even +passwords depending on what you are doing when the snapshot is taken. It +is advisable to create a new, blank profile to use when reproducing the +hang and capturing the memory dump. Please ask for help doing this! + + +Requirements +------------ + +Windows + To get a process dump, you need to be using Windows Vista or above. +A Firefox nightly or release + You need a Firefox version for which symbols are available from the + :ref:`symbol server <Using The Mozilla Symbol Server>`. You + can use any `official nightly + build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released + version of Firefox from Mozilla. You can find the latest trunk + nightly builds under + `http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/>`__. + + +Creating the Dump File +---------------------- + +Ensure that Firefox is not already running. + + +Run Firefox, reproduce the hang +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Start Firefox and perform whatever steps are necessary to cause Firefox +to hang. Once the browser hangs, continue with the steps below. + + +After the hang +~~~~~~~~~~~~~~ + +#. Open Windows Task Manager (CTRL+SHIFT+ESC). +#. Find Firefox.exe among the list of processes. +#. Right-click Firefox.exe and select "Create dump file". Task manager + should indicate where the dump file was written to. + + +See also +-------- + +- :ref:`How to get a stacktrace for a bug report <How to get a stacktrace for a bug report>` +- `How to create a user-mode process dump file in Windows Vista and in + Windows 7 + (MSDN) <https://docs.microsoft.com/en-us/windows/client-management/generate-kernel-or-complete-crash-dump#manually-generate-a-memory-dump-file>`__ diff --git a/docs/contributing/debugging/stacktrace_report.rst b/docs/contributing/debugging/stacktrace_report.rst new file mode 100644 index 0000000000..08f4bded00 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_report.rst @@ -0,0 +1,153 @@ +How to get a stacktrace for a bug report +======================================== + +If you file a bug report in Bugzilla about a crash you should include a +stacktrace (call stack) in your report. A stacktrace will tell Mozilla +developers what crashed and provide a starting point for investigating +its cause. This article describes how to use the Mozilla Crash Reporter +(Breakpad) to get a crash ID, which our engineers can use to get a +stacktrace, and alternative ways to get a stacktrace if you can't get a +crash ID. + +Requirements +------------ + +You need a binary build of Firefox from +`Mozilla.org <https://www.mozilla.org/firefox/>`__. SeaMonkey and +Thunderbird also support crash reporting. + +Mozilla's crash report server currently only has debug information for +Mozilla builds and thus the crash reporter cannot work if you use a +build from a Linux distribution or if you compile from source code. In +these cases you will need to use one of the :ref:`alternative +methods <Alternative ways to get a stacktrace>` listed below. + +.. note:: + + **Note:** When filing a crash report, it is important to know whether + the crash occurs with `Firefox safe + mode <http://support.mozilla.com/kb/Safe+Mode>`__. This helps + engineers determine whether a particular + `extension <http://support.mozilla.com/kb/Troubleshooting+extensions+and+themes>`__ + or + `plugin <http://support.mozilla.com/kb/Troubleshooting+plugins>`__ + is the cause of the crash. + + +How to get a crash ID with the Mozilla Crash Reporter +----------------------------------------------------- + +1 - Crash and submit a report to the system. + +.. image:: img/crashreporter.png + +The Mozilla Crash Reporter window should automatically come up after Firefox crashes. +If you have any additional information about the crash, such as additional detail on +what you were doing at the time that may have triggered the crash, please enter it +into the comments box. Be sure that you **check the "Tell Mozilla about this crash"** +checkbox and click the restart button. The crash reporter should now submit the +crash report and Firefox should open again. + +.. note:: + + The "Details" button gives additional data about the incident, + however this is not useful in a bug report. + + +2 - Tell us the ID of the report you submitted. + +.. image:: img/crashlist.jpg + +To access all of your submitted reports type "about:crashes" into the Firefox address bar +and press enter. Firefox should open a list of IDs for your submitted crash reports. +Copy two or three of the IDs for the appropriate crashes and paste them into your +Bugzilla report. Please check the listed times to avoid copying the ID of an unrelated +crash report. + +.. note:: + + You can prefix a "bp-" to the beginning of an ID to make Bugzilla turn it + into a link: bp-a70759c6-1295-4160-aa30-bc4772090918 + + +How to get the crash ID if Firefox crashes on startup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox crashes on startup you can still access your submitted crash +reports. Crash reports are accessible from all Firefox profiles, so if a +`new +profile <https://support.mozilla.org/kb/profile-manager-create-remove-switch-firefox-profiles>`__ +does not crash you can use it to access them through "about:crashes" as above. + + +Accessing crash report IDs outside of Firefox +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you cannot load Firefox at all you can find the crash report files at +this location depending on your operating system: + +* Windows : ``%APPDATA%\Mozilla\Firefox\Crash Reports\submitted\`` +* macOS : ``~/Library/Application Support/Firefox/Crash Reports/submitted/`` +* Linux : ``~/.mozilla/firefox/Crash Reports/submitted/`` + +Each file in this folder contains one submitted crash report ID. You can +check the modified or creation time for each file to discern which crash +reports are relevant to your bug report. + +.. _Alternative ways to get a stacktrace: + +Alternative ways to get a stacktrace +------------------------------------ + +If the Mozilla crash reporter doesn't come up or isn't available you +will need to obtain a stacktrace manually: + + +Windows +~~~~~~~ + +See the article :ref:`Create a stacktrace with Windbg <How to get a stacktrace with WinDbg>` for information +on how to do this. + +For a full process dump, see :ref:`How to get a process dump with Windows +Task Manager`. + + +macOS +~~~~~ + +Run /Applications/Utilities/Console.app. Expand "~/Library/Logs" and +"CrashReporter", then look for logs for "firefox". + + +Linux +~~~~~ + +Note that for most distros, the package you need to get symbols for will +be something like "xulrunner", not "firefox". + + +Crash reports files on your computer +------------------------------------ + +When Breakpad initially catches a crash it first writes crash report +files (e.g. .dump and .extra files) into the 'pending' subdirectory of +its 'Crash Reports' directory. + +If Breakpad successfully sends the crash report to the reporting server +then, by default, the files added to the 'pending' subdirectory for the +crash are removed, and a .txt file is placed in the 'submitted' +directory containing the crash ID created by the reporting server. + +If you want Breakpad to leave the .dump and .extra files on your +computer so that you can examine them locally, then set the +MOZ_CRASHREPORTER_NO_DELETE_DUMP environment variable to 1. + +- Ubuntu: `Instructions from the Ubuntu + Team <https://wiki.ubuntu.com/MozillaTeam/Bugs#Obtain%20a%20backtrace%20from%20an%20apport%20crash%20report%20(using%20gdb)>`__ +- openSUSE: `General instructions from + openSUSE <https://en.opensuse.org/openSUSE:Bugreport_application_crashed>`__ +- Fedora: `Capturing Stack + Traces <https://fedoraproject.org/wiki/StackTraces>`__ +- Gentoo: `Debugging using + GDB <https://wiki.gentoo.org/wiki/Debugging_with_GDB>`__ diff --git a/docs/contributing/debugging/stacktrace_windbg.rst b/docs/contributing/debugging/stacktrace_windbg.rst new file mode 100644 index 0000000000..a0d2abfc25 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_windbg.rst @@ -0,0 +1,232 @@ +How to get a stacktrace with WinDbg +=================================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Introduction +------------ + +Sometimes you need to get a stacktrace (call stack) for a crash or hang +but `Breakpad <http://kb.mozillazine.org/Breakpad>`__ fails because it's +a special crash or a hang. This article describes how to get a +stacktrace in those cases with WinDbg on Windows. (To get a stacktrace +for Thunderbird or some other product, substitute the product name where +ever you see Firefox in this instructions.) + +Requirements +------------ + +To get such a stacktrace you need to install the following software: + +Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Microsoft distributes the Debugging Tools for Windows for free, those +include WinDbg which you will need here. Download it from `Install +Debugging Tools for +Windows <https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk>`__. +(*You'll want the 32-bit version*, even if you are using a 64-bit +version of Windows) Then install it, the standard settings in the +installation process are fine. + +A Firefox nightly or release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You need a Firefox version for which symbols are availables from the +:ref:`symbol server <Using The Mozilla Symbol Server>` to use +with WinDbg. You can use any `official nightly +build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released +version of Firefox from Mozilla. You can find the latest trunk nightly +builds under +`http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/>`__. + + +Debugging +--------- + +To begin debugging, ensure that Firefox is not already running and open +WinDbg from the Start menu. (Start->All Programs->Debugging Tools for +Windows->WinDbg) Next, open the **"File"** menu and choose **"Open +Executable..."**. In the file chooser window that appears, open the +firefox.exe executable in your Firefox program folder (C:\Program +Files\Mozilla Firefox). + +You should now see a "Command" text window with debug output at the top +and an input box at the bottom. Before debugging can start, several +commands must be entered into the one-line input box at the bottom of +the Command window. + +.. note:: + + Tip: All commands must be entered exactly as written, one line at a + time, into the bottom of the Command box. + + - Copying and pasting each line is the easiest method to avoid + mistakes + - Some commands start with a period (.) or a pipe character (|), + which is required. (The keystroke for a pipe character on US + keyboards is SHIFT+\) + - Submit the log file on a bug or via the support site, even if + nothing seems to happen during the debug process + + +Start debugging +~~~~~~~~~~~~~~~ + +Now that Firefox is opened in the debugger, you need to configure your +WinDbg to download symbols from the Mozilla symbol server. To load the +symbols, enter the three commands below, pressing enter after each one. +(More details are available at :ref:`symbol server <Using The Mozilla Symbol Server>`.) + +:: + + .sympath SRV*c:\symbols*http://symbols.mozilla.org/firefox;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols + .symfix+ c:\symbols + .reload /f + +Now wait for the symbols to download. This may take some time depending +on your connection speed; the total size of the Mozilla and Microsoft +symbols download is around 1.4GB. WinDbg will show "Busy" at the bottom +of the application window until the download is complete. + +Once the download is complete, you need to configure WinDbg to examine +child processes, ignore a specific event caused by Flash Player, and +record a log of loaded modules. You will also want to open a log file to +save data you collect. To do this, enter these four commands, pressing +enter after each one. + +:: + + .logopen /t c:\temp\firefox-debug.log + .childdbg 1 + .tlist + sxn gp + lm + +If you see firefox.exe listed in the output from .tlist more than once, +then you are already running the application and need to close the +running instance first before you start debugging, otherwise you won't +get useful results. + +Now run Firefox by opening the **Debug** menu and clicking **Go**. +**While Firefox is running, you will not be able to type any commands +into the debugger.** After it starts, try to reproduce the crash or +hanging issue that you are seeing. + +.. note:: + + If Firefox fails to start, and you see lines of text followed by a + command prompt in the debugger, a "breakpoint" may have been + triggered. If you are prompted for a command but don't see an error + about a crash, go back to the **Debug** menu and press **Go**. + +Once the browser crashes, you will see an error (such as "Access +violation") in the WinDbg Command window. If Firefox hangs and there is +no command prompt available in the debugger, open the **Debug** menu and +choose **Break.** Once the browser has crashed or been stopped, continue +with the steps below. + + +After the crash or hang +~~~~~~~~~~~~~~~~~~~~~~~ + +You need to capture the debug information to include in a bug comment or +support request. Enter these three commands, one at a time, to get the +stacktrace, crash/hang analysis and log of loaded modules. (Again, press +Enter after each command.) + +:: + + ~* kp + !analyze -v -f + lm + +After these steps are completed, find the file +**c:\temp\firefox-debug-(Today's Date).txt** on your hard drive. To +provide the information to the development community, submit this file +with a `support request <https://support.mozilla.com/>`__ or attach it +to a related bug on `Bugzilla <https://bugzilla.mozilla.org/>`__. + + +Producing a minidump +~~~~~~~~~~~~~~~~~~~~ + +Sometimes the stacktrace alone is not enough information for a developer +to figure out what went wrong. A developer may ask you for a "minidump" +or a "full memory dump", which are files containing more information +about the process. :ref:`You can easily produce minidumps from WinDBG and +provide them to developers <Capturing a minidump>`. + +FAQ + +Q: I am running Windows 7 (32-bit or 64-bit) and I see an exception in +the WinDbg command window that says 'ntdll32!LdrpDoDebuggerBreak+0x2c' +or 'ntdll32!LdrpDoDebuggerBreak+0x30'. What do I do now? + +A: If you see 'int 3' after either of those exceptions, you will need to +execute the following commands in WinDbg. + +:: + + bp ntdll!LdrpDoDebuggerBreak+0x30 + bp ntdll!LdrpDoDebuggerBreak+0x2c + eb ntdll!LdrpDoDebuggerBreak+0x30 0x90 + eb ntdll!LdrpDoDebuggerBreak+0x2c 0x90 + +| Make sure you enter them one at a time and press enter after each one. + If you use the 64-bit version of Windows, you need to replace "ntdll" + in these commands with "ntdll32". +| Q: The first four frames of my stack trace look like this: + +:: + + 0012fe20 7c90e89a ntdll!KiFastSystemCallRet + 0012fe24 7c81cd96 ntdll!ZwTerminateProcess+0xc + 0012ff20 7c81cdee kernel32!_ExitProcess+0x62 + + 0012ff34 6000179e kernel32!ExitProcess+0x14 + +This looks wrong to me?! + +A: You ran the application without the "Debug child processes also" +check box being checked. You need to detach the debugger and open the +application again, this time with the check box being checked. + +Q: WinDbg tells me that it is unable to verify checksum for firefox.exe. +Is this normal? + +A: Yes, this is normal and can be ignored. + +Q: Should I click yes or no when WinDbg asks me to "Save information for +workspace?" + +A: Click yes and WinDbg will save you from having to enter in the symbol +location for Firefox.exe in the future. Click no if you'd rather not +having WinDbg save this information. + +Q: I'm seeing "wow64" on top of each thread, is that ok ? + +A: No, you are running a 64 bit version of Windbg and trying to debug a +32 bit version of the mozilla software. Redownload and install the 32 +bit version of windbg. + + +Troubleshooting: Symbols will not download +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If symbols will not download no matter what you do, the problem may be +that Internet Explorer has been set to the **Work Offline** mode. You +will not receive any warnings of this in Windbg, Visual C++ or Visual +Studio. Even using the command line with symchk.exe to download symbols +will fail. This is because Microsoft uses Internet Explorer's internet & +proxy settings to download the symbol files. Check the File menu of +Internet Explorer to ensure "Work Offline" is unchecked. + + +See also +-------- + +- :ref:`symbol server <Using The Mozilla Symbol Server>` Maps addresses to human readable strings. +- :ref:`source server <Using The Mozilla Source Server>` Maps addresses to source code lines diff --git a/docs/contributing/debugging/understanding_crash_reports.rst b/docs/contributing/debugging/understanding_crash_reports.rst new file mode 100644 index 0000000000..9f8e5bbe03 --- /dev/null +++ b/docs/contributing/debugging/understanding_crash_reports.rst @@ -0,0 +1,325 @@ +Understanding Crash Reports +=========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +If a user experiences a crash they will be prompted to submit a raw +crash report, which is generated by Breakpad. The raw crash report is +received by `Socorro <https://github.com/mozilla/socorro>`__ which +`creates <https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py>`__ +a processed crash report. The processed crash report is based on the raw +crash report but also has a signature, classifications, and a number of +improved fields (e.g. OS, product, version). Many of the fields in both +the raw crash report and the processed crash report are viewable and +searchable on `crash-stats <https://crash-stats.mozilla.org/>`__. +Although there are two distinct crash reports, the raw and the +processed, people typically talk about a single "crash report" because +crash-stats mostly presents them in a combined way. + +Each crash report contains a wealth of data about the crash +circumstances. Despite this, many crash reports lack sufficient data for +a developer to understand why the crash occurred. As well as providing a +general overview, this page aims to highlight parts of a crash report +that may provide non-obvious insights. + +Note that most crash report fields are visible, but a few +privacy-sensitive parts of it are only available to users who are logged +in and have "minidump access". A relatively small number of users have +minidump access, and they are required to follow certain rules. For +access, see the `Protected Data Access docs on Crash Stats +<https://crash-stats.mozilla.org/documentation/protected_data_access/>`__. + +Each crash report has the following tabs: Details, Metadata, Modules, +Raw Dump, Extensions, and (optional) Correlations. + +Details tab +----------- + +The Details tab is the first place to look because it contains the most +important pieces of information. + +Primary fields +~~~~~~~~~~~~~~ + +| The first part of the Details tab shows a table containing the most + important crash report fields. It includes such things as when the + crash occurred, in which product and version, the crash kind, and + various details about the OS and configuration of the machine on which + the crash occurred. The following screenshot shows some of these + fields. +| |Example fields in the "Details" tab of a crash report| + +All fields have a tool-tip. For many fields, the tool-tip describes its +meaning. For all fields, the tool-tip indicates the key to use when you +want to do searches involving this field. (The field name is usually but +not always similar to the search key. E.g. the field "Adapter Device ID" +has the search key "adapter_device_id".) These descriptions are shown in +the `SuperSearchFields +API <https://crash-stats.mozilla.org/api/SuperSearchFields/>`__ and can be +`modified in super_search_fields.py <https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py>`__ +or by writing up a `bug in Socorro <https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro>`__. + +The fields present in this tab vary depending on the crash kind. Not all +fields are always present. + +The "Signature" field is the main identifier or label for a crash report. +Rather than considering each crash report in isolation, we want to put +crash reports into clusters so we can deal with groups of them at once. +An ideal clustering algorithm would put all crash reports with the same +root cause into a single cluster, and all crash reports with different +root causes into different clusters. The crash signature is our +imperfect but still useful attempt at such an algorithm. Most crash +signatures are based on the crashing stack trace, but some +special-purpose annotations are used to indicate particular kinds of +crashes. + +- ``Abort``: A controlled abort, e.g. via ``NS_RUNTIMEABORT``. + (Controlled aborts that occur via ``MOZ_CRASH`` or + ``MOZ_RELEASE_ASSERT`` currently don't get an ``Abort`` annotation, + but they do get a "MOZ_CRASH Reason" field.) +- ``OOM | <size>``, where ``<size>`` is one of ``large``, ``small``, + ``unknown``: an out-of-memory (OOM) abort. The ``<size>`` annotation + is determined by the "OOM Allocation Size" field; if that field is + missing ``<size>`` will be ``unknown``. +- ``hang``: a hang prior to shutdown. +- ``shutdownhang``: a hang during shutdown. +- ``IPCError-browser``: a problem involving IPC. If the parent Firefox + process detects that the child process has sent broken or + unprocessable IPDL data, or is not shutting down in a timely manner, + it kills the child process with a crash report. These crashes will + now have a signature that indicates why the process was killed, + rather than the child stack at the moment. + +When no special-purpose annotation is present and the signature begins +with a stack frame, it's usually a vanilla uncontrolled crash. The crash +cause can be determined from the "Crash Reason" field. Most commonly +it's a bad memory access. In that case, on Windows you can tell from the +reason field if the crash occurred while reading, writing or executing +memory (e.g. ``EXCEPTION_VIOLATION_ACCESS_READ`` indicates a bad memory +read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you +cannot tell from this field what kind of memory access it was. + +See `this +file <https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst>`__ +for a detailed explanation of the crash report signature generation +procedure, and for information on how modify this procedure. + +There are no fields that uniquely identify the user that a crash report +came from, but if you want to know if multiple crashes come from a +single user the "Install Time" field is a good choice. Use it in +conjunction with other fields that don't change, such as those +describing the OS or graphics card, for additional confidence. + +For bad memory accesses, the "Crash Address" field can give additional +indications what went wrong. + +- 0x0 is probably a null pointer deference[*]. +- Small addresses like 0x8 can indicate an object access (e.g. + ``this->mFoo``) via a null ``this`` pointer. +- Addresses like 0xfffffffffd8 might be stack accesses, depending on + the platform[*]. +- Addresses like 0x80cdefd3 might be heap accesses, depending on the + platform. +- Addresses may be poisoned: 0xe4 indicates the address comes from + memory that has been allocated by jemalloc but not yet initialized; + 0xe5 indicates the address comes from memory freed by jemalloc. The + JS engine also has multiple poison values defined in + ``js/src/jsutil.h``. + +[*] Note that due to the way addressing works on x86-64, if the crash +address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for +a Windows crash report, it's highly likely that the value is incorrect. +(There is a `bug +report <https://bugzilla.mozilla.org/show_bug.cgi?id=1493342>`__ open +for this problem.) You can sanity-check these crashes by looking at the +raw dump or minidump in the Raw Dump tab (see below). + +Note that for non-release builds the "Version" field represents multiple +different builds since nightly and beta version numbers are reused for +builds created over a series of days until the version number is bumped. +(The "Build ID" field can disambiguate.) It's not currently possible to +`restrict searches to a given version or +later <https://bugzilla.mozilla.org/show_bug.cgi?id=1401517>`__ (using +>= with a build ID and a given release channel may work around this). + +Some fields, such as "URL" and "Email Address", are privacy-sensitive +and are only visible to users with minidump access. + +The Windows-only "Total Virtual Memory" field indicates if the Firefox +build and OS are 32-bit or 64-bit. + +- A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows. +- A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows + (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox. +- A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit + Firefox. (The "Build Architecture" field should be "amd64" in this + case.) + +Some crash reports might contain a memory report. This memory report will +have been made some time before the crash, at a time when available +memory was low. In this case, a number of key measurements from the +memory report are shown in the Details tab, each one having a field name +starting with "MR:", short for "memory report". The full memory report +can be obtained in the Raw Dump tab (see below). + +Bug-related information +~~~~~~~~~~~~~~~~~~~~~~~ + +The second part of the Details tab shows bug-related information, as the +following screenshot shows. + +|Information relating to bug reports in the "Details" tab of a crash +report| + +The "Report this bug in" links can be used to easily file bug reports. +Each one links to a Bugzilla bug report creation page that has various +fields pre-filled, such as the crash signature. + +The "Related Bugs" section shows related bug reports, as determined by +the crash signature. + +Stack traces +~~~~~~~~~~~~ + +The third part of the Details tab shows the stack trace and thread +number of the crashing thread, as the following screenshot shows. + +|Information relating to threads in the "Details" tab of a crash report| + +Each stack frame has a link to the source code, when possible. If a +crash is new, the regressing changeset can often be identified by +looking for recent changes in the blame annotations for one or more of +the top stack frames. Blame annotations are also good for identifying +who might know about the code in question. + +Sometimes the highlighted source code is puzzling, e.g. the identified +line may not touch memory even though the crash is memory-related. This +can be caused by compiler optimizations. It's often better to look at +the disassembly (e.g. in a minidump) to understand exactly what code is +being executed. + +Stack frame entries take on a variety of forms. + +- The simplest are functions names, such as ``NS_InitXPCOM2``. +- Name/address pairs such as ``nss3.dll@0x1eb720`` are within system + libraries. +- Names such as ``F1398665248_____________________________`` ('F' + followed by many numbers then many underscores) are in Flash. +- Addresses such as ``@0xe1a850ac`` may indicate an address that wasn't + part of any legitimate code. If an address such as this occurs in the + first stack frame, the crash may be + `exploitable <https://developer.mozilla.org/en-US/docs/Mozilla/Security/Exploitable_crashes>`__. + +Stack traces for other threads can be viewed by clicking on the small +"Show other threads" link. + +If the crash report is for a hang, the crashing thread will be the +"watchdog" thread, which exists purely to detect hangs; its top stack +frame will be something +like\ :literal:`mozilla::`anonymous namespace'::RunWatchdog`. In that +case you should look at the other threads' stack traces to determine the +problem; many of them will be waiting on some kind of response, as shown +by a top stack frame containing a function like +``NtWaitForSingleObject`` or ``ZwWaitForMultipleObjects``. + +Metadata tab +------------ + +The Metadata tab is similar to the first part of the Details tab, +containing a table with various fields. These are the fields from the +raw crash report, ordered alphabetically by field name, but with +privacy-sensitive fields shown only to users with minidump access. There +is some overlap with the fields shown in the Details tab. + +Modules tab +----------- + +The modules tab shows all the system libraries loaded at the time of the +crash, as the following screenshot shows. + +|Table of modules in the "Modules" tab of a crash report| + +On Windows these are mostly DLLs, on Mac they are mostly ``.dylib`` +files, and on Linux they are mostly ``.so`` files. + +This information is most useful for Windows crashes, because DLLs loaded +by antivirus software or malware often cause Firefox to crash. +Correlations between loaded modules and crash signatures can be seen in +the "Correlations" tab (see below). + +`This page <https://support.mozilla.org/en-US/kb/helping-crashes>`__ +says that files lacking version/debug identifier/debug filename are +likely to be malware. + +Raw Dump tab +------------ + +The first part of the Raw Dump tab shows the raw crash report, in JSON +format. Once again, privacy-sensitive fields are shown only to users +with minidump access. + +|JSON data in the "Raw Dump" tab of a crash report| + +For users with minidump access, the second part of the Raw Dump tab has +some links, as the following screenshot shows. + +|Links to downloadable files in the "Raw Dump" tab of a crash report| + +These links are to the following items. + +#. A minidump. Minidumps can be extremely useful in understanding a + crash report; see :ref:`this page <Debugging A Minidump>` for an + explanation how to use them. +#. The aforementioned JSON raw crash report. +#. The memory report contained within the crash report. +#. The unredacted crash report, which has additional information. + +Extensions tab +-------------- + +The Extensions tab shows which extensions are installed and enabled. + +|Table of extensions in the "Extensions" tab of a crash report| + +Usually it just shows an ID rather than the proper extension name. + +Note that several extensions ship by default with Firefox and so will be +present in almost all crash reports. (The exact set of default +extensions depends on the release channel.) The least obvious of these +has an Id of ``{972ce4c6-7e08-4474-a285-3208198ce6fd}``, which is the +default Firefox theme. Some (but not all) of the other extensions +shipped by default have the following Ids: ``webcompat@mozilla.org``, +``e10srollout@mozilla.org``, ``firefox@getpocket.com``, +``flyweb@mozilla.org``, ``loop@mozilla.org``. + +If an extension only has a hexadecimal identifier, a Google search of +that identifier is usually enough to identify the extension's name. + +This information is useful because some crashes are caused by +extensions. Correlations between extensions and crash signatures can be +seen in the "Correlations" tab (see below). + +Correlations tab +---------------- + +This tab is only shown when crash-stats identifies correlations between +a crash and modules or extensions that are present, which happens +occasionally. + +See also +-------- + +- `A talk about understanding crash + reports <https://air.mozilla.org/a-talk-about-understanding-crash-reports/>`__, + by David Baron, from March 2016. +- :ref:`A guide to searching crash reports` + +.. |Example fields in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13579/Details1.png +.. |Information relating to bug reports in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13581/Details2.png +.. |Information relating to threads in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13583/Details3.png +.. |Table of modules in the "Modules" tab of a crash report| image:: https://mdn.mozillademos.org/files/13593/Modules1.png +.. |JSON data in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/13595/RawDump1.png +.. |Links to downloadable files in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/14047/raw-dump-links.png +.. |Table of extensions in the "Extensions" tab of a crash report| image:: https://mdn.mozillademos.org/files/13599/Extensions1.png diff --git a/docs/contributing/directory_structure.rst b/docs/contributing/directory_structure.rst new file mode 100644 index 0000000000..fd447278c2 --- /dev/null +++ b/docs/contributing/directory_structure.rst @@ -0,0 +1,565 @@ +Firefox Source Code Directory Structure +======================================= + +This article provides an overview of what the various directories contain. + +To simply take a look at the Firefox source code, you do not need to +download it. You can look at the source directly with your web browser +using Searchfox (start at https://searchfox.org/mozilla-central/source for +the complete firefox source code of branch HEAD). + +In order to modify the source, you have to acquire it either by +downloading a :ref:`snapshot <Mercurial Overview>` of the sources or +by checking out the current sources from +:ref:`the repository <Firefox Contributors' Quick Reference>`. + +This document describes the directory structure -- i.e., directories that +are used by at least some of the +Mozilla project's client products. There are other directories in the +other Mozilla repository, such as those for Web tools and those for the +Classic codebase. + +See the `more detailed overview of the pieces of Gecko <https://wiki.mozilla.org/Gecko:Overview>`__. + +.cargo +------ + +Configuration files for the `Cargo package +manager <https://crates.io/>`__. + +.vscode +------- + +Configuration files used by the `Visual Studio Code +IDE <https://code.visualstudio.com/>`__ when working in the +mozilla-central tree. + +accessible +---------- + +Files for accessibility (i.e., MSAA (Microsoft Active Accessibility), +ATK (Accessibility Toolkit, used by GTK) support files). See +`Accessibility <https://developer.mozilla.org/docs/Web/Accessibility>`__. + + +browser +------- + +Contains the front end code (in XUL, Javascript, XBL, and C++) for the +Firefox desktop browser. Many of these files started off as a copy of files in +`xpfe <#xpfe>`__. + +browser/extensions +------------------ + +Contains `PDF.js <https://mozilla.github.io/pdf.js/>`__ and +`WebCompat <https://github.com/mozilla/webcompat-addon>`__ built-in extensions. + +browser/themes +-------------- + +Contains images and CSS files to skin the browser for each OS (Linux, +Mac and Windows) + +build +----- + +Miscellaneous files used by the build process. See also `config <#config>`__. + +caps +---- + +Capability-based web page security management. It contains C++ interfaces +and code for determining the capabilities of content based on the +security settings or certificates (e.g., VeriSign). See `Component +Security <https://www.mozilla.org/projects/security/components/>`__ . + +chrome +------ + +:ref:`Chrome registry <Chrome Registration>` used with `toolkit <#toolkit>`__/. +These files were originally copies of files in `rdf/chrome/`. + +config +------ + +More files used by the build process, common includes for the makefiles, +etc. + + +devtools +-------- + +The Firefox Developer Tools server and client components. See :ref:`contributor <devtools-contributor-doc>` and :ref:`user <devtools-user-doc>` documentation. + + +docs +---- + +Contains the documentation configuration (`Sphinx <http://www.sphinx-doc.org/>`__ based), the index page +and the contribution pages. + + +docshell +-------- + +Implementation of the docshell, the main object managing things related +to a document window. Each frame has its own docshell. It contains +methods for loading URIs, managing URI content listeners, etc. It is the +outermost layer of the embedding API used to embed a Gecko browser into +an application. + +dom +--- + +- :ref:`IDL definitions <XPIDL>` of the interfaces defined by + the DOM specifications and Mozilla extensions to those interfaces + (implementations of these interfaces are primarily, but not + completely, in `content <#content>`__). +- The parts of the connection between JavaScript and the + implementations of DOM objects that are specific both to JavaScript + and to the DOM. +- Implementations of a few of the core "DOM Level 0" objects, such as + `window <https://developer.mozilla.org/docs/Web/API/Window>`__ , `window.navigator <https://developer.mozilla.org/docs/Web/API/Window/navigator>`__, `window.location <https://developer.mozilla.org/docs/Web/API/Window/location>`__, etc. + +editor +------ + +The editor directory contains XUL/Javascript for the embeddable editor +component, which is used for the HTML Editor("Composer"), for plain and +HTML mail composition, and for text fields and text areas throughout the +product. The editor is designed like a +"browser window with editing features": it adds some special classes for +editing text and managing transaction undo/redo, but reuses browser code +for nearly everything else. + +extensions +---------- + +Contains several extensions to mozilla, which can be enabled at +compile-time using the ``--enable-extensions`` configure argument. + +Note that some of these are now built specially and not using the +``--enable-extensions`` option. For example, disabling xmlextras is done +using ``--disable-xmlextras``. + + +extensions/auth +--------------- + +Implementation of the negotiate auth method for HTTP and other +protocols. Has code for SSPI, GSSAPI, etc. See `Integrated +Authentication <https://www.mozilla.org/projects/netlib/integrated-auth.html>`__. + + +extensions/pref +--------------- + +Preference-related extensions. + +extensions/spellcheck +--------------------- + +Spellchecker for mailnews and composer. + +extensions/universalchardet +--------------------------- + +Detects the character encoding of text. + +gfx +--- + +Contains interfaces that abstract the capabilities of platform specific +graphics toolkits, along with implementations on various platforms. +These interfaces provide methods for things like drawing images, text, +and basic shapes. It also contains basic data structures such as points +and rectangles used here and in other parts of Mozilla. + +gradle +------ + +Containing files related to a Java build system. + +hal +--- + +Contains platform specified functions (e.g. obtaining battery status, +sensor information, memory information, Android +alarms/vibrate/notifications/orientation, etc) + +image +----- + +Image rendering library. Contains decoders for the image formats Firefox +supports. + +intl +---- + +Internationalization and localization support. See +`L10n:NewProjects <https://wiki.mozilla.org/L10n:NewProjects>`__. + +intl/locale +----------- + +Code related to determination of locale information from the operating +environment. + +intl/lwbrk +---------- + +Code related to line breaking and word breaking. + +intl/strres +----------- + +Code related to string resources used for localization. + +intl/uconv +---------- + +Code that converts (both ways: encoders and decoders) between UTF-16 and +many other character encodings. + +intl/unicharutil +---------------- + +Code related to implementation of various algorithms for Unicode text, +such as case conversion. + +ipc +--- + +Container for implementations of IPC (Inter-Process Communication). + +js/src +------ + +The JavaScript engine, also known as +:ref:`SpiderMonkey <SpiderMonkey>`. +See also `JavaScript <https://developer.mozilla.org/docs/JavaScript>`__. + +js/xpconnect +------------ + +Support code for calling JavaScript code from C++ code and C++ code from +JavaScript code, using XPCOM interfaces. See +`XPConnect <https://developer.mozilla.org/docs/XPConnect>`__. + +layout +------ + +Code that implements a tree of rendering objects that describe the types +and locations of the objects that are displayed on the screen (such as +CSS boxes, tables, form controls, XUL boxes, etc.), and code that +manages operations over that rendering tree (such as creating and +destroying it, doing layout, painting, and event handling). See +`documentation <https://www.mozilla.org/newlayout/doc/>`__ and `other +information <https://www.mozilla.org/newlayout/>`__. + +layout/base +----------- + +Code that deals with the rendering tree. + +layout/forms +------------ + +Rendering tree objects for HTML form controls. + +layout/generic +-------------- + +The basic rendering object interface and the rendering tree objects for +basic CSS boxes. + +layout/mathml +------------- + +Rendering tree objects for `MathML <https://developer.mozilla.org/docs/Web/MathML>`__. + +layout/svg +---------- + +Rendering tree objects for `SVG <https://developer.mozilla.org/docs/Web/SVG>`__. + +layout/tables +------------- + +Rendering tree objects for CSS/HTML tables. + +layout/xul +---------- + +Additional rendering object interfaces for `XUL <https://developer.mozilla.org/docs/XUL>`__ and +the rendering tree objects for XUL boxes. + +media +----- + +Contains sources of used media libraries for example *libpng*. + +memory +------ + +Cross-platform wrappers for *memallocs* functions etc. + +mfbt +---- + +Implementations of classes like *WeakPtr*. Multi-platform *assertions* +etc. + +mobile +------ + +mobile/android +-------------- + +Firefox for Android and Geckoview + +modules +------- + +Compression/Archiving, math library, font (and font compression), +Preferences Library + +modules/libjar +-------------- + +Code to read zip files, used for reading the .jar files that contain the +files for the mozilla frontend. + +modules/libpref +--------------- + +Library for reading and writing preferences. + +modules/zlib +------------ + +Source code of zlib, used at least in the networking library for +compressed transfers. + +mozglue +------- + +Glue library containing various low-level functionality, including a +dynamic linker for Android, a DLL block list for Windows, etc. + +netwerk +------- + +:ref:`Networking library <Networking>`, also known as Necko. +Responsible for doing actual transfers from and to servers, as well as +for URI handling and related stuff. + +netwerk/cookie +-------------- + +Permissions backend for cookies, images, etc., as well as the user +interface to these permissions and other cookie features. + +nsprpub +------- + +Netscape Portable Runtime. Used as an abstraction layer to things like +threads, file I/O, and socket I/O. See :ref:`NSPR`. + +nsprpub/lib +----------- + +Mostly unused; might be used on Mac? + +other-licenses +-------------- + +Contains libraries that are not covered by the MPL but are used in some +Firefox code. + +parser +------ + +Group of structures and functions needed to parse files based on +XML/HTML. + +parser/expat +------------ + +Copy of the expat source code, which is the XML parser used by mozilla. + +parser/html +----------- + +The HTML parser (for everything except about:blank). + +parser/htmlparser +----------------- + +The legacy HTML parser that's still used for about:blank. Parts of it +are also used for managing the conversion of the network bytestream into +Unicode in the XML parsing case. + +parser/xml +---------- + +The code for integrating expat (from parser/expat) into Gecko. + +python +------ + +Cross module python code. + +python/mach +----------- + +The code for the :ref:`Mach` building tool. + +security +-------- + +Contains NSS and PSM, to support cryptographic functions in mozilla +(like S/MIME, SSL, etc). See :ref:`Network Security Services (NSS)` +and +`Personal Security Manager +(PSM) <https://www.mozilla.org/projects/security/pki/psm/>`__. + +services +-------- + +Firefox accounts and sync (history, preferences, tabs, bookmarks, +telemetry, startup time, which addons are installed, etc). See +`here <https://docs.services.mozilla.com/>`__. + +servo +----- + +`Servo <https://servo.org/>`__, the parallel browser engine project. + +startupcache +------------ + +XXX this needs a description. + +storage +------- + +`Storage <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Storage>`__: XPCOM wrapper for sqlite. Wants to +unify storage of all profile-related data. Supersedes mork. See also +`Unified Storage <https://wiki.mozilla.org/Mozilla2:Unified_Storage>`__. + +taskcluster +----------- + +Scripts and code to automatically build and test Mozilla trees for the +continuous integration and release process. + +testing +------- + +Common testing tools for mozilla codebase projects, test suite +definitions for automated test runs, tests that don't fit anywhere else, +and other fun stuff. + +third_party +----------- + +Vendored dependencies maintained outside of Mozilla. + +toolkit +------- + +The "new toolkit" used by Thunderbird, Firefox, etc. This contains +numerous front-end components shared between applications as well as +most of the XBL-implemented parts of the XUL language (most of which was +originally forked from versions in `xpfe/`). + +toolkit/mozapps/extensions/test/xpinstall +----------------------------------------- + +The installer, which contains code for installing Mozilla and for +installing XPIs/extensions. This directory also contains code needed to +build installer packages. See `XPInstall <https://developer.mozilla.org/docs/XPInstall>`__ and +the `XPInstall project +page <https://www.mozilla.org/projects/xpinstall/>`__. + +tools +----- + +Some tools which are optionally built during the mozilla build process. + +tools/lint +---------- + +The linter declarations and configurations. +See `linting documentation </code-quality/lint/>`_ + +uriloader +--------- + +uriloader/base +-------------- + +Content dispatch in Mozilla. Used to load uris and find an appropriate +content listener for the data. Also manages web progress notifications. +See `Document Loading: From Load Start to Finding a +Handler <https://www.mozilla.org/docs/docshell/uri-load-start.html>`__ +and `The Life Of An HTML HTTP +Request <https://www.mozilla.org/docs/url_load.html>`__. + + +uriloader/exthandler +-------------------- + +Used to handle content that Mozilla can't handle itself. Responsible for +showing the helper app dialog, and generally for finding information +about helper applications. + +uriloader/prefetch +------------------ + +Service to prefetch documents in order to have them cached for faster +loading. + +view +---- + +View manager. Contains cross-platform code used for painting, scrolling, +event handling, z-ordering, and opacity. Soon to become obsolete, +gradually. + +widget +------ + +A cross-platform API, with implementations on each platform, for dealing +with operating system/environment widgets, i.e., code related to +creation and handling of windows, popups, and other native widgets and +to converting the system's messages related to painting and events into +the messages used by other parts of Mozilla (e.g., `view/` and +`content/`, the latter of which converts many of the +messages to yet another API, the DOM event API). + +xpcom +----- + +`Cross-Platform Component Object Model </en-US/docs/XPCOM>`__. Also +contains data structures used by the rest of the mozilla code. See also +`XPCOM Project <https://www.mozilla.org/projects/xpcom/>`__. + +xpfe +---- + +XPFE (Cross Platform Front End) is the SeaMonkey frontend. It contains +the XUL files for the browser interface, common files used by the other +parts of the mozilla suite, and the XBL files for the parts of the XUL +language that are implemented in XBL. Much of this code has been copied +to `browser/` and `toolkit/` for use in +Firefox, Thunderbird, etc. + + +xpfe/components +--------------- + +Components used by the Mozilla frontend, as well as implementations of +interfaces that other parts of mozilla expect. diff --git a/docs/contributing/editor.rst b/docs/contributing/editor.rst new file mode 100644 index 0000000000..1ce083d4fc --- /dev/null +++ b/docs/contributing/editor.rst @@ -0,0 +1,26 @@ +Editor / IDE integration +======================== + +You can use any editor or IDE to contribute to Firefox, as long as it can edit +text files. However, there are some steps specific to mozilla-central that may +be useful for a better development experience. This page attempts to document +them. + +.. note:: + + Visual Studio Code is the recommended editor for Firefox development. + Not because it is better than the other editors but because we decided to + focus our energy on a single editor. + +.. note:: + + This page is a work in progress. Please enhance this page with instructions + for your favourite editor. + +.. toctree:: + :maxdepth: 1 + + editors/vscode + editors/emacs + editors/vim + editors/others diff --git a/docs/contributing/editors/emacs.rst b/docs/contributing/editors/emacs.rst new file mode 100644 index 0000000000..fbd41195a1 --- /dev/null +++ b/docs/contributing/editors/emacs.rst @@ -0,0 +1,121 @@ +Emacs +===== + +ESLint +------ + +See `the devtools documentation <https://wiki.mozilla.org/DevTools/CodingStandards#Running_ESLint_in_Emacs>`__ +that describes how to integrate ESLint into Emacs. + +C/C++ Development Packages +-------------------------- + +General Guidelines to Emacs C++ Programming +------------------------------------------- + +The following guides give an overview of the C++ editing capabilities of emacs. + +It is worth reading through these guides to see what features are available. +The rest of this section is dedicated to Mozilla/Gecko specific setup for +packages. + + + * `C/C++ Development Environment for Emacs <https://tuhdo.github.io/c-ide.html>`__ + * `Emacs as C++ IDE <https://syamajala.github.io/c-ide.html>`__ + +rtags (LLVM/Clang-based Code Indexing) +-------------------------------------- + +Instructions for the installation of rtags are available at the +`rtags github repo <https://github.com/Andersbakken/rtags>`__. + +rtags requires a :ref:`compilation database <CompileDB back-end-compileflags>`. + +In order for rtags to index correctly, included files need to be copied and +unified compilation files need to be created. Either run a full build of the +tree, or if you only want indexes to be generated for the moment, run the +following commands (assuming you're in the gecko repo root): + +.. code:: + + cd gecko_build_directory + make export + ./config.status + +To increase indexing speed, it's best to remove unified build files and test +files from database updates. This can be done by creating a :code:`~/.rdmrc` +file with the following contents, with :code:`[src_dir]` replaced with either +the repo or build directory for your checkout: + +.. code:: + + -X */[src_dir]/*Unified*;*/[src_dir]/*/test/*;*/[src_dir]/*/tests/* + +Once the rdm daemon is running, the compilation database can be added to rtags +like so: + +.. code:: + + rc -J [gecko_build_directory]/compile_commands.json + +Note that this process will take a while initially. However, once the database +is built, it will only require incremental updates. As long as the rdm daemon +is running in the background, the database will be updated based on changes to +files. + +irony (LLVM/Clang-based Code Completion) +---------------------------------------- + +Instructions on the installation of irony-mode are available at the +`irony-mode github repo <https://github.com/Sarcasm/irony-mode>`__. + +irony-mode requires a :ref:`compilation database <CompileDB back-end-compileflags>`. + +Note that irony-mode, by default, uses elisp to parse the +:code:`compile_commands.json` file. As gecko is a very large codebase, this +file can easily be multiple megabytes, which can make irony-mode take multiple +seconds to load on a gecko file. + +It is recommended to use `this fork of irony-mode <https://github.com/Hylen/irony-mode/tree/compilation-database-guessing-4-pull-request>`__, +which requires the boost System and Filesystem libraries. + +`Checking the bug to get this patch into the mainline of irony-mode <https://github.com/Sarcasm/irony-mode/issues/176>`__ +is recommended, to see if the fork can be used or if the mainline repo can be +used. Using the Boost version of the irony-mode server brings file load times +to under 1s. + +Projectile (Project Management) +------------------------------- + +Instructions on the installation of projectile are available at the +`projectile github repo <https://github.com/bbatsov/projectile>`__. + +Projectile comes preconfigured for many project types. Since, gecko uses its +own special build system (mach), a new project type needs to be added. This can +be done via adding the following elisp configuration command to your emacs +configuration file. + +.. code:: + + (projectile-register-project-type 'gecko + '("mach" "moz.build") + "python mach --log-no-times build" + "python mach mochitest" + "python mach run") + +Assuming projectile-global-mode is on, this will allow projectile to run the +correct commands whenever it is working in a gecko repo. + +gdb +^^^ + +Emacs comes with great integration with gdb, especially when using +`gdb-many-windows <https://www.gnu.org/software/emacs/manual/html_node/emacs/GDB-User-Interface-Layout.html>`__. + +However, when gdb is invoked via mach, some special arguments +need to be passed in order to make sure the correct display mode is used. To +use M-x gdb with mach on firefox, use the following command: + +.. code:: + + gecko_repo_directory/mach run --debug --debugparams=-i=mi diff --git a/docs/contributing/editors/others.rst b/docs/contributing/editors/others.rst new file mode 100644 index 0000000000..b753ff1042 --- /dev/null +++ b/docs/contributing/editors/others.rst @@ -0,0 +1,41 @@ +Eclipse +======= + +You can generate an Eclipse project by running: + +.. code:: + + ./mach ide eclipse + +See also the `Eclipse CDT <https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Eclipse/Eclipse_CDT>`__ docs on MDN. + +Visual Studio +============= + +You can run a Visual Studio project by running: + +.. code:: + + ./mach ide visualstudio + +.. _CompileDB back-end-compileflags: + +CompileDB back-end / compileflags +================================= + +You can generate a :code:`compile_commands.json` in your object directory by +running: + +.. code:: + + ./mach build-backend --backend=CompileDB + +This file, the compilation database, is understood by a variety of C++ editors / IDEs +to provide auto-completion capabilities. You can also get an individual compile command by +running: + +.. code:: + + ./mach compileflags path/to/file + +This is how the :ref:`VIM <VIM>` integration works, for example. diff --git a/docs/contributing/editors/vim.rst b/docs/contributing/editors/vim.rst new file mode 100644 index 0000000000..0cce21d4e9 --- /dev/null +++ b/docs/contributing/editors/vim.rst @@ -0,0 +1,75 @@ +Vim / Neovim +============ + +AutoCompletion +-------------- + +For C++, anything that can use an LSP like :code:`coc.nvim`, +:code:`nvim-lspconfig`, or what not, should work as long as you generate a +:ref:`compilation database <CompileDB back-end-compileflags>` and point to it. + +Additionally, `YouCompleteMe <https://github.com/ycm-core/YouCompleteMe/>`__ +works without the need of a C++ compilation database as long as you have run +:code:`./mach build` or :code:`./mach configure`. Configuration for this lives +in :searchfox:`.ycm_extra_conf <.ycm_extra_conf>` at the root of the repo. + +Rust auto-completion should work both with Rust's LSP :code:`rust-analyzer`. + +Make sure that the LSP is configured in a way that it detects the root of the +tree as a workspace, not the crate you happen to be editing. For example, the +default of :code:`nvim-lspconfig` is to search for the closest +:code:`Cargo.toml` file, which is not what you want. You'd want something like: + +.. code :: + + root_dir = lspconfig.util.root_pattern(".git", ".hg") + +You also need to set some options to get full diagnostics: + +.. code :: + + "rust-analyzer.server.extraEnv": { + "CARGO_TARGET_DIR": "/path/to/objdir" + }, + "rust-analyzer.check.overrideCommand": [ "/path/to/mach", "--log-no-times", "cargo", "check", "--all-crates", "--message-format-json" ], + "rust-analyzer.cargo.buildScripts.overrideCommand": [ "/path/to/mach", "--log-no-times", "cargo", "check", "--all-crates", "--message-format-json" ], + +The easiest way to make these work out of the box is using +`neoconf <https://github.com/folke/neoconf.nvim/>`__, which +automatically supports importing VSCode configuration files. +:code:`./mach ide vscode --no-interactive` will then generate the right +configuration for you. + +ESLint +------ + +The easiest way to integrate ESLint with VIM is using the `Syntastic plugin +<https://github.com/vim-syntastic/syntastic>`__. + +In order for VIM to detect jsm files as JS you might want something like this +in your :code:`.vimrc`: + +.. code:: + + autocmd BufRead,BufNewFile *.jsm set filetype=javascript + +:code:`mach eslint --setup` installs a specific ESLint version and some ESLint +plugins into the repositories' :code:`node_modules`. + +You need something like this in your :code:`.vimrc` to run the checker +automatically on save: + +.. code:: + + autocmd FileType javascript,html,xhtml let b:syntastic_checkers = ['javascript/eslint'] + +You need to have :code:`eslint` in your :code:`PATH`, which you can get with +:code:`npm install -g eslint`. You need at least version 6.0.0. + +You can also use something like `eslint_d +<https://github.com/mantoni/eslint_d.js#editor-integration>`__ which should +also do that automatically: + +.. code:: + + let g:syntastic_javascript_eslint_exec = 'eslint_d' diff --git a/docs/contributing/editors/vscode.rst b/docs/contributing/editors/vscode.rst new file mode 100644 index 0000000000..e8c745342e --- /dev/null +++ b/docs/contributing/editors/vscode.rst @@ -0,0 +1,184 @@ +Visual Studio Code +================== + +.. toctree:: + :maxdepth: 1 + :glob: + +General Knowledge +----------------- + +`VSCode <https://code.visualstudio.com/>`__ is a multi-platform open-source programming editor developed by Microsoft and volunteers. +It has support for many programming languages using extensions. +This is the recommended editor for Firefox development. + +For more general information on the VSCode project see the `repository <https://github.com/Microsoft/vscode/>`__. + +Recommended extensions +---------------------- + +VS Code provides number of extensions for JavaScript, Rust, etc. By default, +Firefox source tree comes with its own set of recommendations of Visual Studio +Code extensions. They will be offered when you first open the project. + +If you need to refer to them later, the extensions are listed in +`.vscode/extensions.json <https://searchfox.org/mozilla-central/source/.vscode/extensions.json>`__. + +For Rust development, the `rust-analyzer <https://marketplace.visualstudio.com/items?itemName=matklad.rust-analyzer>`__ extension is recommended. +`See the manual <https://rust-analyzer.github.io/manual.html>`__ for more information. + +Getting setup +------------- + +Close `VS Code` if it is already open, then build the configuration for `VS Code` +by simply running from the terminal: + +.. code:: + + ./mach ide vscode + +This will automatically set some of the recommended preferences for the workspace, +and if you are set up for a full build, it will enable clangd and rust integrations. + +If successful, `VS Code` will open at the end. You do not need to run this command +every time to open `VS Code`, you may open it in the normal way. + +If you are running full builds, the command above will set up the `Clangd` +integration so that subsequent invocations of ``./mach build`` run and update the +integration. + +.. note:: + + If `VS Code` is already open with a previous configuration generated, please make sure to + restart `VS Code` otherwise the new configuration will not be used, and the `compile_commands.json` + needed by `clangd` server will not be refreshed. This is a known `bug <https://github.com/clangd/vscode-clangd/issues/42>`__ + in `clangd-vscode` extension + +Ignore Files in Mercurial Repositories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + + This is automatically done when running ``./mach ide vscode`` but may be + changed manually. + +When using Mercurial in mozilla-central, VS Code will treat your build directories as ordinary directories by default, causing some undesirable behavior including long indexing times, Go to Definition will open files in the build directory instead of the source tree, and Search Files by Name will find duplicate files from the source tree and the build directory (note: when using Git, VS Code will not do this since it reads ``.gitignore``). You can follow these directions to have VS Code largely ignore your build directories: + +#. Go to Preferences -> Settings +#. Search "exclude" in the Settings +#. (optional) Select "Workspace" below the search bar to only change this setting for the mozilla-central repository +#. Under "Files: Exclude", click "Add Pattern", type ``obj-*`` (assuming your build directory names start with the default text, ``obj-``), and click "OK" +#. Repeat the step above for the "Files: Watcher Exclude" setting +#. Reload VS Code: the easiest way to do this is to quit and reopen it. + +Despite excluding the build directories above, Go to Definition will still correctly open files that only appear in the build directory such as generated source code. + +Recommended Preferences +~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + + These are automatically set when running ``./mach ide vscode`` but may be + changed manually. These are set only for particular file types. + +* ``"editor.formatOnSave": true`` + * This will turn on automatically fixing formatting issues when you save a file. +* ``"editor.defaultFormatter": "esbenp.prettier-vscode"`` + * This sets the default formatter to prettier using the recommended prettier + extension. + +``*.jsm`` and ``*.sjs`` file extensions should also be associated with JavaScript: + +.. code:: + + "files.associations": { + "*.jsm": "javascript", + "*.sjs": "javascript", + }, + +C/C++ Features and Support +-------------------------- + +For C++ support we offer an out of the box configuration based on +`clangd <https://clangd.llvm.org>`__. + +Leveraging the `clang` toolchain compiler we now have support in the IDE for the following features: + +**1.** Syntax highlighting + +**2.** IntelliSense with comprehensive code completion and suggestion + +.. image:: ../img/auto_completion.gif + +**3.** Go-to definition and Go-to declaration + +.. image:: ../img/goto_definition.gif + +**4.** Find all references + +.. image:: ../img/find_references.gif + +**5.** Open type hierarchy + +.. image:: ../img/type_hierarchy.gif + +**6.** Rename symbol, all usages of the symbol will be renamed, including declaration, definition and references + +.. image:: ../img/rename_symbol.gif + +**7.** Code formatting, based on `clang-format` that respects our coding standard using the `.clang-format` and `.clang-format-ignore` files. Format can be performed on an entire file or on a code selection + +.. image:: ../img/format_selection.gif + +**8.** Inline parsing errors with limited auto-fix hints + +.. image:: ../img/diagnostic_error.gif + +**9.** Basic static-code analysis using `clang-tidy` and our list of enabled checkers. (This is still in progress not all checkers are supported by `clangd`) + +Clangd-specific Commands +------------------------ + +Clangd supports some commands that are specific to C/C++: + +.. code:: + + "clangd.switchheadersource" + +This command navigates from the currently open header file to its corresponding source file (if there is one), or vice versa. + +This command can be invoked from the command menu (activated via ``F1``), or using its keybinding of ``Alt+o`` (``Alt+cmd+o`` on Mac). The keybinding can also be customized in ``Keyboard Shortcuts``. + +Remote Development over SSH +--------------------------- + +VS Code provides an `extension <https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh>`__ that lets you use any remote machine with a SSH server as your development environment. No matter if it's Linux based, macOS or Windows, as long as the target machine offers a SSH connection, it can be used for development. + +No source code needs to be on your local machine to use VS Code remotely since the extension runs commands and other extensions directly on the remote machine. + +In order to setup a connection please follow these steps: + +**1.** Open VS Code and select from the left side panel ``Remote Explorer`` + +.. image:: ../img/remote_explorer.png + +**2.** From the ``Remote Explorer`` panel select ``SSH Targets`` and click on ``Add`` and enter the connection details + +.. image:: ../img/remote_explorer_add.png + +.. image:: ../img/remote_explorer_add_wind.png + +**3.** Click on the connection that you just configured at the previous step + +**4.** Finally you should be connected to the desired remote SSH server + +.. image:: ../img/connection_done.png + +Please note that during the first connection VS Code will install itself remotely and also install all of the needed dependencies. + + + +Filing Bugs +----------- + +Bugs should be filed in the `Firefox Build System` product under `Developer Environment Integration`, preferably blocking `Bug 1662709 <https://bugzilla.mozilla.org/show_bug.cgi?id=1662709>`__. diff --git a/docs/contributing/engineering_show_and_tell.rst b/docs/contributing/engineering_show_and_tell.rst new file mode 100644 index 0000000000..87c2013bb8 --- /dev/null +++ b/docs/contributing/engineering_show_and_tell.rst @@ -0,0 +1,83 @@ +Engineering Show and Tell +========================= + +The engineering teams at Mozilla have put together a series of 5 minute long +tips and tricks to help boost productivity. You can find the recordings below. + +August 2021 +----------- + +`Link to the August 2021 recording <https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=bd0c503d-903f-4829-9c9d-ad7c011cee9b>`_ + +Links + +- Mossop: Mercurial Tips n Tricks + + - https://github.com/Mossop/dotfiles/blob/main/shared/hg/hgrc + - https://mikeconley.github.io/documents/How_mconley_uses_Mercurial_for_Mozilla_code + +- mak: Gmail filtering with Google Apps Script + + - https://script.google.com/home + - https://paste.mozilla.org/9zEDnAiA#L9 + - https://developers.google.com/apps-script/reference/ + +- mhowell: Windows Tools You Might Not Know About + + - https://docs.google.com/document/d/1DyeTPIEz3SIcw7ldvSGKUVp4XP34Ue-_pV8lYmRJX6I/ + +- bryce: Debugging Remote Resources Via Proxy + + - https://www.telerik.com/fiddler + - https://www.charlesproxy.com/ + - https://mitmproxy.org/ + +- jody: CircleCI for VS Code + + - https://marketplace.visualstudio.com/items?itemName=jodyh.circleci-vscode + +- sfink: Moz workflow aids for post-landing pruning, taskcluster job replication & more + + - https://github.com/hotsphink/sfink-tools + - https://hg.sr.ht/~sfink/sfink-tools + +- mconley: Multiple mozconfigs with mozconfigwrapper + + - https://github.com/ahal/mozconfigwrapper + +February 2021 +------------- + +`Link to the February 2021 recording <https://mozilla.zoom.us/rec/share/RvN62-Y3ByGmeyQChmsqXy6WXAE3iJETeB2yNj1xq4-z3c80ewwFaz-EAVnoHSo-.8FbmZF3nQEWBPFSr>`_ + +Slide decks: + +- `bash scripts and keyboard shortcuts <https://docs.google.com/presentation/d/1T8z99Hy0rI-_W3wJIZsG-edKRiicxgl-QeBoJuM90qQ/edit?usp=sharing>`_ +- `sccache-dist <https://docs.google.com/presentation/d/1_mN5rgV2LrzRKEOn06j4uaryYC9zRQJOdomgeBrJ8RA/edit#slide=id.g832b271044_1_1173>`_ +- `How to animate things smoothly <https://docs.google.com/presentation/d/11csNTR1GnVs2BdjN1alJcTbjXMhKyhpv9GIx2jNMJ6U/edit#slide=id.p>`_ +- `How I learned to love the M-C <https://docs.google.com/presentation/d/13O06nyDWqfbZyLeGRdLztd234zJ2SQcJu7LkBFDLnYg/edit?usp=sharing>`_ + +Links for "Editor, Lint, & More" talk: + +- :ref:`Linting` +- :ref:`lint-vcs-hook` +- https://eslint.org/docs/user-guide/integrations + +Links for "Stupid Mac Tricks" talk: + +- https://www.obdev.at/products/launchbar/index.html +- Moom: https://itunes.apple.com/app/id496437906?mt=12&ls=1 +- https://kapeli.com/dash +- https://git-fork.com/ +- https://www.usboverdrive.com/ +- https://manytricks.com/download/butler/ +- https://bjango.com/mac/istatmenus/ +- https://iterm2.com/ + +Other links: + +- https://twitter.com/asutherland/status/1274942012316319744 +- https://scootersoftware.com/ +- https://pasteapp.io/ +- https://github.com/acreskeyMoz/browsertime_scripts +- https://chuttenblog.wordpress.com/2021/01/18/doubling-the-speed-of-windows-firefox-builds-using-sccache-dist/ diff --git a/docs/contributing/filing-good-bugs.rst b/docs/contributing/filing-good-bugs.rst new file mode 100644 index 0000000000..d95e1ac5ea --- /dev/null +++ b/docs/contributing/filing-good-bugs.rst @@ -0,0 +1,56 @@ +Filing good bugs +================ + +Getting started working on a bug can be hard, specially if you lack +context. + +This guide is meant to provide a list of steps to provide the necessary +information to open an actionable bug. + +- **Use a descriptive title**. Avoid jargon and abbreviations where + possible, they make it hard for other people to find existing bugs, + and to understand them. +- **Explain the problem in depth** and provide the steps to reproduce. Be + as specific as possible, and include things such as operating system + and version if reporting a bug. +- If you can, **list files and lines of code** that may need to be + modified. Ideally provide a patch for getting started. + Do not hesitate to add permalinks to `Searchfox <https://searchfox.org/mozilla-central/source/>`_ + pointing to the file and lines. +- If applicable, **provide a test case** or document that can be used to + test the bug is solved. For example, if the bug title was “HTML + inspector fails when inspecting a page with one million of nodes”, + you would provide an HTML document with one million of nodes, and we + could use it to test the implementation, and make sure you’re looking + at the same thing we’re looking at. You could use services like + jsfiddle, codepen or jsbin to share your test cases. Other people use + GitHub, or their own web server. + +Good first bugs +--------------- + +If you're looking to open a bug as a "good first bug" for new contributors, please adhere to the following guidelines: + +1. **Complexity** + Ensure the bug is not overly complex. New contributors are already navigating the learning curve of Firefox's codebase and workflows. An overly complicated bug can be overwhelming. + +2. **Tagging** + Use the Bugzilla keyword ``good-first-bug`` to mark it appropriately. + +3. **Language Specification** + In the whiteboard section, specify the primary language of the bug using the ``[lang=XX]`` format. For instance, ``[lang=C++]`` for C++ or ``[lang=py]`` for Python. + +4. **Mentorship** + Commit to guiding the new contributor by adding yourself as a mentor for the bug. Please keep the mentor field up to date, if you no longer have time set it to empty or find another available mentor. + + +5. **Documentation** + Provide links to essential documentation that can assist the contributor. For example, you might include: + + - :ref:`How To Contribute Code To Firefox` + - :ref:`Firefox Contributors' Quick Reference` + - :ref:`Working with stack of patches Quick Reference` + +By following these guidelines, you'll be setting up new contributors for success and fostering a welcoming environment for them. + +These good first bugs can be browsed on https://codetribute.mozilla.org/ diff --git a/docs/contributing/how_to_submit_a_patch.rst b/docs/contributing/how_to_submit_a_patch.rst new file mode 100644 index 0000000000..d8ef62f742 --- /dev/null +++ b/docs/contributing/how_to_submit_a_patch.rst @@ -0,0 +1,244 @@ +How to submit a patch +===================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Submitting a patch, getting it reviewed, and committed to the Firefox +source tree involves several steps. This article explains how. + +.. note:: + + We are also providing a :ref:`Firefox Contributors Quick Reference <Firefox Contributors' Quick Reference>` for contributors. + +The process of submission is illustrated by the following diagram, and +each step is detailed below: + +.. mermaid:: + + graph TD; + Preparation --> c[Working on a patch]; + c[Working on a patch] --> Testing; + Testing --> c[Working on a patch]; + Testing --> e[Submit the patch]; + e[Submit the patch] --> d[Getting Reviews] + d[Getting Reviews] -- Addressing Review comment --> c[Working on a patch]; + d[Getting Reviews] --> h[Push the change]; + + + + +Preparation +----------- + +Every change to the code is tracked by a bug report +in `bugzilla.mozilla.org <https://bugzilla.mozilla.org/>`__. Without a +bug, code will not be reviewed, and without review, code will not be +accepted. To avoid duplication, `search for an existing +bug <https://bugzilla.mozilla.org/query.cgi?format=specific>`__ about +your change, and only if none exists, file a new one. Most communication +about code changes take place in the associated code +review, so be sure the bug describes the exact problem being solved. + +Please verify the bug is for the correct product and component. For more +information, ask questions on the newsgroups, or on the #developers room +on `chat.mozilla.org <https://chat.mozilla.org>`__. + +The person working on a bug should be the 'assignee' of that bug in +Bugzilla. If somebody else is currently the assignee of a bug, email +this person to coordinate changes. If the bug is unassigned, leave a +message in the bug's comments, stating that you intend working on it, +and suggest that someone with bug-editing privileges assign it to you. + +Some teams wait for new contributors to attach their first patch before +assigning a bug. This makes it available for other contributors, in case +the new contributor is unable to level up to patch creation. By +expressing interest in a bug comment, someone from that team should +guide you through their process. + + +Module ownership +---------------- + +All code is supervised by a `module +owner <https://www.mozilla.org/en-US/about/governance/policies/module-ownership/>`__. +This person will be responsible for reviewing and accepting the change. +Before writing your code, determine the module owner, verifying your +proposed change is considered acceptable. They may want to look over any +new user interface (UI review), functions (API review), or testcases for +the proposed change. + +If module ownership is not clear, ask on the newsgroups or `on +Matrix <https://chat.mozilla.org>`__. The revision log for the relevant +file might also be helpful. For example, see the change log for +``browser/base/content/browser.js``, by clicking the "Hg Log" +link at the top of `Searchfox <https://searchfox.org/mozilla-central/source/>`__, or +by running ``hg log browser/base/content/browser.js``. The corresponding +checkin message will contain something like "r=nickname", identifying +active code submissions, and potential code reviewers. + + +Working on a patch +------------------ + +Changes to the Firefox source code are presented in the form of a patch. +A patch is a commit to version control. Firefox and related code is +stored in our `Mercurial +server <https://hg.mozilla.org/mozilla-central>`__. We have extensive +documentation on using Mercurial in our guide, :ref:`Mercurial Overview`. + +Each patch should represent a single complete change, separating +distinct changes into multiple individual patches. If your change +results in a large, complex patch, seek if it can be broken into +`smaller, easy to understand patches representing complete +steps <https://secure.phabricator.com/book/phabflavor/article/writing_reviewable_code/#many-small-commits>`__, +applied on top of each other. This makes it easier to review your +changes, `leading to quicker +reviews, <https://groups.google.com/group/mozilla.dev.planning/msg/2f99460f57f776ef?hl=en>`__ +and improved confidence in this review outcome. + +Also ensure that your commit message is formatted appropriately. A +simple commit message should look like this: + +:: + + Bug 123456 - Change this thing to work better by doing something. r=reviewers + +The ``r=reviewers`` part is optional; if you are using Phabricator, +Lando will add it automatically based on who actually granted review, +and in any case the person who does the final check-in of the patch will +make sure it's added. + +The text of the message should be what you did to fix the bug, not a +description of what the bug was. If it is not obvious why this change is +appropriate, then `explain why in the commit +message <https://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview/commits.html#write-detailed-commit-messages>`__. +If this does not fit on one line, then leave a blank line and add +further lines for more detail and/or reasoning. + +You can edit the message of the current commit at any time using +``hg commit --amend`` or ``hg histedit``. + +Also look at our :ref:`Reviewer Checklist` for a list +of best practices for patch content that reviewers will check for or +require. + + +Testing +------- + +All changes must be tested. In most cases, an `automated +test <https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing>`__ is required for every +change to the code. + +While we desire to have automated tests for all code, we also have a +linter tool which runs static analysis on our JavaScript, for best +practices and common mistakes. See :ref:`ESLint` for more information. + +Ensure that your change has not caused regressions, by running the +automated test suite locally, or using the `Mozilla try +server <https://wiki.mozilla.org/Build:TryServer>`__. Module owners, or +developers `on Matrix <https://chat.mozilla.org>`__ may be willing to +submit jobs for those currently without try server privileges. + + +Submit the patch +---------------- + +.. note:: + + Make sure you rebase your patch on top of the latest build before you + submit to prevent any merge conflicts. + +Mozilla uses Phabricator for code review. See the `Mozilla Phabricator +User +Guide <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`__ +for instructions. + +Don't be shy in posting partial patches, demonstrating potential +approaches, and asking for preliminary feedback. It is easier for others +to comment, and offer suggestions, when a question is accompanied by +some code. + + +Getting reviews for my patch +---------------------------- + +See the dedicated page :ref:`Getting reviews` + + +Addressing review comments +-------------------------- + +It is unusual for patches to be perfect the first time around. The +reviewer may use the ‘Request Changes’ +`action <http://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#reviewing-patches>`__ +and list problems that must be addressed before the patch can be +accepted. Please remember that requesting revisions is not meant to +discourage participation, but rather to encourage the best possible +resolution of a bug. Carefully work through the changes that the +reviewer recommends, attach a new patch, and request review again. + +Sometimes a reviewer will grant conditional review with the ‘Accept +Revision’ action but will also indicate minor necessary changes, such as +spelling, or indentation fixes. All recommended corrections should be +made, but a re-review is unnecessary. Make the changes and submit a new +patch. If there is any confusion about the revisions, another review +should be requested. + +Sometimes, after a patch is reviewed, but before it can be committed, +someone else makes a conflicting change. If the merge is simple, and +non-invasive, post an updated version of the patch. For all non-trivial +changes, another review is necessary. + +If at any point the review process stalls for more than two weeks, see +the previous 'Getting attention' section. + +In many open source projects, developers will accept patches in an +unfinished state, finish them, and apply the completed code. In +Mozilla's culture, **the reviewer will only review and comment on a +patch**. If a submitter declines to make the revisions, the patch will +sit idle, until someone chooses to take it on. + + +Pushing the change +------------------ + +A patch can be pushed (aka. 'landed') after it has been properly +reviewed. + +.. note:: + + Be sure to build the application with the patch applied. This + ensures it runs as expected, passing automated tests, and/or runs + through the `try + server <https://wiki.mozilla.org/Build:TryServerAsBranch>`__. In the + bug, please also mention you have completed this step. + + Submitting untested patches wastes the committer's time, and may burn + the release tree. Please save everyone's time and effort by + completing all necessary verifications. + + +Ask the reviewer to land the patch for you. +For more details, see :ref:`push_a_change` + +`Lando <https://moz-conduit.readthedocs.io/en/latest/lando-user.html>`__ is used +to automatically land your code. + + +Regressions +----------- + +It is possible your code causes functional or performance regressions. +There is a tight +`policy <https://www.mozilla.org/about/governance/policies/regressions/>`__ on +performance regressions, in particular. This means your code may be +dropped, leaving you to fix and resubmit it. Regressions, ultimately +mean the tests you ran before checking in are not comprehensive enough. +A resubmitted patch, or a patch to fix the regression, should be +accompanied by appropriate tests. + +After authoring a few patches, consider `getting commit access to +Mozilla source code <https://www.mozilla.org/about/governance/policies/commit/>`__. diff --git a/docs/contributing/img/auto_completion.gif b/docs/contributing/img/auto_completion.gif Binary files differnew file mode 100644 index 0000000000..4d545c29fd --- /dev/null +++ b/docs/contributing/img/auto_completion.gif diff --git a/docs/contributing/img/connection_done.png b/docs/contributing/img/connection_done.png Binary files differnew file mode 100644 index 0000000000..fc87571e4e --- /dev/null +++ b/docs/contributing/img/connection_done.png diff --git a/docs/contributing/img/diagnostic_error.gif b/docs/contributing/img/diagnostic_error.gif Binary files differnew file mode 100644 index 0000000000..25d50a7d7c --- /dev/null +++ b/docs/contributing/img/diagnostic_error.gif diff --git a/docs/contributing/img/example-stack.png b/docs/contributing/img/example-stack.png Binary files differnew file mode 100644 index 0000000000..c12265ae5f --- /dev/null +++ b/docs/contributing/img/example-stack.png diff --git a/docs/contributing/img/find_references.gif b/docs/contributing/img/find_references.gif Binary files differnew file mode 100644 index 0000000000..e986894566 --- /dev/null +++ b/docs/contributing/img/find_references.gif diff --git a/docs/contributing/img/format_selection.gif b/docs/contributing/img/format_selection.gif Binary files differnew file mode 100644 index 0000000000..28c88b3fa3 --- /dev/null +++ b/docs/contributing/img/format_selection.gif diff --git a/docs/contributing/img/goto_definition.gif b/docs/contributing/img/goto_definition.gif Binary files differnew file mode 100644 index 0000000000..9b3dcbf90f --- /dev/null +++ b/docs/contributing/img/goto_definition.gif diff --git a/docs/contributing/img/remote_explorer.png b/docs/contributing/img/remote_explorer.png Binary files differnew file mode 100644 index 0000000000..3f7a97bdc0 --- /dev/null +++ b/docs/contributing/img/remote_explorer.png diff --git a/docs/contributing/img/remote_explorer_add.png b/docs/contributing/img/remote_explorer_add.png Binary files differnew file mode 100644 index 0000000000..9faf9750fb --- /dev/null +++ b/docs/contributing/img/remote_explorer_add.png diff --git a/docs/contributing/img/remote_explorer_add_wind.png b/docs/contributing/img/remote_explorer_add_wind.png Binary files differnew file mode 100644 index 0000000000..6aa9f8d54d --- /dev/null +++ b/docs/contributing/img/remote_explorer_add_wind.png diff --git a/docs/contributing/img/rename_symbol.gif b/docs/contributing/img/rename_symbol.gif Binary files differnew file mode 100644 index 0000000000..d41fcd7ed6 --- /dev/null +++ b/docs/contributing/img/rename_symbol.gif diff --git a/docs/contributing/img/type_hierarchy.gif b/docs/contributing/img/type_hierarchy.gif Binary files differnew file mode 100644 index 0000000000..139bdf88cb --- /dev/null +++ b/docs/contributing/img/type_hierarchy.gif diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst new file mode 100644 index 0000000000..545e161f8e --- /dev/null +++ b/docs/contributing/index.rst @@ -0,0 +1,53 @@ +Working on Firefox +================== + +Welcome to the Firefox codebase. This is the home of the Firefox +development process and source code documentation. + +.. toctree:: + :caption: Making Changes To Firefox + :maxdepth: 1 + + contribution_quickref + stack_quickref + pocket-guide-shipping-firefox + editor + reviews + levelling-up + how_to_submit_a_patch + + +.. toctree:: + :caption: Code reviews + :maxdepth: 1 + + Code_Review_FAQ + reviewer_checklist + + +.. toctree:: + :caption: The Mercurial Version Control System + :maxdepth: 1 + :glob: + + vcs/* + + +.. toctree:: + :caption: Debugging + :maxdepth: 1 + :glob: + + debugging/* + + +.. toctree:: + :caption: Additional Information + :maxdepth: 1 + + directory_structure + build/artifact_builds + build/building_mobile_firefox + build/supported + engineering_show_and_tell + filing-good-bugs diff --git a/docs/contributing/levelling-up.rst b/docs/contributing/levelling-up.rst new file mode 100644 index 0000000000..02d584b051 --- /dev/null +++ b/docs/contributing/levelling-up.rst @@ -0,0 +1,44 @@ +Levelling up +============ + +Once you’ve fixed a few bugs, it’s quite likely that we’ll ask you to +complete one or more of the following steps. They will grant you access +to useful Mozilla infrastructure for testing or landing code +automatically, so you can be more autonomous and able to contribute with +less supervision. + +If you’re curious, the `Mozilla Commit Access +Policy <https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/>`__ +page explains what does each level mean. + +Request commit access level 1 +----------------------------- + +Once this is granted, you can use the +`Try server <https://wiki.mozilla.org/ReleaseEngineering/TryServer>`__ for +running automated tests of your code. + +To request this, follow the instructions on the `Becoming A Mozilla Committer <https://www.mozilla.org/en-US/about/governance/policies/commit/>`__ page +to file a bug with the title “Commit Access (Level 1) for ${your name}”, +and continue with steps such as adding the public SSH key, etc (it’s not +enough with just filing the bug!). + +The person that asked you to request this can also vouch for you. + +Request commit access level 3 +----------------------------- + +After you landed a few bugs, that are more advanced than the +``good-first-bugs`` and you feel confident about your contributions as +well as the process, you can request level 3 access. Once this is +granted, you will get access to Lando, the autoland feature. + +To request this, follow the instructions on the `Becoming A Mozilla Committer <https://www.mozilla.org/en-US/about/governance/policies/commit/>`__ page +to file a bug with the title “Commit Access (Level 3) for ${your name}”. + +The person that asked you to request this can also vouch for you, and at +this point you might already know a few other people who will vouch for +you. You need two vouchers. + +If you reach this level, well done! That’s amazing, and we’re glad to +have you on board 😄 diff --git a/docs/contributing/pocket-guide-shipping-firefox.rst b/docs/contributing/pocket-guide-shipping-firefox.rst new file mode 100644 index 0000000000..f76fee11ae --- /dev/null +++ b/docs/contributing/pocket-guide-shipping-firefox.rst @@ -0,0 +1,524 @@ +Pocket Guide: Shipping Firefox +============================== + +*Estimated read time:* 15min + + +Introduction +------------ + +The purpose of this document is to provide a high level understanding of +how Mozilla ships Firefox. With the intention of helping new Mozillians +(and those who would like a refresher) understand the basics of our +release process, tools, common terms, and mechanisms employed in +shipping Firefox to our users. Often this document will introduce a +concept, explain how it fits into the process, and then provide a link +to learn more if interested. + +Repositories & Channels +----------------------- + +Shipping Firefox follows a software release :ref:`train model <train model>` +along 3 primary code :ref:`repositories <repositories>`; mozilla-central +(aka “m-c”), mozilla-beta, and mozilla-release. Each of these repositories are +updated within a defined cadence and built into one of our Firefox +products which are released through what is commonly referred to as +:ref:`Channels <channels>`: Firefox Nightly, Firefox Beta, and Firefox Release. + +`Firefox Nightly <https://whattrainisitnow.com/release/?version=nightly>`__ offers access to the latest cutting edge features +still under active development. Released every 12 hours with all the +changes that have :ref:`landed <landing>` on mozilla-central for Desktop and on +`main in firefox-android <https://github.com/mozilla-mobile/firefox-android/tree/main>`__ for Android. + +Every `4 weeks <https://whattrainisitnow.com/calendar/>`__, we +:ref:`merge <merge>` the code from mozilla-central to our +mozilla-beta branch. +For Android, we branch from main on firefox-android to a release branch. +New code or features can be added to mozilla-beta +outside of this 4 week cadence but will be required to land in +mozilla-central and then be :ref:`uplifted <uplift>` into +mozilla-beta. +Similarly for Android, uplifts are required to land in main on firefox-android before +backporting to the firefox-android release branch. + +`Firefox Beta <https://whattrainisitnow.com/release/?version=beta>`__ is for developers and early adopters who want to see +and test what’s coming next in Firefox. We release a new Desktop/Android Beta version +three times a week. + +.. note:: + + The first and second beta builds of a new cycle are shipped to a + subset of our Beta population. The full Beta population gets updated + starting with Beta 3 only.* + +Each Beta cycle lasts a total of 4 weeks where a final build is +validated by our QA and tagged for release into the mozilla-release +branch for Desktop. On Android we release from the same release branch +used during the Beta cycle. + +.. note:: + + **Firefox Developer Edition** *is a separate product based on + the mozilla-beta repo and is specifically tailored for Web Developers.* + +`Firefox Release <https://whattrainisitnow.com/release/?version=release>`__ is released every 4 weeks and is the end result +of our Beta cycle. This is our primary product shipping to hundreds of +millions of users. While a release is live, interim updates (dot releases) +are used to ship important bug fixes to users prior to the next major release. +These can happen on an as-needed basis when there is an important-enough +:ref:`driver <dot release drivers>` to do so (such as a critical bug severely +impairing the usability of the product for some users). In order to provide +better predictability, there is also a planned dot release scheduled for two +weeks after the initial go-live for less-critical fixes and other +:ref:`ride-along fixes <ride alongs>` deemed low-risk enough to include. + +.. note:: + `Firefox ESR (Extended Support Release) <https://whattrainisitnow.com/release/?version=esr>`__ *is a separate + product intended for Enterprise use. Major updates are rolled out once + per year to maintain stability and predictability. ESR also + contains a number of policy options not available in the standard + Firefox Release. Minor updates are shipped in sync with the Firefox + Release schedule for security and select quality fixes only.* + +Further Reading/Useful links: + +- `Firefox + Trains <https://whattrainisitnow.com/>`__ +- `Release + Calendar <https://whattrainisitnow.com/calendar/>`__ +- `Firefox Release + Process <https://wiki.mozilla.org/Release_Management/Release_Process>`__ +- `Firefox Delivery + dashboard <https://mozilla.github.io/delivery-dashboard/>`__ + +Landing Code and Shipping Features +---------------------------------- + +Mozillians (those employed by MoCo and the broader community) land lots +of code in the Mozilla repositories: fixes, enhancements, compatibility, +new features, etc. and is managed by :ref:`Mercurial <Mercurial Overview>` (aka +hg). All new code is tracked in :ref:`Bugzilla <bugzilla>`, reviewed +in :ref:`Phabricator <Phabricator>`, and then checked into the +mozilla-central repository using :ref:`Lando <Lando>`. + +.. note:: + + Some teams use :ref:`GitHub <github>` during development + but will still be required to use Phabricator (tracked in Bugzilla) to + check their code into the mozilla-central hg repository. + +The standard process for code to be delivered to our users is by ‘riding +the trains’, meaning that it’s landed in mozilla-central where it waits +for the next Beta cycle to begin. After merging to Beta the code will +stabilize over a 4 week period (along with everything else that merged +from mozilla-central). At the end of the beta cycle a release candidate +(:ref:`RC <rc>`) build will be generated, tested thoroughly, and +eventually become the next version of Firefox. + +Further Reading/Useful links: + +- `Phabricator and why we use it <https://wiki.mozilla.org/Phabricator>`__ +- `Firefox Release Notes Process <https://wiki.mozilla.org/Release_Management/Release_Notes>`__ +- `Firefox Release Notes Nomination <https://wiki.mozilla.org/Release_Management/Release_Notes_Nomination>`__ + +An exception to this process... +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Not all code can simply wait for the normal train model to be included +in a Firefox build. There are a variety of reasons for this; critical +fixes, security concerns, stabilizing a feature that’s already in Beta, +shipping high priority features faster, and so on. + +In these situations an uplift can be requested to take a recent landing +in mozilla-central and merge specific bits to another repository outside +the standard train model. After the request is made within Bugzilla, +:ref:`Release Management <release management>` will assess the potential risk +and will make a decision on whether it’s accepted. + +Further Reading/Useful links: + +- `Patch uplifting + rules <https://wiki.mozilla.org/Release_Management/Uplift_rules>`__ +- `Requesting an + uplift <https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift>`__ + +Ensuring build stability +~~~~~~~~~~~~~~~~~~~~~~~~ + +Throughout the process of landing code in mozilla-central to riding the +trains to Firefox Release, there are many milestones and quality +checkpoints from a variety of teams. This process is designed to ensure +a quality and compelling product will be consistently delivered to our +users with each new version. See below for a distilled list of those +milestones. + +=========================================== ================ ================= =============================================================================== +Milestone Week Day of Week +------------------------------------------- ---------------- ----------------- ------------------------------------------------------------------------------- +Merge Day Nightly W1 Monday Day 1 of the new Nightly Cycle +PI Request deadline Nightly W1 Friday Manual QA request deadline for high risk features +Feature technical documentation due Nightly W2 Friday Deadline for features requiring manual QA +Beta release notes draft Nightly W4 Wednesday +Nightly features Go/No-Go decisions Nightly W4 Wednesday +Feature Complete Milestone Nightly W4 Thursday Last day to land risky patches and/or enable new features +Nightly soft code freeze start Nightly W4 Thursday Stabilization period in preparation to merge to Beta +QA Test Plan approval due Nightly W4 Friday Last day to provide QA with feature Test Plan sign-offs +String freeze Nightly W4 Friday Modification or deletion of strings exposed to the end-users is not allowed +QA pre-merge regression testing completed Nightly W4 Friday +Merge Day Beta W1 Monday Day 1 of the new Beta cycle +Pre-release sign off Beta W3 Friday Final round of QA testing prior to Release +Firefox RC week Beta W4 Monday Validating Release Candidate builds in preparation for the next Firefox Release +Release Notes ready Beta W4 Tuesday +What’s new page ready Beta W4 Wednesday +Firefox go-live @ 6am PT Release W1 Tuesday Day 1 of the new Firefox Release to 25% of Release users +Firefox Release bump to 100% Release W1 Thursday Increase deployment of new Firefox Release to 100% of Release users +Scheduled dot release approval requests due Release W2 Friday All requests required by EOD +Scheduled dot release go-live Release W3 Tuesday By default, ships when ready. Specific time available upon request. +=========================================== ================ ================= =============================================================================== + + +The Release Management team (aka “Relman”) monitors and enforces this +process to protect the stability of Firefox. Each member of Relman +rotates through end-to-end ownership of a given :ref:`release +cycle <release cycle>`. The Relman owner of a cycle will focus on the +overall release, blocker bugs, risks, backout rates, stability/crash +reports, etc. Go here for a complete overview of the `Relman Release +Process +Checklist <https://wiki.mozilla.org/Release_Management/Release_Process_Checklist_Documentation>`__. + +.. note:: + + While Relman will continually monitor the overall health of each + Release it is the responsibility of the engineering organization to + ensure the code they are landing is of high quality and the potential + risks are understood. Every Release has an assigned :ref:`Regression + Engineering Owner <reo>` (REO) to ensure a decision is made + about each regression reported in the release.* + +Further Reading/Useful links: + +- `Release Tracking + Rules <https://wiki.mozilla.org/Release_Management/Tracking_rules>`__ +- `Release + Owners <https://wiki.mozilla.org/Release_Management/Release_owners>`__ +- `Regression Engineering + Owners <https://wiki.mozilla.org/Platform#Regression_Engineering_Owner_.28REO.29>`__ +- `Commonly used Bugzilla queries for all + Channels <https://trainqueries.herokuapp.com/>`__ + +Enabling/Disabling code (Prefs) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Within Firefox we allow the ability to Enable/Disable bits of code or +entire features using `Preferences <preferences>`. There are many +reasons why this is useful. Here are some examples: + +- Continual development over multiple release cycles without exposing + partially completed features to our users +- Provide the ability to quickly disable a feature if there is a + problem found during the release process +- Control features which are experimental or not ready to be shown to a + specific channel population (e.g. enabled for Beta but disabled for + Release) +- A/B testing via :ref:`telemetry <telemetry>` experiments + +.. note:: + + :ref:`Normandy <normandy>` Pref Rollout is a feature that + allows Mozilla to change the state of a preference for a targeted set of + users, without deploying an update to Firefox. This is especially useful + when conducting experiments or a gradual rollout of high risk features + to our Release population. + +Further Reading/Useful links: + +- `Brief guide to Mozilla + preferences <https://developer.mozilla.org/en-US/docs/Mozilla/Preferences/A_brief_guide_to_Mozilla_preferences>`__ +- `Normandy Pref + rollout <https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout>`__ + +Release & Feature QA +~~~~~~~~~~~~~~~~~~~~ + +Release QA is performed regularly and throughout the Release Cycle. +Organized in two-week sprints its primary goals are: + +- Qualifying builds for release +- Feature testing +- Product Integrity requests +- Bug work +- Community engagement + +Features that can have significant impact and/or pose risk to the code +base should be nominated for QA support by the :ref:`feature +owner <feature owner>` in its intended release. This process is kicked +off by filing a :ref:`Product Integrity <product integrity>` team request +:ref:`PI request <pi request>`. These are due by the end of week 2 +of the Nightly cycle. + +.. note:: + + Manual QA testing is only required for features as they go + through the Beta cycle. Nightly Feature testing is always optional. + +Further Reading/Useful links: + +- `QA Feature + Testing <https://wiki.mozilla.org/QA/Feature_Testing_v2>`__ +- `Release QA + overview <https://docs.google.com/document/d/1ic_3TO9-kNmZr11h1ZpyQbSlgiXzVewr3kSAP5ML4mQ/edit#heading=h.pvvuwlkkvtc4>`__ +- `PI Request template and + overview <https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=PI&title=PI+Request>`__ + +Experiments +~~~~~~~~~~~ + +As we deliver new features to our users we continually ask ourselves +about the potential impacts, both positive and negative. In many new +features we will run an experiment to gather data around these impacts. +A simple definition of an experiment is a way to measure how a change to +our product affects how people use it. + +An experiment has three parts: + +1. A new feature that can be selectively enabled +2. A group of users to test the new feature +3. Telemetry to measure how people interact with the new feature + +Experiments are managed by an in-house tool called +`Experimenter <https://experimenter.services.mozilla.com/>`__. + +Further Reading/Useful links: + +- `More about experiments and + Experimenter <https://github.com/mozilla/experimenter>`__ +- `Requesting a new + Experiment <https://experimenter.services.mozilla.com/experiments/new/>`__ + (Follow the ‘help’ links to learn more) +- `Telemetry <https://wiki.mozilla.org/Telemetry>`__ + +Definitions +----------- + +.. _approval flag: + +**Approval Flag** - A flag that represents a security approval or uplift +request on a patch. + +.. _bugzilla: + +**Bugzilla** - Web-based general purpose bug tracking system and testing +tool. + +.. _channel: + +**Channel** - Development channels producing concurrent releases of +Firefox for Windows, Mac, Linux, and Android. + +.. _chemspill: + +**Chemspill** - Short for Chemical Spill. A chemspill is a rapid +security-driven or critical stsbility dot release of our product. + +.. _channel meeting: + +**Channel Meeting** - A twice weekly time to check in on the status +of the active releases with the release team. + +.. _dot release drivers: + +**Dot Release Drivers** - Issues/Fixes that are significant enough to +warrant a minor dot release to the Firefox Release Channel. Usually to +fix a stability (top-crash) or Security (Chemspill) issue. + +.. _early beta: + +**Early Beta** - Beta releases with the features gated by EARLY_BETA_OR_EARLIER +enabled. The first 2 weeks of Beta releases during the cycle are early beta releases. + +.. _feature owner: + +**Feature Owner** - The person who is ultimately responsible for +developing a high quality feature. This is typically an Engineering +Manager or Product Manager. + +.. _fenix: + +**Fenix** - Also known as Firefox Preview is an all-new browser for +Android based on GeckoView and Android Components + +.. _github: + +**Github** - Web-based version control and collaboration platform for +software developers + +.. _gtb: + +**GTB** - Acronym for Go to build. Mostly used in the release schedule +communication ("Go to build on March 18"), this means that we initiate the +building of a specific release. + +.. _landing: + +**Landing** - A general term used for when code is merged into a +particular source code repository + +.. _lando: + +**Lando** - Automated code lander for Mozilla. It is integrated with +our `Phabricator instance <https://phabricator.services.mozilla.com>`__ +and can be used to land revisions to various repositories. + +.. _mercurial: + +**Mercurial** - A source-code management tool (just like git) +which allows users to keep track of changes to the source code +locally and share their changes with others. It is also called hg. + +.. _merge: + +**Merge** - General term used to describe the process of integrating and +reconciling file changes within the mozilla repositories + +.. _nightly soft code freeze: + +**Nightly Soft Code Freeze** - Last week of the nightly cycle on mozilla-central +just before the merge to beta during which landing risky or experimental code +in the repository is discouraged. + +.. _normandy: + +**Normandy** - Normandy is a collection of servers, workflows, and +Firefox components that enables Mozilla to remotely control Firefox +clients in the wild based on precise criteria + +.. _nucleus: + +**Nucleus** - Name of the internal application used by release managers +to prepare and publish release notes. The data in this application is +fetched by mozilla.org. + +.. _orange_factor: + +**Orange** - Also called flaky or intermittent tests. Describes a state +when a test or a testsuite can intermittently fail. + +.. _phabricator: + +**Phabricator** - Mozilla’s instance of the web-based software +development collaboration tool suite. Read more about `Phabricator as a +product <https://phacility.com/phabricator/>`__. + +.. _pi request: + +**PI Request** - Short for Product Integrity Request is a form +submission request that’s used to engage the PI team for a variety of +services. Most commonly used to request Feature QA it can also be used +for Security, Fuzzing, Performance, and many other services. + +.. _preferences: + +**Preferences** - A preference is any value or defined behavior that can +be set (e.g. enabled or disabled). Preference changes via user interface +usually take effect immediately. The values are saved to the user’s +Firefox profile on disk (in prefs.js). + +.. _rc: + +**Release Candidate** - Beta version with potential to be a final +product, which is ready to release unless significant bugs emerge. + +.. _rc week: + +**RC Week** - The week prior to release go-live is known as RC week. +During this week an RC is produced and tested. + +.. _release cycle: + +**Release Cycle** - The sum of stages of development and maturity for +the Firefox Release Product. + +.. _reo: + +**Regression Engineering Owner** - A partner for release management +assigned to each release. They both keep a mental state of how we are +doing and ensure a decision is made about each regression reported in +the release. AKA *REO*. + +.. _release engineering: + +**Release engineering** - Team primarily responsible for maintaining +the build pipeline, the signature mechanisms, the update servers, etc. aka *releng* + +.. _release management: + +**Release Management** - Team primarily responsible for the process of +managing, planning, scheduling and controlling a software build through +different stages and environments. aka *relman*. + +.. _relnotes: + +**Relnotes** - Short for release notes. Firefox Nightly, Beta, and Release each ship +with release notes. + +.. _Repository: + +**Repository** - a collection of stored data from existing databases +merged into one so that it may be shared, analyzed or updated throughout +an organization. + +.. _ride alongs: + +**Ride Alongs** - Bug fixes that are impacting release users but not +considered severe enough to ship without an identified dot release +driver. + +.. _rollout: + +**Rollout** - Shipping a release to a percentage of the release population. + +.. _status flags: + +**Status Flags** - A flag that represents the status of the bug with +respect to a Firefox release. + +.. _string freeze: + +**String Freeze** - Period during which the introduction, modification, or +deletion of strings exposed to the end-users is not allowed so as to allow our +localizers to translate our product. + +.. _taskcluster: + +**taskcluster** - Our execution framework to build, run tests on multiple +operating system, hardware and cloud providers. + +.. _telemetry: + +**Telemetry** - Firefox measures and collects non-personal information, +such as performance, hardware, usage and customizations. This +information is used by Mozilla to improve Firefox. + +.. _train model: + +**Train model** - a form of software release schedule in which a number +of distinct series of versioned software releases are released as a +number of different "trains" on a regular schedule. + +.. _tracking flags: + +**Tracking Flags** - A Bugzilla flag that shows whether a bug is being investigated +for possible resolution in a Firefox release. Bugs marked tracking-Firefox XX are +bugs that must be resolved one way or another before a particular release ship. + +.. _throttle unthrottle: + +**Throttle/Unthrottle a rollout** - Throttle is restricting a release rollout to 0% +of the release population, users can still choose to update but are not updated +automatically. Unthrottle is removing the release rollout restriction. + +.. _uplift: + +**Uplift** - the action of taking parts from a newer version of a +software system (mozilla-central or mozilla-beta) and porting them to an +older version of the same software (mozilla-beta, mozilla-release or ESR) diff --git a/docs/contributing/reviewer_checklist.rst b/docs/contributing/reviewer_checklist.rst new file mode 100644 index 0000000000..cfe772dba9 --- /dev/null +++ b/docs/contributing/reviewer_checklist.rst @@ -0,0 +1,181 @@ +Reviewer Checklist +================== + + Submitting patches to Mozilla source code needn't be complex. This + article provides a list of best practices for your patch content that + reviewers will check for or require. Following these best practices + will lead to a smoother, more rapid process of review and acceptance. + + +Good web citizenship +-------------------- + +- Make sure new web-exposed APIs actually make sense and are either + standards track or preffed off by default. +- In C++, wrapper-cache as needed. If your object can be gotten from + somewhere without creating it in the process, it needs to be + wrapper-cached. + + +Correctness +----------- + +- The bug being fixed is a valid bug and should be fixed. +- The patch fixes the issue. +- The patch is not unnecessarily complicated. +- The patch does not add duplicates of existing code ('almost + duplicates' could mean a refactor is needed). Commonly this results + in "part 0" of a bug, which is "tidy things up to make the fix easier + to write and review". +- If QA needs to verify the fix, you should provide steps to reproduce + (STR). + + +Quality +------- + +- If you can unit-test it, you should unit-test it. +- If it's JS, try to design and build so that xpcshell can exercise + most functionality. It's quicker. +- Make sure the patch doesn't create any unused code (e.g., remove + strings when removing a feature) +- All caught exceptions should be logged at the appropriate level, + bearing in mind personally identifiable information, but also + considering the expense of computing and recording log output. + [Fennec: Checking for log levels is expensive unless you're using + Logger.] + + +Style +----- + +- Follow the `style + guide <https://firefox-source-docs.mozilla.org/code-quality/coding-style/index.html>`__ + for the language and module in question. +- Follow local style for the surrounding code, even if that local style + isn't formally documented. +- New files have license declarations and modelines. +- New JS files should use strict mode. +- Trailing whitespace (git diff and splinter view both highlight this, + as does hg with the color extension enabled). Whitespace can be fixed + easily in Mercurial using the `CheckFiles + extension <https://www.mercurial-scm.org/wiki/CheckFilesExtension>`__. + In git, you can use git rebase --whitespace=fix. + + +Security issues +--------------- + +- There should be no writing to arbitrary files outside the profile + folder. +- Be careful when reading user input, network input, or files on disk. + Assume that inputs will be too big, too short, empty, malformed, or + malicious. +- Tag for sec review if unsure. +- If you're writing code that uses JSAPI, chances are you got it wrong. + Try hard to avoid doing that. + + +Privacy issues +-------------- + +- There should be no logging of URLs or content from which URLs may be + inferred. +- [Fennec: Android Services has Logger.pii() for this purpose (e.g., + logging profile dir)]. +- Tag for privacy review if needed. + + +Resource leaks +-------------- + +- In Java, memory leaks are largely due to singletons holding on to + caches and collections, or observers sticking around, or runnables + sitting in a queue. +- In C++, cycle-collect as needed. If JavaScript can see your object, + it probably needs to be cycle-collected. +- [Fennec: If your custom view does animations, it's better to clean up + runnables in onDetachFromWindow().] +- Ensure all file handles and other closeable resources are closed + appropriately. +- [Fennec: When writing tests that use PaintedSurface, ensure the + PaintedSurface is closed when you're done with it.] + + +Performance impact +------------------ + +- Check for main-thread IO [Fennec: Android may warn about this with + strictmode]. +- Remove debug logging that is not needed in production. + + +Threading issues +---------------- + +- Enormous: correct use of locking and volatility; livelock and + deadlock; ownership. +- [Fennec: All view methods should be touched only on UI thread.] +- [Fennec: Activity lifecycle awareness (works with "never keep + activities"). Also test with oom-fennec + (`https://hg.mozilla.org/users/blassey_mozilla.com/oom-fennec/) <https://hg.mozilla.org/users/blassey_mozilla.com/oom-fennec/%29>`__]. + + +Compatibility +------------- + +- Version files, databases, messages +- Tag messages with ids to disambiguate callers. +- IDL UUIDs are updated when the interface is updated. +- Android permissions should be 'grouped' into a common release to + avoid breaking auto-updates. +- Android APIs added since Froyo should be guarded by a version check. + + +Preffability +------------ + +- If the feature being worked on is covered by prefs, make sure they + are hooked up. +- If working on a new feature, consider adding prefs to control the + behavior. +- Consider adding prefs to disable the feature entirely in case bugs + are found later in the release cycle. +- [Fennec: "Prefs" can be Gecko prefs, SharedPreferences values, or + build-time flags. Which one you choose depends on how the feature is + implemented: a pure Java service can't easily check Gecko prefs, for + example.] + + +Strings +------- + +- There should be no string changes in patches that will be uplifted + (including string removals). +- Rev entity names for string changes. +- When making UI changes, be aware of the fact that strings will be + different lengths in different locales. + + +Documentation +------------- + +- The commit message should describe what the patch is changing (not be + a copy of the bug summary). The first line should be a short + description (since only the first line is shown in the log), and + additional description, if needed, should be present, properly + wrapped, in later lines. +- Adequately document any potentially confusing pieces of code. +- Flag a bug with dev-doc-needed if any addon or web APIs are affected. +- Use Javadocs extensively, especially on any new non-private methods. +- When moving files, ensure blame/annotate is preserved. + + +Accessibility +------------- + +- For HTML pages, images should have the alt attribute set when + appropriate. Similarly, a button that is not a native HTML button + should have role="button" and the aria-label attribute set. +- [Fennec: Make sure contentDescription is set for parts of the UI that + should be accessible] diff --git a/docs/contributing/reviews.rst b/docs/contributing/reviews.rst new file mode 100644 index 0000000000..d2424724a2 --- /dev/null +++ b/docs/contributing/reviews.rst @@ -0,0 +1,154 @@ +Getting reviews +=============== + + +Thorough code reviews are one of Mozilla's ways of ensuring code quality. +Every patch must be reviewed by the module owner of the code, or one of their designated peers. + +To request a review, you will need to specify a review group (starts with #). If there is not, you should select one or more usernames either when you submit the patch, or afterward in the UI. +If you have a mentor, the mentor can usually either also review or find a suitable reviewer on your behalf. + +For example, the syntax to request review from a group should be: + +.. code-block:: + + Bug xxxx - explain what you are doing and why r?#group-name + + or + + Bug xxxx - explain what you are doing and why r?developer-nickname + +Getting attention: If a reviewer doesn't respond within a week, or so of the review request: + + * Contact the reviewer directly (either via e-mail or on Matrix). + * Join developers on `Mozilla's Matrix server <https://chat.mozilla.org>`_, and ask if anyone knows why a review may be delayed. Please link to the bug too. + * If the review is still not addressed, mail the reviewer directly, asking if/when they'll have time to review the patch, or might otherwise be able to review it. + +For simple documentation changes, reviews are not required. + +For more information about the review process, see the :ref:`Code Review FAQ`. + +Review groups +------------- + + +.. list-table:: + :header-rows: 1 + + * - Name + - Owns + - Members + * - #anti-tracking + - `Core: Anti-Tracking </mots/index.html#core-anti-tracking>`__ + - `Member list <https://phabricator.services.mozilla.com/project/members/157/>`__ + * - #build or #firefox-build-system-reviewers + - The configure & build system + - `Member list <https://phabricator.services.mozilla.com/project/members/20/>`__ + * - #cookies + - `Core: Cookies </mots/index.html#core-cookies>`__ + - `Member list <https://phabricator.services.mozilla.com/project/members/177/>`__ + * - #desktop-theme-reviewers + - User interface CSS + - `Member list <https://phabricator.services.mozilla.com/project/members/141/>`__ + * - #devtools-reviewers + - Firefox DevTools + - `Member list <https://phabricator.services.mozilla.com/project/members/153/>`__ + * - #dom-workers-and-storage-reviewers + - DOM Workers & Storage + - `Member list <https://phabricator.services.mozilla.com/project/members/115/>`__ + * - #fluent-reviewers + - Changes to Fluent (FTL) files (translation). + - `Member list <https://phabricator.services.mozilla.com/project/members/105/>`__ + * - #firefox-source-docs-reviewers + - Documentation files and its build + - `Member list <https://phabricator.services.mozilla.com/project/members/118/>`__ + * - #firefox-ux-team + - User experience (UX) + - `Member list <https://phabricator.services.mozilla.com/project/members/91/>`__ + * - #firefox-svg-reviewers + - SVG-related changes + - `Member list <https://phabricator.services.mozilla.com/project/members/97/>`__ + * - #geckoview-reviewers + - Changes to GeckoView + - `Member list <https://phabricator.services.mozilla.com/project/members/92/>`__ + * - #gfx-reviewers + - Changes to Graphics code + - `Member list <https://phabricator.services.mozilla.com/project/members/122/>`__ + * - #webgpu-reviewers + - Changes to WebGPU code + - `Member list <https://phabricator.services.mozilla.com/project/members/170/>`__ + * - #intermittent-reviewers + - Test manifest changes + - `Member list <https://phabricator.services.mozilla.com/project/members/110/>`__ + * - #layout-reviewers + - Layout changes. + - `Member list <https://phabricator.services.mozilla.com/project/members/126/>`__ + * - #linter-reviewers + - tools/lint/* + - `Member list <https://phabricator.services.mozilla.com/project/members/119/>`__ + * - #mac-reviewers + - Changes to Mac-specific code + - `Member list <https://phabricator.services.mozilla.com/project/members/149/>`__ + * - #mozbase + - Changes to Mozbase + - `Member list <https://phabricator.services.mozilla.com/project/members/113/>`__ + * - #mozbase-rust + - Changes to Mozbase in Rust + - `Member list <https://phabricator.services.mozilla.com/project/members/114/>`__ + * - #necko-reviewers + - Changes to network code (aka necko, aka netwerk) + - `Member list <https://phabricator.services.mozilla.com/project/members/127/>`__ + * - #nss-reviewers + - Changes to Network Security Services (NSS) + - `Member list <https://phabricator.services.mozilla.com/project/members/156/>`__ + * - #perftest-reviewers + - Perf Tests + - `Member list <https://phabricator.services.mozilla.com/project/members/102/>`__ + * - #permissions or #permissions-reviewers + - `Permissions </mots/index.html#core-permissions>`__ + - `Member list <https://phabricator.services.mozilla.com/project/members/158/>`__ + * - #places-reviewers + - `Bookmarks & History (Places) </mots/index.html#bookmarks-history>`__ + - `Member list <https://phabricator.services.mozilla.com/project/members/186/>`__ + * - #platform-i18n-reviewers + - Platform Internationalization + - `Member list <https://phabricator.services.mozilla.com/project/members/150/>`__ + * - #preferences-reviewers + - Firefox for Desktop Preferences (Options) user interface + - `Member list <https://phabricator.services.mozilla.com/project/members/132/>`__ + * - #remote-debugging-reviewers + - Remote Debugging UI & tools + - `Member list <https://phabricator.services.mozilla.com/project/members/108/>`__ + * - #search-reviewers + - Search Reviewers (search parts of `Search and Address Bar </mots/index.html#search-and-address-bar>`__) + - `Member list <https://phabricator.services.mozilla.com/project/members/169/>`__ + * - #spidermonkey-reviewers + - SpiderMonkey JS/Wasm Engine + - `Member list <https://phabricator.services.mozilla.com/project/members/173/>`__ + * - #static-analysis-reviewers + - Changes related to Static Analysis + - `Member list <https://phabricator.services.mozilla.com/project/members/120/>`__ + * - #style or #firefox-style-system-reviewers + - Firefox style system (servo, layout/style). + - `Member list <https://phabricator.services.mozilla.com/project/members/90/>`__ + * - #webcompat-reviewers + - System addons maintained by the Web Compatibility team + - `Member list <https://phabricator.services.mozilla.com/project/members/124/>`__ + * - #webdriver-reviewers + - Marionette and geckodriver (including MozBase Rust), and Remote Protocol with WebDriver BiDi, and CDP. + - `Member list <https://phabricator.services.mozilla.com/project/members/103/>`__ + * - #webidl + - Changes related to WebIDL + - `Member list <https://phabricator.services.mozilla.com/project/members/112/>`__ + * - #xpcom-reviewers + - Changes related to XPCOM + - `Member list <https://phabricator.services.mozilla.com/project/members/125/>`__ + * - #media-playback-reviewers + - `Media playback <https://wiki.mozilla.org/Modules/All#Media_Playback>`__ + - `Member list <https://phabricator.services.mozilla.com/project/profile/159/>`__ + * - #cubeb-reviewers + - Changes related to cubeb, Gecko's audio input/output library and associated projects (audioipc, cubeb-rs, rust cubeb backends) + - `Member list <https://phabricator.services.mozilla.com/project/profile/129/>`__ + +To create a new group, fill a `new bug in Conduit::Administration <https://bugzilla.mozilla.org/enter_bug.cgi?product=Conduit&component=Administration>`__. +See `bug 1613306 <https://bugzilla.mozilla.org/show_bug.cgi?id=1613306>`__ as example. diff --git a/docs/contributing/stack_quickref.rst b/docs/contributing/stack_quickref.rst new file mode 100644 index 0000000000..13942bd1b1 --- /dev/null +++ b/docs/contributing/stack_quickref.rst @@ -0,0 +1,167 @@ +Working with stack of patches Quick Reference +============================================= + +Working on Firefox, we strongly recommend working with stack of patches. +Patches should be small and could be landed in the order used to push them. +This also helps to breakdown the work for different reviewers. + +As it can be complex for newcomers, this documentation explains the +various commands. + +In Phabricator, the stack can be seen in the `Revision Contents` section. +The top of the stack (most recent change) is first in the list. + +This is also sometimes called "stack of revisions", "stack of commits" or "series of commits". + +**Example:** + +.. image:: img/example-stack.png + + +For the overall quick reference guide, see the :ref:`Firefox Contributors Quick Reference <Firefox Contributors' Quick Reference>` + +Visualize the stack +------------------- + +.. code-block:: shell + + # Mercurial + $ hg wip + + # Git + $ git log + + +Merge two patches +----------------- + +It can happen that, instead of updating a patch, a new revision is +created on Phabricator. For this, merge the patches locally: + +.. code-block:: shell + + # Mercurial + # Mark the patch to be merged with "roll" (key: "r") + # or "fold" (key: "f") + $ hg histedit + + # Git + # Replace "pick" by "squash" or "fixup" + $ git rebase -i + +Then, push to Phabricator and abandon the old change. + + +Submitting the first patch on the stack +--------------------------------------- + +There are times when you are working on multiple patches and +just want to submit the first one. For this, you can use: + +.. code-block:: shell + + $ moz-phab submit . + + +Reorder the stack +----------------- + +Sometimes, we want to change the order the patches in the stack. +Fortunately, VCS support this easily. + +.. code-block:: shell + + # Mercurial + # Just change the order of the patches. The tool should highlight + # potential risks of conflicts. + # Note that ctrl+c works well if used. + $ hg histedit + + # Git + # In the editor, just move the patches to the line below/above to + # reorder commits. + # Remove everything if you want to cancel the operation. + $ git rebase -i + + +Make a change on a patch at the beginning of the stack +------------------------------------------------------ + +In some cases, the reviewer is asking for a change at the bottom of the stack (ie not at the top). +So, a simple `hg/git commit --amend` would not work. + +In such case, the following approach can be used: + +.. code-block:: shell + + # Mercurial + # hg will try to guess in which an unambiguous prior commit + $ hg absorb + + # if this doesn't work, create a temporary commit + # and merge it using "fold" or "roll" + $ hg histedit + + # Git + $ git commit --fixup <hash of the commit> + + +Removing patches in the stack +----------------------------- + +To remove a patch in the stack: + +.. code-block:: shell + + # Mercurial + # select "drop" (letter "d") + $ hg histedit + + # Git + # Replace "pick" by "drop" + # Or simply remove the line for this commit + $ git rebase -i + + +Rebasing the stack +------------------ + +As the codebase moves fast, it can be necessary to pull changes from +mozilla-central before landing the changes. + +.. code-block:: shell + + # Mercurial + # First, see where your patches are in the stack + $ hg wip + # Then, rebase it: + # If you are a beginner, don't hesitate to add "--dry-run" + $ hg pull + $ hg rebase -b . -d central + + + # Git + $ git remote update + $ git rebase mozilla/central + + +Reorganizing the stack in Phabricator +------------------------------------- + +.. code-block:: shell + + $ moz-phab reorg [start_rev] [end_rev] + +allows you to reorganize the stack in Phabricator. + +If you've changed the local stack by adding, removing or moving the commits around, you need to change the parent/child relation of the revisions in Phabricator. + +.. code-block:: shell + + $ moz-phab reorg + +command will compare the stack, display what will be changed and ask for permission before taking any action. + +.. note:: + + Note that reviewbot will not restart the analysis. diff --git a/docs/contributing/vcs/mercurial.rst b/docs/contributing/vcs/mercurial.rst new file mode 100644 index 0000000000..dc52b97e8a --- /dev/null +++ b/docs/contributing/vcs/mercurial.rst @@ -0,0 +1,194 @@ +Mercurial Overview +================== + +Mercurial is a source-code management tool which allows users to keep track of changes to the source code locally and share their changes with others. +We also support Git as VCS (see :ref:`Firefox Contributors' Quick Reference`). + +Installation +------------ + +See `Mercurial Page <https://www.mercurial-scm.org/downloads>`__ for installation. + + +Using `hg clone` +---------------- + +If you are not worried about network interruptions, then you can simply +use Mercurial to directly clone the repository you're interested in +using its URL, as given below. For example, to use the command line to +clone ``mozilla-central`` into a directory called ``firefox-source``, +you would use the following: + +.. code-block:: shell + + hg clone https://hg.mozilla.org/mozilla-central/ firefox-source + cd firefox-source + +Using Mercurial bundles +----------------------- + +If you are worried that your Internet connection is not fast or robust +enough to download such a large amount of data all in one go without +being interrupted and cannot clone using the command given above, then you are recommended to try :ref:`Mercurial bundles <Mercurial bundles>`. If interrupted, they can be resumed (continued without downloading +from the beginning) if the app you're using to download supports it. For +example, in Firefox you would right click on the download and select +`Resume` once your connection to the Internet was reestablished. + +Basic configuration +------------------- + +You should configure Mercurial before submitting patches to Mozilla. + +If you will be pulling the Firefox source code or one of the derived repositories, the easiest way to configure Mercurial is to run the vcs-setup mach command: + +.. code-block:: shell + + $ ./mach vcs-setup + +This command starts an interactive wizard that will help ensure your Mercurial is configured with the latest recommended settings. This command will not change any files on your machine without your consent. + + +Other configuration tips +------------------------ + +If you don't have the Firefox source code available, you should edit your Mercurial configuration file to look like the following: + +.. code-block:: shell + + [ui] + username = Your Real Name <user@example.com> + merge = your-merge-program (or internal:merge) + + [diff] + git = 1 + showfunc = 1 + unified = 8 + + [defaults] + commit = -v + +These settings can be added to ``$HOME/.hgrc`` (Linux/macOS) or ``$HOME\Mercurial.ini`` (Windows). + +You can configure the editor to use for commit messages using the `editor` option in the `[ui]` section or by setting the `EDITOR` environment variable. + +If you are trying to access the repository through a proxy server, see `these +instructions <http://www.selenic.com/mercurial/hgrc.5.html#http-proxy>`__ + + +Selecting a repository (tree) +----------------------------- + +There are multiple hg repositories hosted at mozilla.org to choose from. +A summary of the main trees is given below, but see +https://hg.mozilla.org/ for the full list. + +mozilla-central +--------------- + +This is the main development tree for Firefox. Most developers write +patches against mozilla-central. + +URL: https://hg.mozilla.org/mozilla-central/ + + +mozilla-beta +------------ + +The source for the current beta version of Firefox (and the next and all +previous betas). This code represents the expected next release of the +Firefox browser, and should be pretty stable. + +URL: https://hg.mozilla.org/releases/mozilla-beta/ + +mozilla-release +--------------- + +The source for the current release of Firefox (and the next and all +previous releases). + +URL: https://hg.mozilla.org/releases/mozilla-release/ + +autoland +-------- + +This is the integration tree for Firefox. Patches land in this repository first, +and then are merged by the sheriffs in mozilla-central. + +URL: https://hg.mozilla.org/integration/autoland/ + +L10n repos +---------- + +Mainly useful for localizers working on localizing Firefox. Code for all +l10n projects lives here and is organized into separate repos that (in +most cases) have the locale's two character ISO code. To get the repo +that you need look for the repo you're interested in on the following +page. + +URL: https://hg.mozilla.org/l10n-central/ + +Unified Repositories +-------------------- + +It is common for advanced users to want to interact with more than one +firefox repository. If you get to the point where having individual +copies of repositories is annoying you, then see +https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html +for instructions on doing this efficiently. + +Selecting a revision to build +----------------------------- + +Most of the time the `tip` revision of most repositories will build +without issue. If you are worried about it not, then you may want to +stick to mozilla-central. + +Building +-------- + +By default with no configuration a similar-to-release build is done. If +you wish you can :ref:`configure <Configuring Build Options>` the build using a ``.mozconfig`` file +and ``mach build``. +Different OSs have different prerequisites for a successful build, +please refer to the :ref:`build documentation <Getting Set Up To Work On The Firefox Codebase>` +to verify they are available on your build machine. + +Extensions +---------- + +There's a number of extensions you can enable. See http://mercurial.selenic.com/wiki/UsingExtensions. Almost everyone should probably enable the following, most of them are enabled by ``mach bootstrap``: + +#. color - Colorize terminal output +#. histedit - Provides git rebase --interactive behavior. +#. rebase - Ability to easily rebase patches on top of other heads. +#. evolve - Enable and enhance the inprogress ChangesetEvolution work. +#. firefoxtree - Enhances the interaction with Firefox repositories. + +These can all be turned on by just adding this to your `.hgrc` file: + +.. code-block:: shell + + [extensions] + color = + rebase = + histedit = + firefoxtree = + evolve = + +In addition, there are some 3rd party extensions that are incredibly +useful for basic development: + +`mozext <https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/mozext>`__ + Mozilla-specific functionality to aid in developing Firefox/Gecko. + +Configuring the try repository +------------------------------ + +About :ref:`Pushing to Try <Pushing to Try>`. + +Learning to use Mercurial +------------------------- + +If you are new to Mercurial, you should start with the `official guide <https://www.mercurial-scm.org/guide>`__. + +Then, move on to the `version control tool docs <https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/>`__ for Mozilla-centric Mercurial information. diff --git a/docs/contributing/vcs/mercurial_bundles.rst b/docs/contributing/vcs/mercurial_bundles.rst new file mode 100644 index 0000000000..66d6c26c72 --- /dev/null +++ b/docs/contributing/vcs/mercurial_bundles.rst @@ -0,0 +1,65 @@ +Mercurial Bundles +================= + +If you have a poor network connection that is preventing ``hg clone`` from completing, you may want to try downloading a bundle of the repository you're interested in. This is useful since a file download, unlike ``hg clone``, can be resumed if the connection is interrupted. Once you have the bundle, staying up-to-date shouldn't take much time at all, if you keep up with it regularly. + +This document explains the steps to setup the `mozilla-unified <https://hg.mozilla.org/mozilla-unified/>`__ repository using a bundle file. Be sure to replace "``mozilla-unified``" with the project you're working with as appropriate. + +Download the bundle +------------------- + +1. Open https://hg.cdn.mozilla.net/ : + +It lists up-to-date bundles for some of the repositories listed at https://hg.mozilla.net/ . +Each row corresponds to each repository, and each column corresponds to each compression format. + +2. Download the bundle file for the ``mozilla-unified`` repository: + +Click the link in the "mozilla-unified" row, the "zstd (max)" column. + +Setting up the repository +------------------------- + +Once you have downloaded the repository bundle, follow the steps below to recreate the repository locally based upon that bundle. + +1. Initialize a new repository (in a directory called ``mozilla-unified`` here): + +.. code-block:: shell + + mkdir mozilla-unified + hg init mozilla-unified + +2. Un-bundle the bundle file to that repository: + +Move the bundle file next to ``mozilla-unified`` directory, and rename it to ``bundle.hg``. + +.. code-block:: shell + + cd mozilla-unified + hg unbundle ../bundle.hg + +Get comfortable. Grab a coffee (or your favorite tasty beverage). Maybe a nap. This unbundling process is going to take quite a lot of time. + +3. Create the repository's config file ``.hg/hgrc``, and add the following lines, so that Mercurial will automatically know where to pull changes from future updates. You can open the template config file in your editor by running ``hg config --local --edit`` or ``EDITOR=<editor-of-your-choice> hg config --local --edit`` + +.. code-block:: shell + + [paths] + default = https://hg.mozilla.org/mozilla-unified/ + +4. Update the repository to get all the changes since the bundle was created (this step also doubles as a check of the bundle integrity since if its contents are not exactly the same as what's in the official repository then the ``hg pull`` will fail): + +.. code-block:: shell + + hg pull + +5. Check out a working copy from your new up to date repository: + +.. code-block:: shell + + hg update + +You now have a clone of ``mozilla-unified`` that is identical to one made via ``hg clone``. You can adjust your build settings, or you can go straight ahead and build Firefox! + +If at any point you are stuck, feel free to ask on Riot/Matrix at `https://chat.mozilla.org <https://chat.mozilla.org>`__ +in `#introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`__ channel. |
