diff options
Diffstat (limited to 'docs')
125 files changed, 15226 insertions, 0 deletions
diff --git a/docs/_static/custom_theme.css b/docs/_static/custom_theme.css new file mode 100644 index 0000000000..4f76e575f4 --- /dev/null +++ b/docs/_static/custom_theme.css @@ -0,0 +1,28 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ + +/* Increase the size of the content */ +.wy-nav-content { + max-width: 80% !important; +} + +/* Increase the size of the tables */ +table.docutils { + width: 90%; +} + +/* Override the default values for multiline text in a table */ +table.docutils td, table.docutils th +{ + font-size: 16px !important; +} + +.rst-content .line-block { + margin-bottom: 0px !important; +} + +/* Add the strikethrough feature */ +span.strikethrough { + text-decoration: line-through; +} diff --git a/docs/bug-mgmt/guides/bug-types.rst b/docs/bug-mgmt/guides/bug-types.rst new file mode 100644 index 0000000000..3626336de1 --- /dev/null +++ b/docs/bug-mgmt/guides/bug-types.rst @@ -0,0 +1,29 @@ +Bug Types +========= + +We organize bugs by type to make it easier to make triage decisions, get +the bug to the right person to make a decision, and understand release +quality. + +- **Defect** regression, crash, hang, security vulnerability and any + other reported issue +- **Enhancement** new feature, improvement in UI, performance, etc. and + any other request for user-facing enhancements to the product, not + engineering changes +- **Task** refactoring, removal, replacement, enabling or disabling of + functionality and any other engineering task + +All bug types need triage decisions. Engineering :ref:`triages defects and +tasks <Triage for Bugzilla>`. Product management :ref:`triages +enhancements <New Feature Triage>`. + +It’s important to distinguish an enhancement from other types because +they use different triage queues. + +Distinguishing between defects and tasks is important because we want to +understand code quality and reduce the number of defects we introduce as +we work on new features and fix existing defects. + +When triaging, a task can be as important as a defect. A behind the +scenes change to how a thread is handled can affect performance as seen +by a user. diff --git a/docs/bug-mgmt/guides/other-metadata.rst b/docs/bug-mgmt/guides/other-metadata.rst new file mode 100644 index 0000000000..f1b94f16d8 --- /dev/null +++ b/docs/bug-mgmt/guides/other-metadata.rst @@ -0,0 +1,28 @@ +Other Bug Metadata +================== + +Performance +----------- + +- Use the ``perf`` keyword +- Add ``[qf:?]`` to the whiteboard if you think the Performance team + should look at this bug + +Privacy +------- + +- Use the ``privacy`` keyword + +User Security +------------- + +- Will this bug adversely affect Firefox users if left public? + + - Add to security group + +- Otherwise move bug to one of: + + - Core:: Security + - Firefox:: Security + - Toolkit:: Security + - Webkit:: Security diff --git a/docs/bug-mgmt/guides/priority.rst b/docs/bug-mgmt/guides/priority.rst new file mode 100644 index 0000000000..db0c8ee874 --- /dev/null +++ b/docs/bug-mgmt/guides/priority.rst @@ -0,0 +1,27 @@ +Priority Definitions +==================== + +We use these definitions across all components: + ++----------------------------------------+-----------------------------+ +| Priority | Description | ++========================================+=============================+ +| \- | No decision | ++----------------------------------------+-----------------------------+ +| P1 | Fix in the current release | +| | cycle | ++----------------------------------------+-----------------------------+ +| P2 | Fix in the next release | +| | cycle or the following | +| | (nightly + 1 or nightly + | +| | 2) | ++----------------------------------------+-----------------------------+ +| P3 | Backlog | ++----------------------------------------+-----------------------------+ +| P4 | Do not use, this priority | +| | is for web platform test | +| | bots | ++----------------------------------------+-----------------------------+ +| P5 | Will not fix, but will | +| | accept a patch | ++----------------------------------------+-----------------------------+ diff --git a/docs/bug-mgmt/guides/severity.rst b/docs/bug-mgmt/guides/severity.rst new file mode 100644 index 0000000000..c292da19cc --- /dev/null +++ b/docs/bug-mgmt/guides/severity.rst @@ -0,0 +1,85 @@ +Defect Severity +=============== + +Definition +---------- + +We use the ``severity`` field in Bugzilla to indicate the scope of a +bug's effect on Firefox. + +The field is display alongside the bug's priority. + +Values +------ + +`Severities are +enumerated <https://wiki.mozilla.org/BMO/UserGuide/BugFields#severity>`__ +as: + +- ``--``: Default for new bugs +- ``N/A``: (not applicable): Only applies to bugs of type Task or Enhancement. +- ``S1``: (Catastrophic) Blocks development/testing, may impact more than 25% + of users, causes data loss, potential chemspill, and no workaround available +- ``S2``: (Serious) Major Functionality/product severely impaired and a + satisfactory workaround doesn't exist +- ``S3``: (Normal) Blocks non-critical functionality and a work around exists +- ``S4``: (Small/Trivial) minor significance, cosmetic issues, low or no impact to users + +By default, new bugs have a severity of ``--``. + +Examples of S1 bugs +^^^^^^^^^^^^^^^^^^^ + +- WebExtensions disabled for all users +- Web search not working from URL bar +- Crashes with data loss + +Examples of S2 bugs +^^^^^^^^^^^^^^^^^^^ + +Bugs that could reasonably be expected to cause a Firefox user to switch browsers, +either because the severity is bad enough, or the frequency of occurrence is high enough. + +- `Bug 1640441 <https://bugzilla.mozilla.org/show_bug.cgi?id=1640441>`__ - Slack hangs + indefinitely in a onResize loop +- `Bug 1645651 <https://bugzilla.mozilla.org/show_bug.cgi?id=1645651>`__ - Changes in + Reddit's comment section JS code makes selecting text slow on Nightly + +Bugs involving contractual partners (if not an S1) + +Bugs reported from QA + +- `Bug 1640913 <https://bugzilla.mozilla.org/show_bug.cgi?id=1640913>`__ - because an + important message is not visible with the dark theme. It's not marked as S1 since the + issue is reproducible only on one OS and the functionality is not affected. +- `Bug 1641521 <https://bugzilla.mozilla.org/show_bug.cgi?id=1641521>`__ - because videos + are not working on several sites with ETP on (default). This is not an S1 since turning + ETP off fixes the issue. + +Examples of S3 bugs +^^^^^^^^^^^^^^^^^^^ + +Bugs filed by contributors as part of daily refactoring and maintenance of the code base. + +`Bug 1634171 <https://bugzilla.mozilla.org/show_bug.cgi?id=1634171>`__ - Visual artifacts around circular images + +Bugs reported from QA + +- `Bug 1635105 <https://bugzilla.mozilla.org/show_bug.cgi?id=1635105>`__ because + the associated steps to reproduce are uncommon, + and the issue is no longer reproducible after refresh. +- `Bug 1636063 <https://bugzilla.mozilla.org/show_bug.cgi?id=1636063>`__ since it's + reproducible only on a specific web app, and only with a particular set of configurations. + + +Rules of thumb +-------------- + +- *A crash may be be a ``S1`` or ``S2`` defect, but not all crashes are + critical defects* +- High severity defects (``S1`` or ``S2``) do not need to be assigned + immediately as they will be reviewed by Engineering Leadership, QA, and + Release Management +- The severity of most bugs of type ``task`` and ``enhancement`` will be + ``N/A`` +- **Do not** assign bugs of type ``defect`` the severity ``N/A`` diff --git a/docs/bug-mgmt/guides/status-flags.rst b/docs/bug-mgmt/guides/status-flags.rst new file mode 100644 index 0000000000..d3a67f993b --- /dev/null +++ b/docs/bug-mgmt/guides/status-flags.rst @@ -0,0 +1,33 @@ +Release Status Flags +==================== + +The flag ``status_firefoxNN`` has many values, here’s a cheat sheet. + +== ========== ========== ============ ================= +— ? unaffected affected fixed +== ========== ========== ============ ================= +? unaffected wontfix verified +\ affected fix-optional disabled +\ fixed verified-disabled +== ========== ========== ============ ================= + +The headers of the table are values of the status flag. Each column are +the states reachable from the column headings. + +- ``---`` we don’t know whether Firefox N is affected +- ``?`` we don’t know whether Firefox N is affected, but we want to find + out. +- ``affected`` - present in this release +- ``unaffected`` - not present in this release +- ``fixed`` - a contributor has landed a change set in the tree + to address the issue +- ``verified`` - the fix has been verified by QA or other contributors +- ``disabled`` - the fix or the feature has been backed out or disabled +- ``verified disabled`` - QA or other contributors confirmed the fix or + the feature has been backed out or disabled +- ``wontfix`` - we have decided not to accept/uplift a fix for this + release cycle (it is not the same as the bug resolution WONTFIX). + This can also mean that we don’t know how to fix that and will ship + with this bug +- ``fix-optional`` - we would take a fix for the current release but + don’t consider it as important/blocking for the release diff --git a/docs/bug-mgmt/index.rst b/docs/bug-mgmt/index.rst new file mode 100644 index 0000000000..215b2e7317 --- /dev/null +++ b/docs/bug-mgmt/index.rst @@ -0,0 +1,40 @@ +Bug Handling +============ + +Guides +------ + +.. toctree:: + :maxdepth: 1 + :glob: + + guides/* + +Policies +-------- + +.. toctree:: + :maxdepth: 1 + :glob: + + policies/* + +Processes +--------- + +.. toctree:: + :maxdepth: 1 + :glob: + + processes/* + +Related documentation +--------------------- + +- `bugzilla.mozilla.org documentation <https://bmo.readthedocs.org/>`__ +- `bugzilla.mozilla.org field + definitions <https://wiki.mozilla.org/BMO/UserGuide/BugFields>`__ +- `Lando + documentation <https://moz-conduit.readthedocs.io/en/latest/lando-user.html>`__ +- `Mozilla Phabricator (Code Review) + documentation <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`__ diff --git a/docs/bug-mgmt/policies/new-feature-triage.rst b/docs/bug-mgmt/policies/new-feature-triage.rst new file mode 100644 index 0000000000..7bc8022777 --- /dev/null +++ b/docs/bug-mgmt/policies/new-feature-triage.rst @@ -0,0 +1,55 @@ +New Feature Triage +================== + +Identifying Feature Requests +---------------------------- + +Bugs which request new features or enhancements should be of +type=\ ``enhancement``. + +Older bugs may also be feature requests if some or all of the following +are true: + +- Bugs with ``feature`` or similar in whiteboard or short description +- ``[RFE]`` in whiteboard, short description, or description +- Bugs not explicitly marked as a feature request, but appear to be + feature requests +- Bugs marked with ``feature`` keyword + +Initial Triage +-------------- + +Staff, contractors, and other contributors looking at new bugs in +*Firefox::Untriaged* and *::General* components should consider if a +bug, if not marked as a feature request, should be one, and if so: + +- Update the bug’s type to ``enhancement`` +- Determine which product and component the bug belongs to and update + it **or** + + - Use *needinfo* to ask a component’s triage owner or a module’s + owner where the request should go + +Product Manager Triage +---------------------- + +- The product manager for the component reviews bugs of type + ``enhancement`` + + - This review should be done a least weekly + +- Reassigns to another Product::Component if necessary **or** +- Determines next steps + + - Close bug as ``RESOLVED WONTFIX`` with comment as to why and + thanking submitter + - If bug is similar enough to work in roadmap, close bug as + ``RESOLVED DUPLICATE`` of the feature bug it is similar to + + - If there’s not a feature bug created already, then consider + making this bug the feature bug + + - Set type to ``enhancement`` + + - Set bug to ``P5`` priority with comment thanking submitter, and + explaining that the request will be considered for future roadmaps diff --git a/docs/bug-mgmt/policies/regressions-github.rst b/docs/bug-mgmt/policies/regressions-github.rst new file mode 100644 index 0000000000..1a3c6b2a4d --- /dev/null +++ b/docs/bug-mgmt/policies/regressions-github.rst @@ -0,0 +1,151 @@ +Regressions from GitHub +======================= + +Release Management and the weekly regression triage must be aware of the +status of all reported regressions in order to assure we are not +shipping known regressions in Firefox releases. + +If a team is using GitHub to manage their part of the Firefox project, +there’s a risk that those groups might not see a regression. + +We need an agreed to standard for how we keep track of these. + +Policy +------ + +*All Firefox components, even if their bugs are tracked off of Bugzilla, +must have a component in Bugzilla.* + +*If a regression bug is found in any of the release trains (Nightly, +Beta, Release, or ESR) and the bug is in a Firefox component which uses +an external repository, the regression must be tracked by a bug in +Bugzilla (whether or not the component in question uses an external +issue tracker).* + +*Unless approved by Release Management, any GitHub managed code with +open regressions will not be merged to mozilla-central. Even if the +regression is not code that has been previously merged into +mozilla-central.* + +*All Firefox code managed in GitHub which uses GitHub to manage +issues* `must use the shared +tags <https://mozilla.github.io/bmo-harmony/labels>`__. + +Comments +~~~~~~~~ + +The bug **must** have the regression keyword. + +The bug **must** have release flags set. + +If the team works in an external bug tracker, then the Bugzilla bug +**must** reference, using the see-also field, the URL of the bug in the +external tracker. + +The bug **must not** be RESOLVED until the code from the external +repository containing the change set for the bug has landed in +mozilla-central. When the change set lands in mozilla-central, the +Bugzilla tracking bug should be set to RESOLVED FIXED and release status +flags should be updated to reflect the release trains the fix has been +landed or uplifted into. + +If the change set containing the patch for the regression is reverted +from mozilla-central, for any reason, then the tracking bug for the +regression **must** be set to REOPENED and the release status flags +updated accordingly. + +If the change set containing the patch for the bug is backed out, for +any reason, the bug must be reopened and the status flags on the +Bugzilla tracking bug updated. + +The team responsible for the component with the regression **should** +strive to create a patch for mozilla-central which contains the fix for +the bug alone, not a monolithic patch containing changes for several +other bugs or features. + +Landings of third-party libraries `must contain a manifest +file <https://docs.google.com/document/d/12ihxPXBo9zBBaU_pBsPrc_wNHds4Upr-PwFfiSHrbu8>`__. + +Best Practices +-------------- + +You must file a regression bug in Bugzilla +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*If the code with the regression has landed in mozilla-central, you must +file a regression bug.* + +Example +^^^^^^^ + +While using a release of Firefox (Nightly, Beta, Release, ESR) you run +across a bug. Upon research using MozRegression or other tools you find +that the bug was introduced in a change set imported from a component +whose code and issues are managed in GitHub. + +Actions to take +''''''''''''''' + +- Open a new bug in Bugzilla in appropriate component and add the + REGRESSION keyword +- Set affected status for the releases where the bug appears +- Open an issue in the corresponding GitHub project, put the Bugzilla + bug number in the title with the prefix ‘Bug’ (i.e. “Bug 99999: + Regression in foo”) +- Add the REGRESSION label to the new issue +- Add the link to the GitHub issue into the ‘See Also” field in the + Bugzilla bug + +Consequences +'''''''''''' + +*Until the regression is fixed or backed out of the GitHub repo, the +project cannot merge code into mozilla-central* + +Example +^^^^^^^ + +You import a development branch of a component managed in GitHub into +your copy of master. You find a bug and isolate it to the imported +branch. The code is managed in their own GitHub project, but bugs are +managed in Bugzilla. + +Actions to take +''''''''''''''' + +- Open a new bug in Bugzilla in appropriate component and add the + REGRESSION keyword +- Set affected status for the releases where the bug appears + +Consequences +'''''''''''' + +*Until the regression is fixed or backed out of the GitHub repo, the +project cannot merge code into mozilla-central* + +Do not file a regression bug in Bugzilla +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*If the code with the regression has not landed in mozilla-central, you +do not need to file a bug.* + + +Example +^^^^^^^ + +You import a development branch of a component managed in GitHub into +your copy of master. You find a bug and isolate it to the imported +branch. The code and issues are managed in their own GitHub project. + + +Actions to take +''''''''''''''' + +- File new issue in the GitHub repository of the imported code. +- Label issue as REGRESSION + +Consequence +''''''''''' + +*Issue blocks merge of GitHub project with mozilla-central until +resolved or backed out.* diff --git a/docs/bug-mgmt/policies/triage-bugzilla.rst b/docs/bug-mgmt/policies/triage-bugzilla.rst new file mode 100644 index 0000000000..498e9955a8 --- /dev/null +++ b/docs/bug-mgmt/policies/triage-bugzilla.rst @@ -0,0 +1,278 @@ +Triage for Bugzilla +=================== + +Expectations +------------ + +All teams working on Firefox using either or both Mozilla-central and +Bugzilla are expected to follow the following process. + +What is a Triaged Bug +--------------------- + +The new definition of Triaged will be Firefox-related bugs of type +``defect`` where the component is not +``UNTRIAGED``, and a severity value not equal to ``--`` or ``N/A``. + +Bugs of type Task or Enhancement may have a severity of ``N/A``, +but defects must have a severity that is neither ``--`` or +``N/A``. + +Why Triage +---------- + +We want to make sure that we looked at every defect and a severity has +been defined. This way, we make sure that we did not miss any critical +issues during the development and stabilization cycles. + +Staying on top of the bugs in your component means: + +- You get ahead of critical regressions and crashes which could trigger + a point release if uncaught + + - And you don’t want to spend your holiday with the Release + Management team (not that they don’t like you) + +- Your bug queue is not overwhelming + + - Members of your team do not see the bug queue and get the + ‘wiggins’ + +Who Triages +----------- + +Engineering managers and directors are responsible for naming the +individuals responsible for triaging :ref:`all types of bugs <Bug Types>` in a component. + +We use Bugzilla to manage this. See the `list of triage +owners <https://bugzilla.mozilla.org/page.cgi?id=triage_owners.html>`__. + +If you need to change who is responsible for triaging a bug in a +component, please `file a bug against bugzilla.mozilla.org in the +Administration +component <https://bugzilla.mozilla.org/enter_bug.cgi?product=bugzilla.mozilla.org&component=Administration>`__. +When a new component is created, a triage owner **must** be named. + +Rotating triage +~~~~~~~~~~~~~~~ + +Some components are monitored by a rotation of triagers. In those cases, +the triage owner should be seen as the person responsible for assuring +the component is triaged, but the work is done by the people in the +rotation. The `rotations are managed as +calendars <https://github.com/mozilla/relman-auto-nag/tree/master/auto_nag/scripts/configs>`__. + +If you wish to set up a rotation for triaging one or more components, +contact the Bugzilla team on Slack (#bmo.) + +Firefox::General and Toolkit::General +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Bugs in Firefox::General are fitted with Bug Bug’s model to see if +there’s another component with a high liklihood of fit, and if a +threshold confidence is achieved, the bug is moved to that component. + +Members of the community also review bugs in this component and try to +move them. + +What Do You Triage +------------------ + +As a triage owner the queries you should be following for your component +are: + +- All open bugs, in your components without a pending ``needinfo`` flag + which do not have a valid value of severity set +- All bugs with active review requests in your component which have not + been modified in five days +- All bugs with reviewed, but unlanded patches in your components +- All bugs with a needinfo request unanswered for more than 10 days + +There’s a tool with these queries to help you find bugs +https://mozilla.github.io/triage-center/ and the source is at +https://github.com/mozilla/triage-center/. + +If a bug is an enhancement it needs a priority set and a target release +or program milestone. These bugs are normally reviewed by product +managers. Enhancements can lead to release notes and QA needed that we +also need to know about + +If a bug is a task resulting in a changeset, release managers will need +to known when this work will be done. A task such as refactoring fragile +code can be risky. + +Weekly or More Frequently (depending on the component) find un-triaged +bugs in the components you triage. + +Decide the :ref:`Severity <Defect Severity>` for each untriaged bug +(you can override what’s already been set.) + +These bugs are reviewed in the weekly Regression Triage meeting + +- Bugs of type ``defect`` with the ``regression`` keyword without + ``status-firefoxNN`` decisions +- Bugs of type ``defect`` with the ``regression`` keyword without + a regression range + +Automatic Bug Updates +~~~~~~~~~~~~~~~~~~~~~ + +When a bug is tracked for a release, i.e. the ``tracking_firefoxNN`` +flag is set to ``+`` or ``blocking`` triage decisions will be overridden, +or made as follows: + +- If a bug is tracked for or blocking beta, release or ESR, its + priority will be set to ``P1`` +- If a bug is tracked for or blocking nightly, its priority will be set + to ``P2`` + +Because bugs can be bumped in priority it’s essential that triage owners +review their +`P1 <https://bugzilla.mozilla.org/buglist.cgi?priority=P1&f1=triage_owner&o1=equals&resolution=---&v1=%25user%25>`__ +and +`P2 <https://bugzilla.mozilla.org/buglist.cgi?priority=P2&f1=triage_owner&o1=equals&resolution=---&v1=%25user%25>`__ +bugs frequently. + +Assumptions +~~~~~~~~~~~ + +If a bug's release status in Firefox version N was ``affected`` or ``wontfix``, +its Severity is ``S3`` or ``S4`` and its Priority is ``P3`` or lower (backlog,) +then its release status in Firefox version N+1, if the bug is still open, +is considered to be ``wontfix``. + +Questions and Edge Cases +------------------------ + +This bug is a feature request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the bug’s type to ``enhancement``, add the ``feature`` keyword if +relevant, and state to ``NEW``. Set the bug's Severity to ``N/A``. This +bug will be excluded from future triage queries. + +This bug is a task, not a defect +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the bug’s type to ``task``, and state to ``NEW``. Set the bug's +Severity to ``N/A``. This bug will be excluded from future triage queries. + + +If you are not sure of a bug’s type, check :ref:`our rules for bug +types <Bug Types>`. + +This bug’s state is ``UNCONFIRMED`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Are there steps to reproduce? If not, needinfo the person who filed the +bug, requesting steps to reproduce. You are not obligated to wait +forever for a response, and bugs for which open requests for information +go unanswered can be ``RESOLVED`` as ``INCOMPLETE``. + +I need help reproducing the bug +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set a needinfo for the QA managers, Softvision project managers, or the +QA owner of the component of the bug. + +I don’t have enough information to make a decision +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you don’t have a reproduction or confirmation, or have questions +about how to proceed, ``needinfo`` the person who filed the bug, or +someone who can answer. + +The ``stalled`` keyword +~~~~~~~~~~~~~~~~~~~~~~~ + +The extreme case of not-enough-information is one which cannot be +answered with a ``needinfo`` request. The reporter has shared all they +know about the bug, we are out of strategies to take to resolve it, but +the bug should be kept open. + +Mark the bug as stalled by adding the ``stalled`` keyword to it. The +keyword will remove it from the list of bugs to be triaged. + +If a patch lands on a ``stalled`` bug, automation will remove the +keyword. Otherwise, when the ``keyword`` is removed, the bug will have +its priority reset to ``--`` and the components triage owner notified by +automation. + +Bugs which remain ``stalled`` for long periods of time should be +reviewed, and closed if necessary. + +Bug is in the wrong Component +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the bug has a Severity of ``S3``, ``S4``, or ``N/A`` move the what +you think is the correct component, or needinfo the person +responsible for the component to ask them. + +If the bug has a Severity of ``S1`` or ``S2`` then notify Release Management +and contact the triage owner of the component for which you think it belongs to. +We cannot lose track of a high severity bug because it is in the wrong component. + +My project is on GitHub +~~~~~~~~~~~~~~~~~~~~~~~ + +We have :ref:`a guide for GitHub projects to follow <GitHub Metadata Recommendations>` when +triaging. (Note: this guide needs updating.) + +Summary +------- + +Multiple times weekly +~~~~~~~~~~~~~~~~~~~~~ + +Use queries for the components you are responsible for in +https://mozilla.github.io/triage-center/ to find bugs in +need of triage. + +For each untriaged bug: + +- Assign a Severity +- **Do not** assign a ``defect`` a Severity of + ``N/A`` + +You can, but are not required to set the bug's :ref:`Priority <Priority Definitions>`. + +Watch open needinfo flags +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Don’t let open needinfo flags linger for more than two weeks. + +Close minor bugs with unresponded needinfo flags. + +Follow up on needinfo flag requests. + +The `Triage Center tool <https://mozilla.github.io/triage-center/>`__ will help you find these. + +End of Iteration/Release Cycle +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Any open ``S1`` or ``S2`` bugs at the end of the release cycle +will require review by engineering and release management. A +policy on this is forthcoming. + +Optional +^^^^^^^^ + +(The guidelines on bug priority are under review.) + +Are there open P1s? Revisit their priority, +and move to them to the backlog (``P3``) or ``P2``. + +Are there ``P2`` bugs that should move to ``P1`` +for the next cycle? + +Are there ``P2`` you now know are lower priority, +move to ``P3``. + +Are there ``P3`` bugs you now know you won’t get to? +Either demote to ``P5`` (will accept patch) or +resolve as ``WONTFIX``. + +Getting help +------------ + +- Ask in #bug-handling on chat.mozilla.org diff --git a/docs/bug-mgmt/processes/accessibility-review.md b/docs/bug-mgmt/processes/accessibility-review.md new file mode 100644 index 0000000000..413eb8e6c6 --- /dev/null +++ b/docs/bug-mgmt/processes/accessibility-review.md @@ -0,0 +1,49 @@ +# Accessibility Review + +## Introduction +At Mozilla, accessibility is a fundamental part of our mission to ensure the +internet is "open and accessible to all," helping to empower people, regardless +of their abilities, to contribute to the common good. Accessibility Review is a +service provided by the Mozilla Accessibility team to review features and +changes to ensure they are accessible to and inclusive of people with +disabilities. + +## Do I Need Accessibility Review? +You should consider requesting accessibility review if you aren't certain +whether your change is accessible to people with disabilities. Accessibility +review is optional, but it is strongly encouraged if you are introducing new +user interface or are significantly redesigning existing user interface. + +## When Should I Request Accessibility Review? +Generally, it's best to request accessibility review as early as possible, even +during the product requirements or UI design stage. Particularly for more +complex user interfaces, accessibility is much easier when incorporated into the +design, rather than attempting to retro-fit accessibility after the +implementation is well underway. + +The accessibility team has developed the [Mozilla Accessibility Release +Guidelines](https://wiki.mozilla.org/Accessibility/Guidelines) which outline +what is needed to make user interfaces accessible. To make accessibility review +faster, you may wish to try to verify and implement these guidelines prior to +requesting accessibility review. + +The deadline for accessibility review requests is Friday of the first week of +nightly builds for the release in which the feature/change is expected to ship. +This is the same date as the PI Request deadline. + +## How Do I Request Accessibility Review? +You request accessibility review by setting the a11y-review flag to "requested" +on a bug in Bugzilla and filling in the template that appears in the comment +field. For features spanning several bugs, you may wish to file a new, dedicated +bug for the accessibility review. Otherwise, particularly for smaller changes, +you may do this on an existing bug. Note that if you file a new bug, you will +need to submit the bug and then edit it to set the flag. + +## Questions? +If you have any questions, please don't hesitate to contact the Accessibility +team: + +* \#accessibility on + [Matrix](https://matrix.to/#/!jmuErVonajdNMbgdeY:mozilla.org?via=mozilla.org&via=matrix.org) + or Slack +* Email: accessibility@mozilla.com diff --git a/docs/bug-mgmt/processes/doc-requests.rst b/docs/bug-mgmt/processes/doc-requests.rst new file mode 100644 index 0000000000..b725e10747 --- /dev/null +++ b/docs/bug-mgmt/processes/doc-requests.rst @@ -0,0 +1,44 @@ +User documentation requests +=========================== + +If you are working on a change (bugfix, enhancement, or feature) which +would benefit from user-facing documentation, please use the +``user-doc-firefox`` flag to request it. + +This flag can be modified by anyone with ``EDITBUGS`` privileges. + +.. figure:: /public/images/sumo-flag.png + :alt: Image of flag in tracking section of bug + + Image of flag in tracking section of bug + +The default value of the flag is ``---``. + +If the bug needs user-facing documentation, set the flag to +``docs-needed``. This flag will be monitored by the support.mozilla.org +(SUMO) team. + +Once the docs are ready to be published, set the flag to +``docs-completed``. + +If it’s determined that documentation is not need after setting the flag +to ``docs-needed``, update the flag to ``none-needed`` so we know that +it’s been reviewed. + +Summary +------- + +=========== == ============== +From To +=========== == ============== +— to none-needed +— to docs-needed +docs-needed to none-needed +docs-needed to docs-completed +=========== == ============== + +Notes +----- + +A flag is used instead of the old keywords because flags can be +restricted to a subset of products and components. diff --git a/docs/bug-mgmt/processes/fixing-security-bugs.rst b/docs/bug-mgmt/processes/fixing-security-bugs.rst new file mode 100644 index 0000000000..075cf7d65a --- /dev/null +++ b/docs/bug-mgmt/processes/fixing-security-bugs.rst @@ -0,0 +1,159 @@ +Fixing Security Bugs +==================== + +A bug has been reported as security-sensitive in Bugzilla and received a +security rating. + +If this bug is private - which is most likely for a reported security +bug - **the process for patching is slightly different than the usual +process for fixing a bug**. + +Here are security guidelines to follow if you’re involved in reviewing, +testing and landing a security patch. See +:ref:`Security Bug Approval Process` +for more details about how to request sec-approval and land the patch. + +Keeping private information private +----------------------------------- + +A security-sensitive bug in Bugzilla means that all information about +the bug except its ID number are hidden. This includes the title, +comments, reporter, assignee and CC’d people. + +A security-sensitive bug usually remains private until a fix is shipped +in a new release, **and after a certain amount of time to ensure that a +maximum number of users updated their version of Firefox**. Bugs are +usually made public after 6 months and a couple of releases. + +From the moment a security bug has been privately reported to the moment +a fix is shipped and the bug is set public, all information about that +bug need to be handled carefully in order to avoid an unmitigated +vulnerability to be known and exploited out there before we release a +fix (0-day). + +During a normal process, information about the nature of bug can be +accessed through: + +- Bug comments (Bugzilla, GitHub issue) +- Commit message (visible on Bugzilla, tree check-ins and test servers) +- Code comments +- Test cases +- Bug content can potentially be discussed on public IRC/Slack channels + and mailing list emails. + +When patching for a security bug, you’ll need to be mindful about what +type of information you share and where. + +In commit messages +~~~~~~~~~~~~~~~~~~ + +People are watching code check-ins, so we want to avoid sharing +information which would disclose or help finding a vulnerability too +easily before we shipped the fix to our users. This includes: + +- The **nature of the vulnerability** (overflow, use-after-free, XSS, + CSP bypass...) +- **Ways to trigger and exploit that vulnerability** + - In commit messages, code comments and test cases. +- The fact that a bug / commit is security-related: + + - **Trigger words** in the commit message or code comments such as "security", "exploitable" or the nature of a security vulnerability (overflow, use-after-free…) + - **Security approver’s name** in a commit message. +- The Firefox versions and components affected by the vulnerability. +- Patches with an obvious fix. + +In Bugzilla and other public channels +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In addition to commits, you’ll need to be mindful of not disclosing +sensitive information about the bug in public places, such as Bugzilla: + +- **Do not add public bugs in the “duplicate”, “depends on”, “blocks” + or “see also” section if these bugs could give hints about the nature + of the security issue.** + + - Mention the bugs in comment of the private bug instead. +- Do not comment sensitive information in public related bugs. +- Also be careful about who you give bug access to: **double check + before CC’ing the wrong person or alias**. + +On IRC, Slack channels, GitHub issues, mailing lists: If you need to +discuss about a security bug, use a private channel (protected with a +password or with proper right access management) + +During Development +------------------ + +Testing sec-high and sec-critical bugs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Pushing to Try servers requires Level 1 Commit access but **content +viewing is publicly accessible**. + +As much as possible, **do not push to Try servers**. Testing should be +done locally before checkin in order to prevent public disclosing of the +bug. + +If you need to push to Try servers, make sure your tests don’t disclose +what the vulnerability is about or how to trigger it. Do not mention +anywhere it is security related. + +Obfuscating a security patch +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If your security patch looks obvious because of the code it contains +(e.g. a one-line fix), or if you really need to push to Try servers, +**consider integrating your security-related patch to non-security work +in the same area**. And/or pretend it is related to something else, like +some performance improvement or a correctness fix. **Definitely don't +include the bug number in the commit message.** This will help making +the security issue less easily identifiable. (The absolute ban against +"Security through Obscurity" is in relation to cryptographic systems. In +other situations you still can't *rely* on obscurity but it can +sometimes buy you a little time. In this context we need to get the +fixes into the hands of our users faster than attackers can weaponize +and deploy attacks and a little extra time can help.) + +Requesting sec-approval +~~~~~~~~~~~~~~~~~~~~~~~ + +See :ref:`Security Bug Approval Process` +for more details + +Landing tests +~~~~~~~~~~~~~ + +Tests can be landed **after the release has gone live** and **not before +at least 4 weeks following that release**. + +The exception is if a security issue has never been shipped in a release +build and has been fixed in all development branches. + +Making a security bug public +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is the responsibility of the security management team. + +Essentials +---------- + +- **Do not disclose any information about the vulnerability before a + release with a fix has gone live for enough time for users to update + their software**. + + - This includes code comments, commit messages, tests, public + communication channels. + +- If any doubt: '''request sec-approval? ''' +- If any doubt: **needinfo security folks**. +- **If there’s no rating, assume the worst and treat the bug as + sec-critical**. + +Documentation & Contacts +------------------------ + +- :ref:`Normal process for submitting a patch <How to submit a patch>` +- `How to file a security bug <https://wiki.mozilla.org/Security/Fileabug>`__ +- `Handling Mozilla security bugs (policy) <https://www.mozilla.org/en-US/about/governance/policies/security-group/bugs/>`__ +- `Security Bug Approval Process <security-approval>`__ +- `Contacting the Security team(s) at Mozilla: <https://wiki.mozilla.org/Security>`__ diff --git a/docs/bug-mgmt/processes/labels.rst b/docs/bug-mgmt/processes/labels.rst new file mode 100644 index 0000000000..0af3b6ef03 --- /dev/null +++ b/docs/bug-mgmt/processes/labels.rst @@ -0,0 +1,155 @@ +GitHub Metadata Recommendations +=============================== + +To have better consistency with code and task tracking among Mozilla +Central, Bugzilla, and GitHub, we request that you use a common set of +labels in your projects. Benefits of improved consistency in our +conventions include: + +- Consistency makes measurement of processes simpler across the + organization +- Consistency makes it easier to write re-usable process tools +- Consistency increases clarity for those than need to work across + different repositories and bug trackers +- Consistency reduces friction around engineering mobility between + projects + +We recommend creating sets of labels in your project to do this. + +Bug types +--------- + +In Bugzilla bugs are distinguished by type: ``defect``, ``enhancement``, +and ``tasks``. Use a label to make this distinction in your project. + +Statuses +-------- + +Bugs in GitHub issues have two states: closed and open. Bugzilla has a +richer set of states. + +When you close a bug, add a label indicating `the +resolution <https://wiki.mozilla.org/BMO/UserGuide/BugStatuses#Resolutions>`__. + +- ``fixed`` + + - A change set for the bug has been landed in Mozilla-Central + - A GitHub issue could be closed, but the change set has not + landed so it would be still considered open from the + Bugzilla point of view + +- ``invalid`` + + - The problem described is not a bug. + +- ``incomplete`` + + - The problem is vaguely described with no steps to reproduce, or is + a support request. + +- ``wontfix`` + + - The problem described is a bug which will never be fixed. + +- ``duplicate`` + + - The problem is a duplicate of an existing bug. Be sure to link the + bug this is a duplicate of. + +- ``worksforme`` + + - All attempts at reproducing this bug were futile, and reading the + code produces no clues as to why the described behavior would + occur. + +Severities (Required) +--------------------- + +The triage process for Firefox bugs in Bugzilla requires a non default +value of a bug's :ref:`Severity (definitions) <Defect Severity>`. + +Release Status Flags +------------------------------- + +Open Firefox bugs may also have :ref:`status flags <Release Status Flags>` +(``status_firefoxNN``) set for Nightly, Beta, Release, or ESR. + +Priorities +---------- + +Firefox projects in Bugzilla can use the :ref:`priority field <Priority Definitions>` +to indicate when a bug will be worked on. + +Keywords +-------- + +In GitHub issues metadata is either a label or the bug’s open/closed +state. + +Some Bugzilla metadata behaves like labels, but you need to be careful +with how you use it in order not to confuse QA. + +Regressions +~~~~~~~~~~~ + +In Bugzilla, the ``regression`` keyword indicates a regression in +existing behavior introduced by a code change. + +When a bug is labeled as a regression in GitHub does it imply the +regression is in the code module in GitHub, or the module that’s landed +in Mozilla Central? Using the label ``regression-internal`` will signal +QA that the regression is internal to your development cycle, and not +one introduced into the Mozilla Central tree. + +If it is not clear which pull request caused the regression, add the +``regressionwindow-wanted`` label. + +Other Keywords +~~~~~~~~~~~~~~ + +Other useful labels include ``enhancement`` to distinguish feature +requests, and ``good first issue`` to signal to contributors (`along +with adequate +documentation <http://blog.humphd.org/why-good-first-bugs-often-arent/>`__.) + +Summary +------- + +To represent Bugzilla fields, use labels following this scheme. + +- Bug types + + - ``defect``, ``enhancement``, ``task`` + +- Resolution statuses + + - ``invalid``, ``duplicate``, ``incomplete``, ``worksforme``, + ``wontfix`` + +- Regressions + + - ``regression``, ``regressionwindow-wanted``, + ``regression-internal`` + + +- :ref:`Severity <Defect Severity>` (required) + + - ``S1``, ``S2``, ``S3``, ``S4``, ``N/A`` (reserved for bugs + of type ``task`` or ``enhancement``) + +- :ref:`Status flags <Firefox Status Flags>` + + - ``status_firefoxNN:<status>`` + (example ``status_firefox77:affected``) + +- :ref:`Priority <Priority Definitions>` + + - ``P1``, ``P2``, ``P3``, ``P5`` + +- Other keywords + + - ``good first bug``, ``perf``, &etc. + + +You may already have a set of tags, so do an edit to convert them +or use `the GitHub settings app <https://github.com/probot/settings>`__. diff --git a/docs/bug-mgmt/processes/regressions.rst b/docs/bug-mgmt/processes/regressions.rst new file mode 100644 index 0000000000..991771f38d --- /dev/null +++ b/docs/bug-mgmt/processes/regressions.rst @@ -0,0 +1,64 @@ +How to Mark Regressions +======================= + +Regressions +----------- + +For regression bugs in Mozilla-Central, our policy is to tag the bug as +a regression, identify the commits which caused the regression, then +mark the bugs associated with those commits as causing the regression. + +What is a regression? +--------------------- + +A regression is a bug (in our scheme a ``defect``) introduced by a +`changeset <https://en.wikipedia.org/wiki/Changeset>`__. + +- Bug 101 *fixes* Bug 100 with Change Set A +- Bug 102 *reported which describes previously correct behavior now not + happening* +- Bug 102 *investigated and found to be introduced by Change Set A* + +Marking a Regression Bug +------------------------ + +These things are true about regressions: + +- **Bug Type** is ``defect`` +- **Keywords** include ``regression`` +- **Status_FirefoxNN** is ``affected`` for each version (in current + nightly, beta, and release) of Firefox in which the bug was found +- The bug’s description covers previously working behavior which is no + longer working [ed. I need a better phrase for this] + +Until the change set which caused the regression has been found through +`mozregression <https://mozilla.github.io/mozregression/>`__ or another +bisection tool, the bug should also have the ``regressionwindow-wanted`` +keyword. + +Once the change set which caused the regression has been identified, +remove the ``regressionwindow-wanted`` keyword and set the **Regressed +By** field to the id of the bug associated with the change set. + +Setting the **Regressed By** field will update the **Regresses** field +in the other bug. + +Set a needinfo for the author of the regressing patch asking them to fix +or revert the regression. + +Previous Method +--------------- + +Previously we over-loaded the **Blocks** and **Blocked By** fields to +track the regression, setting **Blocks** to the id of the bug associated +with the change set causing the regression, and using the +``regression``, ``regressionwindow-wanted`` keywords and the status +flags as described above. + +This made it difficult to understand what was a dependency and what was +a regression when looking at dependency trees in Bugzilla. + +FAQs +---- + +*To be written* diff --git a/docs/bug-mgmt/processes/security-approval.rst b/docs/bug-mgmt/processes/security-approval.rst new file mode 100644 index 0000000000..e8148525ab --- /dev/null +++ b/docs/bug-mgmt/processes/security-approval.rst @@ -0,0 +1,219 @@ +Security Bug Approval Process +============================= + +How to fix a core-security bug in Firefox - developer guidelines +---------------------------------------------------------------- + +Follow these security guidelines if you’re involved in reviewing, +testing and landing a security patch: +:ref:`Fixing Security Bugs`. + +Purpose: don't 0-day ourselves +------------------------------ + +People watch our check-ins. They may be able to start exploiting our +users before we can get an update out to them if + +- the patch is an obvious security fix (bounds check, kungFuDeathGrip, + etc.) +- the check-in comment says "security fix" or includes trigger words + like "exploitable", "vulnerable", "overflow", "injection", "use after + free", etc. +- comments in the code mention those types of things or how someone + could abuse the bug +- the check-in contains testcases that show exactly how to trigger the + vulnerability + +Principle: assume the worst +--------------------------- + +- If there's no rating we assume it could be critical +- If we don't know the regression range we assume it needs porting to + all supported branches + +Process for Security Bugs (Developer Perspective) +------------------------------------------------- + +Filing / Managing Bugs +~~~~~~~~~~~~~~~~~~~~~~ + +- Try whenever possible to file security bugs marked as such when + filing, instead of filing them as open bugs and then closing later. + This is not always possible, but attention to this, especially when + filing from crash-stats, is helpful. +- Avoid linking it to non-security bugs with Blocks, Depends, + Regressions, or See Also, especially if those bugs may give a hint to + the sort of security issue involved. Mention the bug in a comment on + the security bug instead. We can always fill in the links later after + the fix has shipped. + +Developing the Patch +~~~~~~~~~~~~~~~~~~~~ + +- Comments in the code should not mention a security issue is being + fixed. Don’t paint a picture or an arrow pointing to security issues + any more than the code changes already do. +- Avoid linking it to non-security bugs with Blocks, Depends, or See + Also, especially if those bugs may give a hint to the sort of + security issue involved. Mention the bug in a comment on the security + bug instead. We can always fill in the links later after the fix has + shipped. +- Do not push to Try servers if possible: this exposes the security + issues for these critical and high rated bugs to public viewing. In + an ideal case, testing of patches is done locally before final + check-in to mozilla-central. +- If pushing to Try servers is necessary, **do not include the bug + number in the patch**. Ideally, do not include tests in the push as + the tests can illustrate the exact nature of the security problem + frequently. +- If you must push to Try servers, with or without tests, try to + obfuscate what this patch is for. Try to push it with other, + non-security work, in the same area. + +Request review of the patch in the same process as normal. After the +patch has received an r+ you will request sec-approval. See +:ref:`Fixing Security Bugs` +for more examples/details of these points. + +On Requesting sec-approval +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For security bugs with no sec- severity rating assume the worst and +follow the rules for sec-critical. During the sec-approval process we +will notice it has not been rated and rate it during the process. + +Core-security bug fixes can be landed by a developer without any +explicit approval if: + +| **A)** The bug has a sec-low, sec-moderate, sec-other, or sec-want + rating. +| **or** +| **B)** The bug is a recent regression on mozilla-central. This means + +- A specific regressing check-in has been identified +- The developer can (**and has**) marked the status flags for ESR, + Beta, and Aurora as "unaffected" +- We have not shipped this vulnerability in anything other than a + nightly build + +If it meets the above criteria, check that patch in. + +Otherwise, if the bug has a patch \*and\* is sec-high or sec-critical, +the developer should prepare the patch for sec-approval. This entails: + +- Commit should occur without specific mention of security, security + bugs, or sec-approvers if possible. While comprehensive commit + messages are generally encouraged; they should be omitted for + security bugs and instead be posted in the bug (which will eventually + become public.) +- Separate out tests into a separate commit. **Do not commit tests when + checking in** when the security bug fix is initially checked-in. + **Remember we don’t want to 0-day ourselves!** + + - Tests should only be checked in later, after an official Firefox + release that contains the fix has gone live and not for at least + four weeks following that release. For example, if Firefox 53 + contains a fix for a security issue that affects the world and is + then fixed in 54, tests for this fix should not be checked in + until four weeks after 54 goes live. The exception to this is if + there is a security issue that hasn’t shipped in a release build + and it is being fixed on multiple development branches (such as + mozilla-central and beta). Since the security problem was never + released to the world, once the bug is fixed in all affected + places, tests can be checked in to the various branches. + - There are two main techniques for remembering to check in the + tests later: + + - clone the sec bug into a hidden "task" bug "land tests for bug + xxxxx" and assign to yourself. It should get a "sec-other" + keyword rating. + - Or, set the "in-testsuite" flag to "?", and later set it to "+" + when the tests get checked in. + +Following that, set the sec-approval flag to '?' on the patch when it is +ready to be checked into mozilla-central (or elsewhere if it is branch +only). + +If developers are unsure about a bug and it has a patch ready, just +request sec-approval anyway and move on. Don't overthink it! + +An automatic nomination comment will be added to bugzilla when +sec-approval is set to '?'. The questions in this need to be filled out +as best as possible when sec-approval is requested for the patch. + +It is as follows (courtesy of Dan Veditz):: + + [Security approval request comment] + How easily can the security issue be deduced from the patch? + Do comments in the patch, the check-in comment, or tests included in + the patch paint a bulls-eye on the security problem? + Which older supported branches are affected by this flaw? + If not all supported branches, which bug introduced the flaw? + Do you have backports for the affected branches? If not, how + different, hard to create, and risky will they be? + How likely is this patch to cause regressions; how much testing does + it need? + +This is similar to the ESR approval nomination form and is meant to help +us evaluate the risks around approving the patch for checkin. + +When the bug is approved for landing, the sec-approval flag will be set +to '+' with a comment from the approver to land the patch. At that +point, land it according to instructions provided.. + +This will allow us to control when we can land security bugs without +exposing them too early and to make sure they get landed on the various +branches. + +If you have any questions or are unsure about anything in this document +contact us on Slack in the #security channel or the current +sec-approvers Dan Veditz and Tom Ritter. + +Process for Security Bugs (sec-approver Perspective) +---------------------------------------------------- + +The security assurance team and release management will have their own +process for approving bugs: + +#. The Security assurance team goes through sec-approval ? bugs daily + and approves low risk fixes for central (if early in cycle). + Developers can also ping the Security Assurance Team (specifically + Tom Ritter & Dan Veditz) in #security on Slack when important. + + #. If a bug lacks a security-rating one should be assigned - possibly + in coordination with the (other member of) the Security Assurance + Team + +#. Security team marks tracking flags to ? for all affected versions + when approved for central. (This allows release management to decide + whether to uplift to branches just like always.) +#. Weekly security/release management triage meeting goes through + sec-approval + and ? bugs where beta and ESR is affected, ? bugs with + higher risk (sec-high and sec-critical), or ? bugs near end of cycle. + +Options for sec-approval including a logical combination of the +following: + +- Separate out the test and comments in the code into a followup commit + we will commit later. +- Remove the commit message and place it in the bug or comments in a + followup commit. +- Please land it bundled in with another commit +- Land today +- Land today, land the tests after +- Land closer to the release date +- Land in Nightly to assess stability +- Land today and request uplift to all branches +- Request uplift to all branches and we'll land as close to shipping as + permitted +- Chemspill time + +The decision process for which of these to choose is perceived risk on +multiple axes: + +- ease of exploitation +- reverse engineering risk +- stability risk + +The most common choice is: not much stability risk, not an immediate RE +risk, moderate to high difficulty of exploitation: "land whenever"
\ No newline at end of file diff --git a/docs/bug-mgmt/processes/shared-bug-queues.rst b/docs/bug-mgmt/processes/shared-bug-queues.rst new file mode 100644 index 0000000000..dc2df9bbf9 --- /dev/null +++ b/docs/bug-mgmt/processes/shared-bug-queues.rst @@ -0,0 +1,34 @@ +Shared Bug Queues +================= + +Reviewers for change sets can be suggested at the product and component +level, but only the person who has been asked to review code will be +notified. + +Realizing that Bugzilla users can *watch* other users, `Chris +Cooper <https://mozillians.org/en-US/u/coop/>`__ came up with the idea +of having `a shared reviews alias for review +requests <http://coopcoopbware.tumblr.com/post/170952242320/experiments-in-productivity-the-shared-bug-queue>`__. + +If you want to watch a particular part of the tree in Mozilla Central, +then `use the Herald +tool <https://phabricator.services.mozilla.com/book/phabricator/article/herald/>`__. + +Process +------- + +1. Create a new bugzilla.mozilla.com account for an address which can + receive mail. + Use the ``name+extension@domain.tld`` trick such as + ``jmozillian+reviews@mozilla.com`` to create a unique address +2. Respond to the email sent by Bugzilla and set a password on the + account +3. `Open a bug <https://mzl.la/2Mg8Sli>`__ to convert the account to a + bot and make it the shared review queue for your component +4. BMO administrator updates the email address of the new account to the + ``@mozilla.bugs`` address +5. BMO administrator updates the default reviewer for the component + requested and sets it to the shared review account +6. Reviewers `follow the shared review account in + bugzilla <https://bugzilla.mozilla.org/userprefs.cgi?tab=email>`__ +7. Reviewers get notified when shared review account is ``r?``\ ed diff --git a/docs/code-quality/coding-style/coding_style_cpp.rst b/docs/code-quality/coding-style/coding_style_cpp.rst new file mode 100644 index 0000000000..09319cfd12 --- /dev/null +++ b/docs/code-quality/coding-style/coding_style_cpp.rst @@ -0,0 +1,867 @@ +================ +C++ Coding style +================ + + +This document attempts to explain the basic styles and patterns used in +the Mozilla codebase. New code should try to conform to these standards, +so it is as easy to maintain as existing code. There are exceptions, but +it's still important to know the rules! + +This article is particularly for those new to the Mozilla codebase, and +in the process of getting their code reviewed. Before requesting a +review, please read over this document, making sure that your code +conforms to recommendations. + +.. container:: blockIndicator warning + + The Firefox code base adopts parts of the `Google Coding style for C++ + code <https://google.github.io/styleguide/cppguide.html>`__, but not all of its rules. + A few rules are followed across the code base, others are intended to be + followed in new or significantly revised code. We may extend this list in the + future, when we evaluate the Google Coding Style for C++ Code further and/or update + our coding practices. However, the plan is not to adopt all rules of the Google Coding + Style for C++ Code. Some rules are explicitly unlikely to be adopted at any time. + + Followed across the code base: + + - `Formatting <https://google.github.io/styleguide/cppguide.html#Formatting>`__, + except for subsections noted here otherwise + - `Implicit Conversions <https://google.github.io/styleguide/cppguide.html#Implicit_Conversions>`__, + which is enforced by a custom clang-plugin check, unless explicitly overridden using + ``MOZ_IMPLICIT`` + + Followed in new/significantly revised code: + + - `Include guards <https://google.github.io/styleguide/cppguide.html#The__define_Guard>`__ + + Unlikely to be ever adopted: + + - `Forward declarations <https://google.github.io/styleguide/cppguide.html#Forward_Declarations>`__ + - `Formatting/Conditionals <https://google.github.io/styleguide/cppguide.html#Conditionals>`__ + w.r.t. curly braces around inner statements, we require them in all cases where the + Google style allows to leave them out for single-line conditional statements + + This list reflects the state of the Google Google Coding Style for C++ Code as of + 2020-07-17. It may become invalid when the Google modifies its Coding Style. + + +Formatting code +--------------- + +Formatting is done automatically via clang-format, and controlled via in-tree +configuration files. See :ref:`Formatting C++ Code With clang-format` +for more information. + +Unix-style linebreaks (``\n``), not Windows-style (``\r\n``). You can +convert patches, with DOS newlines to Unix via the ``dos2unix`` utility, +or your favorite text editor. + +Static analysis +--------------- + +Several of the rules in the Google C++ coding styles and the additions mentioned below +can be checked via clang-tidy (some rules are from the upstream clang-tidy, some are +provided via a mozilla-specific plugin). Some of these checks also allow fixes to +be automatically applied. + +``mach static-analysis`` provides a convenient way to run these checks. For example, +for the check called ``google-readability-braces-around-statements``, you can run: + +.. code-block:: shell + + ./mach static-analysis check --checks="-*,google-readability-braces-around-statements" --fix <file> + +It may be necessary to reformat the files after automatically applying fixes, see +:ref:`Formatting C++ Code With clang-format`. + +Additional rules +---------------- + +*The norms in this section should be followed for new code. For existing code, +use the prevailing style in a file or module, ask the owner if you are +in another team's codebase or it's not clear what style to use.* + + + + +Control structures +~~~~~~~~~~~~~~~~~~ + +Always brace controlled statements, even a single-line consequent of +``if else else``. This is redundant, typically, but it avoids dangling +else bugs, so it's safer at scale than fine-tuning. + +Examples: + +.. code-block:: cpp + + if (...) { + } else if (...) { + } else { + } + + while (...) { + } + + do { + } while (...); + + for (...; ...; ...) { + } + + switch (...) { + case 1: { + // When you need to declare a variable in a switch, put the block in braces. + int var; + break; + } + case 2: + ... + break; + default: + break; + } + +``else`` should only ever be followed by ``{`` or ``if``; i.e., other +control keywords are not allowed and should be placed inside braces. + +.. note:: + + For this rule, clang-tidy provides the ``google-readability-braces-around-statements`` + check with autofixes. + + +C++ namespaces +~~~~~~~~~~~~~~ + +Mozilla project C++ declarations should be in the ``mozilla`` +namespace. Modules should avoid adding nested namespaces under +``mozilla``, unless they are meant to contain names which have a high +probability of colliding with other names in the code base. For example, +``Point``, ``Path``, etc. Such symbols can be put under +module-specific namespaces, under ``mozilla``, with short +all-lowercase names. Other global namespaces besides ``mozilla`` are +not allowed. + +No ``using`` directives are allowed in header files, except inside class +definitions or functions. (We don't want to pollute the global scope of +compilation units that use the header file.) + +.. note:: + + For parts of this rule, clang-tidy provides the ``google-global-names-in-headers`` + check. It only detects ``using namespace`` directives in the global namespace. + + +``using namespace ...;`` is only allowed in ``.cpp`` files after all +``#include``\ s. Prefer to wrap code in ``namespace ... { ... };`` +instead, if possible. ``using namespace ...;``\ should always specify +the fully qualified namespace. That is, to use ``Foo::Bar`` do not +write ``using namespace Foo; using namespace Bar;``, write +``using namespace Foo::Bar;`` + +Use nested namespaces (ex: ``namespace mozilla::widget {`` + +.. note:: + + clang-tidy provides the ``modernize-concat-nested-namespaces`` + check with autofixes. + + +Anonymous namespaces +~~~~~~~~~~~~~~~~~~~~ + +We prefer using ``static``, instead of anonymous C++ namespaces. This may +change once there is better debugger support (especially on Windows) for +placing breakpoints, etc. on code in anonymous namespaces. You may still +use anonymous namespaces for things that can't be hidden with ``static``, +such as types, or certain objects which need to be passed to template +functions. + + +C++ classes +~~~~~~~~~~~~ + +.. code-block:: cpp + + namespace mozilla { + + class MyClass : public A + { + ... + }; + + class MyClass + : public X + , public Y + { + public: + MyClass(int aVar, int aVar2) + : mVar(aVar) + , mVar2(aVar2) + { + ... + } + + // Special member functions, like constructors, that have default bodies + // should use '= default' annotation instead. + MyClass() = default; + + // Unless it's a copy or move constructor or you have a specific reason to allow + // implicit conversions, mark all single-argument constructors explicit. + explicit MyClass(OtherClass aArg) + { + ... + } + + // This constructor can also take a single argument, so it also needs to be marked + // explicit. + explicit MyClass(OtherClass aArg, AnotherClass aArg2 = AnotherClass()) + { + ... + } + + int LargerFunction() + { + ... + ... + } + + private: + int mVar; + }; + + } // namespace mozilla + +Define classes using the style given above. + +.. note:: + + For the rule on ``= default``, clang-tidy provides the ``modernize-use-default`` + check with autofixes. + + For the rule on explicit constructors and conversion operators, clang-tidy + provides the ``mozilla-implicit-constructor`` check. + +Existing classes in the global namespace are named with a short prefix +(For example, ``ns``) as a pseudo-namespace. + + +Methods and functions +~~~~~~~~~~~~~~~~~~~~~ + + +C/C++ +^^^^^ + +In C/C++, method names should use ``UpperCamelCase``. + +Getters that never fail, and never return null, are named ``Foo()``, +while all other getters use ``GetFoo()``. Getters can return an object +value, via a ``Foo** aResult`` outparam (typical for an XPCOM getter), +or as an ``already_AddRefed<Foo>`` (typical for a WebIDL getter, +possibly with an ``ErrorResult& rv`` parameter), or occasionally as a +``Foo*`` (typical for an internal getter for an object with a known +lifetime). See `the bug 223255 <https://bugzilla.mozilla.org/show_bug.cgi?id=223255>`_ +for more information. + +XPCOM getters always return primitive values via an outparam, while +other getters normally use a return value. + +Method declarations must use, at most, one of the following keywords: +``virtual``, ``override``, or ``final``. Use ``virtual`` to declare +virtual methods, which do not override a base class method with the same +signature. Use ``override`` to declare virtual methods which do +override a base class method, with the same signature, but can be +further overridden in derived classes. Use ``final`` to declare virtual +methods which do override a base class method, with the same signature, +but can NOT be further overridden in the derived classes. This should +help the person reading the code fully understand what the declaration +is doing, without needing to further examine base classes. + +.. note:: + + For the rule on ``virtual/override/final``, clang-tidy provides the + ``modernize-use-override`` check with autofixes. + + +Operators +~~~~~~~~~ + +The unary keyword operator ``sizeof``, should have its operand parenthesized +even if it is an expression; e.g. ``int8_t arr[64]; memset(arr, 42, sizeof(arr));``. + + +Literals +~~~~~~~~ + +Use ``\uXXXX`` unicode escapes for non-ASCII characters. The character +set for XUL, DTD, script, and properties files is UTF-8, which is not easily +readable. + + +Prefixes +~~~~~~~~ + +Follow these naming prefix conventions: + + +Variable prefixes +^^^^^^^^^^^^^^^^^ + +- k=constant (e.g. ``kNC_child``). Not all code uses this style; some + uses ``ALL_CAPS`` for constants. +- g=global (e.g. ``gPrefService``) +- a=argument (e.g. ``aCount``) +- C++ Specific Prefixes + + - s=static member (e.g. ``sPrefChecked``) + - m=member (e.g. ``mLength``) + - e=enum variants (e.g. ``enum Foo { eBar, eBaz }``). Enum classes + should use ``CamelCase`` instead (e.g. + ``enum class Foo { Bar, Baz }``). + + +Global functions/macros/etc +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Macros begin with ``MOZ_``, and are all caps (e.g. + ``MOZ_WOW_GOODNESS``). Note that older code uses the ``NS_`` prefix; + while these aren't being changed, you should only use ``MOZ_`` for + new macros. The only exception is if you're creating a new macro, + which is part of a set of related macros still using the old ``NS_`` + prefix. Then you should be consistent with the existing macros. + + +Error Variables +^^^^^^^^^^^^^^^ + +- Local variables that are assigned ``nsresult`` result codes should be named ``rv`` + (i.e., e.g., not ``res``, not ``result``, not ``foo``). `rv` should not be + used for bool or other result types. +- Local variables that are assigned ``bool`` result codes should be named `ok`. + + +C/C++ practices +--------------- + +- **Have you checked for compiler warnings?** Warnings often point to + real bugs. `Many of them <https://searchfox.org/mozilla-central/source/build/moz.configure/warnings.configure>`__ + are enabled by default in the build system. +- In C++ code, use ``nullptr`` for pointers. In C code, using ``NULL`` + or ``0`` is allowed. + +.. note:: + + For the C++ rule, clang-tidy provides the ``modernize-use-nullptr`` check + with autofixes. + +- Don't use ``PRBool`` and ``PRPackedBool`` in C++, use ``bool`` + instead. +- For checking if a ``std`` container has no items, don't use + ``size()``, instead use ``empty()``. +- When testing a pointer, use ``(!myPtr)`` or ``(myPtr)``; + don't use ``myPtr != nullptr`` or ``myPtr == nullptr``. +- Do not compare ``x == true`` or ``x == false``. Use ``(x)`` or + ``(!x)`` instead. ``if (x == true)`` may have semantics different from + ``if (x)``! + +.. note:: + + clang-tidy provides the ``readability-simplify-boolean-expr`` check + with autofixes that checks for these and some other boolean expressions + that can be simplified. + +- In general, initialize variables with ``nsFoo aFoo = bFoo,`` and not + ``nsFoo aFoo(bFoo)``. + + - For constructors, initialize member variables with : ``nsFoo + aFoo(bFoo)`` syntax. + +- To avoid warnings created by variables used only in debug builds, use + the + `DebugOnly<T> <https://developer.mozilla.org/docs/Mozilla/Debugging/DebugOnly%3CT%3E>`__ + helper when declaring them. +- You should `use the static preference + API <https://developer.mozilla.org/docs/Mozilla/Preferences/Using_preferences_from_application_code>`__ for + working with preferences. +- One-argument constructors, that are not copy or move constructors, + should generally be marked explicit. Exceptions should be annotated + with ``MOZ_IMPLICIT``. +- Use ``char32_t`` as the return type or argument type of a method that + returns or takes as argument a single Unicode scalar value. (Don't + use UTF-32 strings, though.) +- Forward-declare classes in your header files, instead of including + them, whenever possible. For example, if you have an interface with a + ``void DoSomething(nsIContent* aContent)`` function, forward-declare + with ``class nsIContent;`` instead of ``#include "nsIContent.h"`` +- Include guards are named per the Google coding style and should not + include a leading ``MOZ_`` or ``MOZILLA_``. For example + ``dom/media/foo.h`` would use the guard ``DOM_MEDIA_FOO_H_``. +- Avoid the usage of ``typedef``, instead, please use ``using`` instead. + +.. note:: + + For parts of this rule, clang-tidy provides the ``modernize-use-using`` + check with autofixes. + + +COM and pointers +---------------- + +- Use ``nsCOMPtr<>`` + If you don't know how to use it, start looking in the code for + examples. The general rule, is that the very act of typing + ``NS_RELEASE`` should be a signal to you to question your code: + "Should I be using ``nsCOMPtr`` here?". Generally the only valid use + of ``NS_RELEASE`` is when you are storing refcounted pointers in a + long-lived datastructure. +- Declare new XPCOM interfaces using `XPIDL <https://developer.mozilla.org/docs/Mozilla/Tech/XPIDL>`__, so they + will be scriptable. +- Use `nsCOMPtr <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Reference/Glue_classes/nsCOMPtr>`__ for strong references, and + `nsWeakPtr <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Weak_reference>`__ for weak references. +- Don't use ``QueryInterface`` directly. Use ``CallQueryInterface`` or + ``do_QueryInterface`` instead. +- Use `Contract + IDs <news://news.mozilla.org/3994AE3E.D96EF810@netscape.com>`__, + instead of CIDs with ``do_CreateInstance``/``do_GetService``. +- Use pointers, instead of references for function out parameters, even + for primitive types. + + +IDL +--- + + +Use leading-lowercase, or "interCaps" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When defining a method or attribute in IDL, the first letter should be +lowercase, and each following word should be capitalized. For example: + +.. code-block:: cpp + + long updateStatusBar(); + + +Use attributes wherever possible +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Whenever you are retrieving or setting a single value, without any +context, you should use attributes. Don't use two methods when you could +use an attribute. Using attributes logically connects the getting and +setting of a value, and makes scripted code look cleaner. + +This example has too many methods: + +.. code-block:: cpp + + interface nsIFoo : nsISupports + { + long getLength(); + void setLength(in long length); + long getColor(); + }; + +The code below will generate the exact same C++ signature, but is more +script-friendly. + +.. code-block:: cpp + + interface nsIFoo : nsISupports + { + attribute long length; + readonly attribute long color; + }; + + +Use Java-style constants +~~~~~~~~~~~~~~~~~~~~~~~~ + +When defining scriptable constants in IDL, the name should be all +uppercase, with underscores between words: + +.. code-block:: cpp + + const long ERROR_UNDEFINED_VARIABLE = 1; + + +See also +~~~~~~~~ + +For details on interface development, as well as more detailed style +guides, see the `Interface development +guide <https://developer.mozilla.org/docs/Mozilla/Developer_guide/Interface_development_guide>`__. + + +Error handling +-------------- + + +Check for errors early and often +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Every time you make a call into an XPCOM function, you should check for +an error condition. You need to do this even if you know that call will +never fail. Why? + +- Someone may change the callee in the future to return a failure + condition. +- The object in question may live on another thread, another process, + or possibly even another machine. The proxy could have failed to make + your call in the first place. + +Also, when you make a new function which is failable (i.e. it will +return a ``nsresult`` or a ``bool`` that may indicate an error), you should +explicitly mark the return value should always be checked. For example: + +:: + + // for IDL. + [must_use] nsISupports + create(); + + // for C++, add this in *declaration*, do not add it again in implementation. + [[nodiscard]] nsresult + DoSomething(); + +There are some exceptions: + +- Predicates or getters, which return ``bool`` or ``nsresult``. +- IPC method implementation (For example, ``bool RecvSomeMessage()``). +- Most callers will check the output parameter, see below. + +.. code-block:: cpp + + nsresult + SomeMap::GetValue(const nsString& key, nsString& value); + +If most callers need to check the output value first, then adding +``[[nodiscard]]`` might be too verbose. In this case, change the return value +to void might be a reasonable choice. + +There is also a static analysis attribute ``MOZ_MUST_USE_TYPE``, which can +be added to class declarations, to ensure that those declarations are +always used when they are returned. + + +Use the NS_WARN_IF macro when errors are unexpected. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``NS_WARN_IF`` macro can be used to issue a console warning, in debug +builds if the condition fails. This should only be used when the failure +is unexpected and cannot be caused by normal web content. + +If you are writing code which wants to issue warnings when methods fail, +please either use ``NS_WARNING`` directly, or use the new ``NS_WARN_IF`` macro. + +.. code-block:: cpp + + if (NS_WARN_IF(somethingthatshouldbefalse)) { + return NS_ERROR_INVALID_ARG; + } + + if (NS_WARN_IF(NS_FAILED(rv))) { + return rv; + } + +Previously, the ``NS_ENSURE_*`` macros were used for this purpose, but +those macros hide return statements, and should not be used in new code. +(This coding style rule isn't generally agreed, so use of ``NS_ENSURE_*`` +can be valid.) + + +Return from errors immediately +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In most cases, your knee-jerk reaction should be to return from the +current function, when an error condition occurs. Don't do this: + +.. code-block:: cpp + + rv = foo->Call1(); + if (NS_SUCCEEDED(rv)) { + rv = foo->Call2(); + if (NS_SUCCEEDED(rv)) { + rv = foo->Call3(); + } + } + return rv; + +Instead, do this: + +.. code-block:: cpp + + rv = foo->Call1(); + if (NS_FAILED(rv)) { + return rv; + } + + rv = foo->Call2(); + if (NS_FAILED(rv)) { + return rv; + } + + rv = foo->Call3(); + if (NS_FAILED(rv)) { + return rv; + } + +Why? Error handling should not obfuscate the logic of the code. The +author's intent, in the first example, was to make 3 calls in +succession. Wrapping the calls in nested if() statements, instead +obscured the most likely behavior of the code. + +Consider a more complicated example to hide a bug: + +.. code-block:: cpp + + bool val; + rv = foo->GetBooleanValue(&val); + if (NS_SUCCEEDED(rv) && val) { + foo->Call1(); + } else { + foo->Call2(); + } + +The intent of the author, may have been, that ``foo->Call2()`` would only +happen when val had a false value. In fact, ``foo->Call2()`` will also be +called, when ``foo->GetBooleanValue(&val)`` fails. This may, or may not, +have been the author's intent. It is not clear from this code. Here is +an updated version: + +.. code-block:: cpp + + bool val; + rv = foo->GetBooleanValue(&val); + if (NS_FAILED(rv)) { + return rv; + } + if (val) { + foo->Call1(); + } else { + foo->Call2(); + } + +In this example, the author's intent is clear, and an error condition +avoids both calls to ``foo->Call1()`` and ``foo->Call2();`` + +*Possible exceptions:* Sometimes it is not fatal if a call fails. For +instance, if you are notifying a series of observers that an event has +fired, it might be trivial that one of these notifications failed: + +.. code-block:: cpp + + for (size_t i = 0; i < length; ++i) { + // we don't care if any individual observer fails + observers[i]->Observe(foo, bar, baz); + } + +Another possibility, is you are not sure if a component exists or is +installed, and you wish to continue normally, if the component is not +found. + +.. code-block:: cpp + + nsCOMPtr<nsIMyService> service = do_CreateInstance(NS_MYSERVICE_CID, &rv); + // if the service is installed, then we'll use it. + if (NS_SUCCEEDED(rv)) { + // non-fatal if this fails too, ignore this error. + service->DoSomething(); + + // this is important, handle this error! + rv = service->DoSomethingImportant(); + if (NS_FAILED(rv)) { + return rv; + } + } + + // continue normally whether or not the service exists. + + +Strings +------- + +.. note:: + + This section overlaps with the more verbose advice given in + `Internal strings <https://wiki.developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Guide/Internal_strings>`__. + These should eventually be merged. For now, please refer to that guide for + more advice. + +- String arguments to functions should be declared as ``[const] nsA[C]String&``. +- Prefer using string literals. In particular, use empty string literals, + i.e. ``u""_ns`` or ``""_ns``, instead of ``Empty[C]String()`` or + ``const nsAuto[C]String empty;``. Use ``Empty[C]String()`` only if you + specifically need a ``const ns[C]String&``, e.g. with the ternary operator + or when you need to return/bind to a reference or take the address of the + empty string. +- For 16-bit literal strings, use ``u"..."_ns`` or, if necessary + ``NS_LITERAL_STRING_FROM_CSTRING(...)`` instead of ``nsAutoString()`` + or other ways that would do a run-time conversion. + See :ref:`Avoid runtime conversion of string literals` below. +- To compare a string with a literal, use ``.EqualsLiteral("...")``. +- Use ``str.IsEmpty()`` instead of ``str.Length() == 0``. +- Use ``str.Truncate()`` instead of ``str.SetLength(0)``, + ``str.Assign(""_ns)`` or ``str.AssignLiteral("")``. +- Don't use functions from ``ctype.h`` (``isdigit()``, ``isalpha()``, + etc.) or from ``strings.h`` (``strcasecmp()``, ``strncasecmp()``). + These are locale-sensitive, which makes them inappropriate for + processing protocol text. At the same time, they are too limited to + work properly for processing natural-language text. Use the + alternatives in ``mozilla/TextUtils.h`` and in ``nsUnicharUtils.h`` + in place of ``ctype.h``. In place of ``strings.h``, prefer the + ``nsStringComparator`` facilities for comparing strings or if you + have to work with zero-terminated strings, use ``nsCRT.h`` for + ASCII-case-insensitive comparison. + + +Use the ``Auto`` form of strings for local values +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When declaring a local, short-lived ``nsString`` class, always use +``nsAutoString`` or ``nsAutoCString``. These pre-allocate a 64-byte +buffer on the stack, and avoid fragmenting the heap. Don't do this: + +.. code-block:: cpp + + nsresult + foo() + { + nsCString bar; + .. + } + +instead: + +.. code-block:: cpp + + nsresult + foo() + { + nsAutoCString bar; + .. + } + + +Be wary of leaking values from non-XPCOM functions that return char\* or PRUnichar\* +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is an easy trap to return an allocated string, from an internal +helper function, and then using that function inline in your code, +without freeing the value. Consider this code: + +.. code-block:: cpp + + static char* + GetStringValue() + { + .. + return resultString.ToNewCString(); + } + + .. + WarnUser(GetStringValue()); + +In the above example, ``WarnUser`` will get the string allocated from +``resultString.ToNewCString()`` and throw away the pointer. The +resulting value is never freed. Instead, either use the string classes, +to make sure your string is automatically freed when it goes out of +scope, or make sure that your string is freed. + +Automatic cleanup: + +.. code-block:: cpp + + static void + GetStringValue(nsAWritableCString& aResult) + { + .. + aResult.Assign("resulting string"); + } + + .. + nsAutoCString warning; + GetStringValue(warning); + WarnUser(warning.get()); + +Free the string manually: + +.. code-block:: cpp + + static char* + GetStringValue() + { + .. + return resultString.ToNewCString(); + } + + .. + char* warning = GetStringValue(); + WarnUser(warning); + nsMemory::Free(warning); + + +Avoid runtime conversion of string literals +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is very common to need to assign the value of a literal string, such +as ``"Some String"``, into a unicode buffer. Instead of using ``nsString``'s +``AssignLiteral`` and ``AppendLiteral``, use a user-defined literal like `u"foo"_ns` +instead. On most platforms, this will force the compiler to compile in a +raw unicode string, and assign it directly. In cases where the literal is defined +via a macro that is used in both 8-bit and 16-bit ways, you can use +`NS_LITERAL_STRING_FROM_CSTRING` to do the conversion at compile time. + +Incorrect: + +.. code-block:: cpp + + nsAutoString warning; + warning.AssignLiteral("danger will robinson!"); + ... + foo->SetStringValue(warning); + ... + bar->SetUnicodeValue(warning.get()); + +Correct: + +.. code-block:: cpp + + constexpr auto warning = u"danger will robinson!"_ns; + ... + // if you'll be using the 'warning' string, you can still use it as before: + foo->SetStringValue(warning); + ... + bar->SetUnicodeValue(warning.get()); + + // alternatively, use the wide string directly: + foo->SetStringValue(u"danger will robinson!"_ns); + ... + + // if a macro is the source of a 8-bit literal and you cannot change it, use + // NS_LITERAL_STRING_FROM_CSTRING, but only if necessary. + #define MY_MACRO_LITERAL "danger will robinson!" + foo->SetStringValue(NS_LITERAL_STRING_FROM_CSTRING(MY_MACRO_LITERAL)); + + // If you need to pass to a raw const char16_t *, there's no benefit to + // go through our string classes at all, just do... + bar->SetUnicodeValue(u"danger will robinson!"); + + // .. or, again, if a macro is the source of a 8-bit literal + bar->SetUnicodeValue(u"" MY_MACRO_LITERAL); + + +Usage of PR_(MAX|MIN|ABS|ROUNDUP) macro calls +--------------------------------------------- + +Use the standard-library functions (``std::max``), instead of +``PR_(MAX|MIN|ABS|ROUNDUP)``. + +Use ``mozilla::Abs`` instead of ``PR_ABS``. All ``PR_ABS`` calls in C++ code have +been replaced with ``mozilla::Abs`` calls, in `bug +847480 <https://bugzilla.mozilla.org/show_bug.cgi?id=847480>`__. All new +code in ``Firefox/core/toolkit`` needs to ``#include "nsAlgorithm.h"`` and +use the ``NS_foo`` variants instead of ``PR_foo``, or +``#include "mozilla/MathAlgorithms.h"`` for ``mozilla::Abs``. diff --git a/docs/code-quality/coding-style/coding_style_general.rst b/docs/code-quality/coding-style/coding_style_general.rst new file mode 100644 index 0000000000..950cd6ccd3 --- /dev/null +++ b/docs/code-quality/coding-style/coding_style_general.rst @@ -0,0 +1,18 @@ + +Mode line +~~~~~~~~~ + +Files should have Emacs and vim mode line comments as the first two +lines of the file, which should set ``indent-tabs-mode`` to ``nil``. For new +files, use the following, specifying two-space indentation: + +.. code-block:: cpp + + /* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */ + /* vim: set ts=2 et sw=2 tw=80: */ + /* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at https://mozilla.org/MPL/2.0/. */ + +Be sure to use the correct ``Mode`` in the first line, don't use ``C++`` in +JavaScript files. diff --git a/docs/code-quality/coding-style/coding_style_java.rst b/docs/code-quality/coding-style/coding_style_java.rst new file mode 100644 index 0000000000..f2206d8e2d --- /dev/null +++ b/docs/code-quality/coding-style/coding_style_java.rst @@ -0,0 +1,68 @@ +================= +Java Coding style +================= + +- We use the `Java Coding + Style <https://www.oracle.com/technetwork/java/codeconvtoc-136057.html>`__. + Quick summary: + + - FirstLetterUpperCase for class names. + - camelCase for method and variable names. + - One declaration per line: + + .. code-block:: java + + int x, y; // this is BAD! + int a; // split it over + int b; // two lines + +- Braces should be placed like so (generally, opening braces on same + line, closing braces on a new line): + + .. code-block:: java + + public void func(int arg) { + if (arg != 0) { + while (arg > 0) { + arg--; + } + } else { + arg++; + } + } + +- Places we differ from the Java coding style: + + - Start class variable names with 'm' prefix (e.g. + mSomeClassVariable) and static variables with 's' prefix (e.g. + sSomeStaticVariable) + - ``import`` statements: + + - Do not use wildcard imports like \`import java.util.*;\` + - Organize imports by blocks separated by empty line: + org.mozilla.*, android.*, com.*, net.*, org.*, then java.\* + This is basically what Android Studio does by default, except + that we place org.mozilla.\* at the front - please adjust + Settings -> Editor -> Code Style -> Java -> Imports + accordingly. + - Within each import block, alphabetize import names with + uppercase before lowercase. For example, ``com.example.Foo`` is + before ``com.example.bar`` + + - 4-space indents. + - Spaces, not tabs. + - Don't restrict yourself to 80-character lines. Google's Android + style guide suggests 100-character lines, which is also the + default setting in Android Studio. Java code tends to be long + horizontally, so use appropriate judgement when wrapping. Avoid + deep indents on wrapping. Note that aligning the wrapped part of a + line, with some previous part of the line (rather than just using + a fixed indent), may require shifting the code every time the line + changes, resulting in spurious whitespace changes. + +- For additional specifics on Firefox for Android, see the `Coding + Style guide for Firefox on + Android <https://wiki.mozilla.org/Mobile/Fennec/Android#Coding_Style>`__. +- The `Android Coding + Style <https://source.android.com/source/code-style.html>`__ has some + useful guidelines too. diff --git a/docs/code-quality/coding-style/coding_style_js.rst b/docs/code-quality/coding-style/coding_style_js.rst new file mode 100644 index 0000000000..46ec1dd8a6 --- /dev/null +++ b/docs/code-quality/coding-style/coding_style_js.rst @@ -0,0 +1,150 @@ +======================= +JavaScript Coding style +======================= + +Coding style +~~~~~~~~~~~~ + +`prettier <https://prettier.io/>`_ is the tool used to reformat the JavaScript code. + + +Methods and functions +~~~~~~~~~~~~~~~~~~~~~ + +In JavaScript, functions should use camelCase, but should not capitalize +the first letter. Methods should not use the named function expression +syntax, because our tools understand method names: + +.. code-block:: cpp + + doSomething: function (aFoo, aBar) { + ... + } + +In-line functions should have spaces around braces, except before commas +or semicolons: + +.. code-block:: cpp + + function valueObject(aValue) { return { value: aValue }; } + + +JavaScript objects +~~~~~~~~~~~~~~~~~~ + +.. code-block:: cpp + + var foo = { prop1: "value1" }; + + var bar = { + prop1: "value1", + prop2: "value2" + }; + +Constructors for objects should be capitalized and use Pascal Case: + +.. code-block:: cpp + + function ObjectConstructor() { + this.foo = "bar"; + } + + +Operators +~~~~~~~~~ + +In JavaScript, overlong expressions not joined by ``&&`` and +``||`` should break so the operator starts on the second line and +starting in the same column as the beginning of the expression in the +first line. This applies to ``?:``, binary arithmetic operators +including ``+``, and member-of operators. Rationale: an operator at the +front of the continuation line makes for faster visual scanning, as +there is no need to read to the end of line. Also there exists a +context-sensitive keyword hazard in JavaScript; see {{bug(442099, "bug", +19)}}, which can be avoided by putting . at the start of a continuation +line, in long member expression. + +In JavaScript, ``==`` is preferred to ``===``. + +Unary keyword operators, such as ``typeof``, should have their operand +parenthesized; e.g. ``typeof("foo") == "string"``. + +Literals +~~~~~~~~ + +Double-quoted strings (e.g. ``"foo"``) are preferred to single-quoted +strings (e.g. ``'foo'``), in JavaScript, except to avoid escaping +embedded double quotes, or when assigning inline event handlers. + + +Prefixes +~~~~~~~~ + +- k=constant (e.g. ``kNC_child``). Not all code uses this style; some + uses ``ALL_CAPS`` for constants. +- g=global (e.g. ``gPrefService``) +- a=argument (e.g. ``aCount``) + +- JavaScript Specific Prefixes + + - \_=member (variable or function) (e.g. ``_length`` or + ``_setType(aType)``) + - k=enumeration value (e.g. ``const kDisplayModeNormal = 0``) + - on=event handler (e.g. ``function onLoad()``) + - Convenience constants for interface names should be prefixed with + ``nsI``: + + .. code-block:: javascript + + const nsISupports = Components.interfaces.nsISupports; + const nsIWBN = Components.interfaces.nsIWebBrowserNavigation; + + + +Other advices +~~~~~~~~~~~~~ + +- Make sure you are aware of the `JavaScript + Tips <https://developer.mozilla.org/docs/Mozilla/JavaScript_Tips>`__. +- Do not compare ``x == true`` or ``x == false``. Use ``(x)`` or + ``(!x)`` instead. ``x == true``, is certainly different from if + ``(x)``! Compare objects to ``null``, numbers to ``0`` or strings to + ``""``, if there is chance for confusion. +- Make sure that your code doesn't generate any strict JavaScript + warnings, such as: + + - Duplicate variable declaration. + - Mixing ``return;`` with ``return value;`` + - Undeclared variables or members. If you are unsure if an array + value exists, compare the index to the array's length. If you are + unsure if an object member exists, use ``"name"`` in ``aObject``, + or if you are expecting a particular type you may use + ``typeof(aObject.name) == "function"`` (or whichever type you are + expecting). + +- Use ``['value1, value2']`` to create a JavaScript array in preference + to using + ``new {{JSxRef("Array", "Array", "Syntax", 1)}}(value1, value2)`` + which can be confusing, as ``new Array(length)`` will actually create + a physically empty array with the given logical length, while + ``[value]`` will always create a 1-element array. You cannot actually + guarantee to be able to preallocate memory for an array. +- Use ``{ member: value, ... }`` to create a JavaScript object; a + useful advantage over ``new {{JSxRef("Object", "Object", "", 1)}}()`` + is the ability to create initial properties and use extended + JavaScript syntax, to define getters and setters. +- If having defined a constructor you need to assign default + properties, it is preferred to assign an object literal to the + prototype property. +- Use regular expressions, but use wisely. For instance, to check that + ``aString`` is not completely whitespace use + ``/\S/.{{JSxRef("RegExp.test", "test(aString)", "", 1)}}``. Only use + {{JSxRef("String.search", "aString.search()")}} if you need to know + the position of the result, or {{JSxRef("String.match", + "aString.match()")}} if you need to collect matching substrings + (delimited by parentheses in the regular expression). Regular + expressions are less useful if the match is unknown in advance, or to + extract substrings in known positions in the string. For instance, + {{JSxRef("String.slice", "aString.slice(-1)")}} returns the last + letter in ``aString``, or the empty string if ``aString`` is empty. + diff --git a/docs/code-quality/coding-style/coding_style_other.rst b/docs/code-quality/coding-style/coding_style_other.rst new file mode 100644 index 0000000000..2c82cc4a76 --- /dev/null +++ b/docs/code-quality/coding-style/coding_style_other.rst @@ -0,0 +1,11 @@ +================== +Other coding style +================== + +SVG practices +------------- + +Check `SVG +Guidelines <https://developer.mozilla.org/docs/Mozilla/Developer_guide/SVG_Guidelines>`__ for +more details. + diff --git a/docs/code-quality/coding-style/coding_style_python.rst b/docs/code-quality/coding-style/coding_style_python.rst new file mode 100644 index 0000000000..3a818fcfd4 --- /dev/null +++ b/docs/code-quality/coding-style/coding_style_python.rst @@ -0,0 +1,71 @@ +=================== +Python Coding style +=================== + +Coding style +~~~~~~~~~~~~ + + :ref:`black` is the tool used to reformat the Python code. + +Linting +~~~~~~~ + +The Python linting is done by :ref:`Flake8` and :ref:`pylint` +They are executed by mozlint both at review phase and in the CI. + +Indentation +~~~~~~~~~~~ + +Four spaces in Python code. + + +Makefile/moz.build practices +---------------------------- + +- Changes to makefile and moz.build variables do not require + build-config peer review. Any other build system changes, such as + adding new scripts or rules, require review from the build-config + team. +- Suffix long ``if``/``endif`` conditionals with #{ & #}, so editors + can display matched tokens enclosing a block of statements. + + :: + + ifdef CHECK_TYPE #{ + ifneq ($(flavor var_type),recursive) #{ + $(warning var should be expandable but detected var_type=$(flavor var_type)) + endif #} + endif #} + +- moz.build are python and follow normal Python style. +- List assignments should be written with one element per line. Align + closing square brace with start of variable assignment. If ordering + is not important, variables should be in alphabetical order. + + .. code-block:: python + + var += [ + 'foo', + 'bar' + ] + +- Use ``CONFIG['CPU_ARCH'] {=arm}`` to test for generic classes of + architecture rather than ``CONFIG['OS_TEST'] {=armv7}`` (re: bug 886689). + + +Other advices +~~~~~~~~~~~~~ + +- Install the + `mozext <https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/mozext>`__ + Mercurial extension, and address every issue reported on commit + or the output of ``hg critic``. +- Follow `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`__. Please run :ref:`black` for this. +- Do not place statements on the same line as ``if/elif/else`` + conditionals to form a one-liner. +- Global vars, please avoid them at all cost. +- Exclude outer parenthesis from conditionals.Use + ``if x > 5:,``\ rather than ``if (x > 5):`` +- Use string formatters, rather than var + str(val). + ``var = 'Type %s value is %d'% ('int', 5).`` +- Testing/Unit tests, please write them and make sure that they are executed in the CI. diff --git a/docs/code-quality/coding-style/format_cpp_code_with_clang-format.rst b/docs/code-quality/coding-style/format_cpp_code_with_clang-format.rst new file mode 100644 index 0000000000..10d2bc00fc --- /dev/null +++ b/docs/code-quality/coding-style/format_cpp_code_with_clang-format.rst @@ -0,0 +1,272 @@ +===================================== +Formatting C++ Code With clang-format +===================================== + +Mozilla uses the Google coding style for whitespace, which is enforced +using `clang-format <https://clang.llvm.org/docs/ClangFormat.html>`__. A +specific version of the binary will be installed when +``./mach clang-format`` or ``./mach bootstrap`` are run. We build our +own binaries and update them as needed. + +Options are explicitly defined `in clang-format +itself <https://github.com/llvm-mirror/clang/blob/e8a55f98df6bda77ee2eaa7f7247bd655f79ae0e/lib/Format/Format.cpp#L856>`__. +If the options are changed in clang upstream, this might cause some +changes in the Firefox tree. For this reason, it is best to use the +mozilla-provided binaries. + +Manual formatting +----------------- + +We provide a mach subcommand for running clang-format from the +command-line. This wrapper handles ensuring the correct version of +clang-format is installed and run. + +If clang-format isn’t installed, the binaries will be automatically +downloaded from taskcluster and installed into ~/.mozbuild. We build our +own clang-format binaries. + + +Formatting local changes +~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ ./mach clang-format + +When run without arguments, it will run on a local diff. This could miss +some reformatting (for example, when blocks are touched). +(`searchfox <https://searchfox.org/mozilla-central/rev/501eb4718d73870892d28f31a99b46f4783efaa0/python/mozbuild/mozbuild/code-analysis/mach_commands.py#1620>`__) + + +Formatting specific paths +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ ./mach clang-format -p <path> # Format <path> in-place + $ ./mach clang-format -p <path> -s # Show changes + +The command also accepts a ``-p`` argument to reformat a specific +directory or file, and a ``-s`` flag to show the changes instead of +applying them to the working directory +(`searchfox <https://searchfox.org/mozilla-central/rev/501eb4718d73870892d28f31a99b46f4783efaa0/python/mozbuild/mozbuild/code-analysis/mach_commands.py#1633>`__) + + +Formatting specific commits / revisions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ ./mach clang-format -c HEAD # Format a single git commit + $ ./mach clang-format -c HEAD~~..HEAD # Format a range of git commits + $ ./mach clang-format -c . # Format a single mercurial revision + +The command accepts a ``-c`` argument that takes a revision number or +commit range, and will format the lines modified by those commits. +(`searchfox <https://searchfox.org/mozilla-central/rev/501eb4718d73870892d28f31a99b46f4783efaa0/python/mozbuild/mozbuild/code-analysis/mach_commands.py#1635>`__) + + +Scripting Clang-Format +~~~~~~~~~~~~~~~~~~~~~~ + +Clang format expects that the path being passed to it is the path +on-disk. If this is not the case, for example when formatting a +temporary file, the "real" path must be specified. This can be done with +the ``--assume-filename <path>`` argument. + + +Configuring the clang-format commit hook +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To run clang-format at commit phase, run ``mach boostrap`` or just add +the following line in the ``hgrc`` file: + +.. code:: ini + + [extensions] + clang-format = ~/.mozbuild/version-control-tools/hgext/clang-format + +We use a hg extension as they are more flexible than hooks. + +With git, the configuration is the following: + +:: + + # From the root git directory: + $ ln -s $(pwd)/tools/lint/hooks_clang_format.py .git/hooks/pre-commit + +You'll likely need to install the ``python-hglib`` package for your OS, +or else you may get errors like ``abort: No module named hglib.client!`` +when you try to commit. + + +Editor integration +------------------ + +It is possible to configure many editors to support running +``clang-format`` automatically on save, or when run from within the +editor. + + +Editor plugins +~~~~~~~~~~~~~~ + +- `Atom <https://atom.io/packages/clang-format>`__ +- `BBEdit <http://clang.llvm.org/docs/ClangFormat.html#bbedit-integration>`__ + + - `clang-format-bbedit.applescript <https://raw.githubusercontent.com/llvm-mirror/clang/master/tools/clang-format/clang-format-bbedit.applescript>`__ + +- Eclipse + + - Install the + `CppStyle <https://marketplace.eclipse.org/content/cppstyle>`__ + plugin + - In Preferences -> C/C++ -> CppStyle, set the clang-format path to + ~/.mozbuild/clang-tools/clang-tidy/bin/clang-format + - (Optional) check "Run clang-format on file save" + +- `Emacs <http://clang.llvm.org/docs/ClangFormat.html#emacs-integration>`__ + + - `clang-format.el <https://raw.githubusercontent.com/llvm-mirror/clang/master/tools/clang-format/clang-format.el>`__ + (Or install + `clang-format <http://melpa.org/#/clang-format>`__ from MELPA) + - `google-c-style <http://melpa.org/#/google-c-style>`__ from MELPA + +- `Sublime Text <https://packagecontrol.io/packages/Clang%20Format>`__ + + - `alternative + tool <https://github.com/rosshemsley/SublimeClangFormat>`__ + +- `Vim <http://clang.llvm.org/docs/ClangFormat.html#vim-integration>`__ + + - `clang-format.py <https://raw.githubusercontent.com/llvm-mirror/clang/master/tools/clang-format/clang-format.py>`__ + - `vim-clang-format <https://github.com/rhysd/vim-clang-format>`__ + +- `Visual + Studio <https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.ClangFormat>`__ + + - `llvm.org plugin <http://llvm.org/builds/>`__ + - `Integrated support in Visual Studio + 2017 <https://blogs.msdn.microsoft.com/vcblog/2018/03/13/clangformat-support-in-visual-studio-2017-15-7-preview-1/>`__ + +- `Visual Studio + Code <https://marketplace.visualstudio.com/items?itemName=xaver.clang-format>`__ +- `XCode <https://github.com/travisjeffery/ClangFormat-Xcode>`__ +- `Script for patch + reformatting <http://clang.llvm.org/docs/ClangFormat.html#script-for-patch-reformatting>`__ + + - `clang-format-diff.py <https://raw.githubusercontent.com/llvm-mirror/clang/master/tools/clang-format/clang-format-diff.py>`__ + + +Configuration +~~~~~~~~~~~~~ + +These tools generally run clang-format themselves, and won't use +``./mach clang-format``. The binary installed by our tooling will be +located at ``~/.mozbuild/clang-tools/clang-tidy/bin/clang-format``. + +You typically shouldn't need to specify any other special configuration +in your editor besides the clang-format binary. Most of the +configuration that clang-format relies on for formatting is stored +inside our source tree. More specifically, using the .clang-format file +located in the root of the repository. Please note that this doesn't +include the list of ignored files and directories (provided by +.clang-format-ignore which is a feature provided by the mach command +wrapper). + +Coding style configuration is done within clang-format itself. When we +change the configuration (incorrect configuration, new feature in clang, +etc), we use `local +overrides <https://searchfox.org/mozilla-central/rev/501eb4718d73870892d28f31a99b46f4783efaa0/.clang-format>`__. + + +Ignored files & directories +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We maintain a `list of ignored directories and +files <https://searchfox.org/mozilla-central/rev/501eb4718d73870892d28f31a99b46f4783efaa0/.clang-format-ignore>`__, +which is used by ``./mach clang-format``. This is generally only used +for code broken by clang-format, and third-party code. + + +Ignored code hunks +~~~~~~~~~~~~~~~~~~ + +Sections of code may have formatting disabled using comments. If a +section must not be formatted, the following comments will disable the +reformat: + +:: + + // clang-format off + my code which should not be reformated + // clang-format on + +You can find an `example of code not +formatted <https://searchfox.org/mozilla-central/rev/501eb4718d73870892d28f31a99b46f4783efaa0/xpcom/io/nsEscape.cpp#22>`__. + + +Merging formatted and unformatted code +-------------------------------------- + +During the transition to using chromium style enforced by clang-format +for all code in tree, it will often be necessary to rebase non-formatted +code onto a formatted tree. + + +Mercurial +~~~~~~~~~ + +The ``format-source`` extension, now bundled with +``version-control-tools``, and installed by ``./mach bootstrap``, may be +used to seamlessly handle this situation. More details may be found in +this +`document <https://docs.google.com/document/d/13AwAsvKMhH0mflDlfatBqn6LmZHiQih76oxM4zfrPl4/edit>`__. + +The parent changeset of the reformat has been tagged as +``PRE_TREEWIDE_CLANG_FORMAT``. + + +Git +~~~ + +To perform a rebase onto mozilla-central after the merge, a handy merge +driver, ``clang-format-merge``, has been written: + +.. code:: shell + + $ git clone https://github.com/emilio/clang-format-merge + $ /path/to/clang-format-merge/git-wrapper rebase <upstream> + +The wrapper should clean up after itself, and the clone may be deleted +after the rebase is complete. + + +Ignore lists +------------ + +To make sure that the blame/annotate features of Mercurial or git aren't +affected. Two files are maintained to keep track of the reformatting +commits. + + +With Mercurial +~~~~~~~~~~~~~~ + +| The list is stored in + `https://searchfox.org/mozilla-central/source/.hg-annotate-ignore-revs </en-US/docs/>`__ +| Commit messages should also contain the string ``# ignore-this-changeset`` + +The syntax in this file is generated using the following syntax: + +:: + + $ hg log --template '{node} - {author|person} - {desc|strip|firstline}\n' + +With git +~~~~~~~~ + +The list is stored in +`https://searchfox.org/mozilla-central/source/.git-blame-ignore-revs </en-US/docs/>`__ +and contains git revisions for both gecko-dev and the git cinnabar +repository. diff --git a/docs/code-quality/coding-style/index.rst b/docs/code-quality/coding-style/index.rst new file mode 100644 index 0000000000..e62ce910ca --- /dev/null +++ b/docs/code-quality/coding-style/index.rst @@ -0,0 +1,20 @@ +Coding style +============ + +Firefox code is using different programming languages. +For each language, we are enforcing a specific coding style. + +Getting Help +------------ + +If you need help or have questions, please don’t hesitate to contact us via Matrix +in the "Lint and Formatting" room +(`#lint:mozilla.org <https://chat.mozilla.org/#/room/#lint:mozilla.org>`_). + + +.. toctree:: + :caption: Coding Style User Guide + :maxdepth: 2 + :glob: + + * diff --git a/docs/code-quality/coding-style/using_cxx_in_firefox_code.rst b/docs/code-quality/coding-style/using_cxx_in_firefox_code.rst new file mode 100644 index 0000000000..513098855a --- /dev/null +++ b/docs/code-quality/coding-style/using_cxx_in_firefox_code.rst @@ -0,0 +1,1065 @@ +Using C++ in Mozilla code +========================= + +C++ language features +--------------------- + +Mozilla code only uses a subset of C++. Runtime type information (RTTI) +is disabled, as it tends to cause a very large increase in codesize. +This means that ``dynamic_cast``, ``typeid()`` and ``<typeinfo>`` cannot +be used in Mozilla code. Also disabled are exceptions; do not use +``try``/``catch`` or throw any exceptions. Libraries that throw +exceptions may be used if you are willing to have the throw instead be +treated as an abort. + +On the side of extending C++, we compile with ``-fno-strict-aliasing``. +This means that when reinterpreting a pointer as a differently-typed +pointer, you don't need to adhere to the "effective type" (of the +pointee) rule from the standard (aka. "the strict aliasing rule") when +dereferencing the reinterpreted pointer. You still need make sure that +you don't violate alignment requirements and need to make sure that the +data at the memory location pointed to forms a valid value when +interpreted according to the type of the pointer when dereferencing the +pointer for reading. Likewise, if you write by dereferencing the +reinterpreted pointer and the originally-typed pointer might still be +dereferenced for reading, you need to make sure that the values you +write are valid according to the original type. This value validity +issue is moot for e.g. primitive integers for which all bit patterns of +their size are valid values. + +- As of Mozilla 59, C++14 mode is required to build Mozilla. +- As of Mozilla 67, MSVC can no longer be used to build Mozilla. +- As of Mozilla 73, C++17 mode is required to build Mozilla. + +This means that C++17 can be used where supported on all platforms. The +list of acceptable features is given below: + +.. list-table:: + :widths: 25 25 25 25 + :header-rows: 3 + + * - + - GCC + - Clang + - + * - Current minimal requirement + - 7.1 + - 5.0 + - + * - Feature + - GCC + - Clang + - Can be used in code + * - ``type_t &&`` + - 4.3 + - 2.9 + - Yes (see notes) + * - ref qualifiers on methods + - 4.8.1 + - 2.9 + - Yes + * - default member - initializers (except for bit-fields) + - 4.7 + - 3.0 + - Yes + * - default member - initializers (for bit-fields) + - 8 + - 6 + - **No** + * - variadic templates + - 4.3 + - 2.9 + - Yes + * - Initializer lists + - 4.4 + - 3.1 + - Yes + * - ``static_assert`` + - 4.3 + - 2.9 + - Yes + * - ``auto`` + - 4.4 + - 2.9 + - Yes + * - lambdas + - 4.5 + - 3.1 + - Yes + * - ``decltype`` + - 4.3 + - 2.9 + - Yes + * - ``Foo<Bar<T>>`` + - 4.3 + - 2.9 + - Yes + * - ``auto func() -> int`` + - 4.4 + - 3.1 + - Yes + * - Templated aliasing + - 4.7 + - 3.0 + - Yes + * - ``nullptr`` + - 4.6 + - 3.0 + - Yes + * - ``enum foo : int16_t`` {}; + - 4.4 + - 2.9 + - Yes + * - ``enum class foo {}``; + - 4.4 + - 2.9 + - Yes + * - ``enum foo;`` + - 4.6 + - 3.1 + - Yes + * - ``[[attributes]]`` + - 4.8 + - 3.3 + - **No** (see notes) + * - ``constexpr`` + - 4.6 + - 3.1 + - Yes + * - ``alignas`` + - 4.8 + - 3.3 + - Yes + * - ``alignof`` + - 4.8 + - 3.3 + - Yes, but see notes ; only clang 3.6 claims as_feature(cxx_alignof) + * - Delegated constructors + - 4.7 + - 3.0 + - Yes + * - Inherited constructors + - 4.8 + - 3.3 + - Yes + * - ``explicit operator bool()`` + - 4.5 + - 3.0 + - Yes + * - ``char16_t/u"string"`` + - 4.4 + - 3.0 + - Yes + * - ``R"(string)"`` + - 4.5 + - 3.0 + - Yes + * - ``operator""()`` + - 4.7 + - 3.1 + - Yes + * - ``=delete`` + - 4.4 + - 2.9 + - Yes + * - ``=default`` + - 4.4 + - 3.0 + - Yes + * - unrestricted unions + - 4.6 + - 3.1 + - Yes + * - ``for (auto x : vec)`` (`be careful about the type of the iterator <https://stackoverflow.com/questions/15176104/c11-range-based-loop-get-item-by-value-or-reference-to-const>`__) + - 4.6 + - 3.0 + - Yes + * - ``override``/``final`` + - 4.7 + - 3.0 + - Yes + * - ``thread_local`` + - 4.8 + - 3.3 + - **No** (see notes) + * - function template default arguments + - 4.3 + - 2.9 + - Yes + * - local structs as template parameters + - 4.5 + - 2.9 + - Yes + * - extended friend declarations + - 4.7 + - 2.9 + - Yes + * - ``0b100`` (C++14) + - 4.9 + - 2.9 + - Yes + * - `Tweaks to some C++ contextual conversions` (C++14) + - 4.9 + - 3.4 + - Yes + * - Return type deduction (C++14) + - 4.9 + - 3.4 + - Yes (but only in template code when you would have used ``decltype (complex-expression)``) + * - Generic lambdas (C++14) + - 4.9 + - 3.4 + - Yes + * - Initialized lambda captures (C++14) + - 4.9 + - 3.4 + - Yes + * - Digit separator (C++14) + - 4.9 + - 3.4 + - Yes + * - Variable templates (C++14) + - 5.0 + - 3.4 + - Yes + * - Relaxed constexpr (C++14) + - 5.0 + - 3.4 + - Yes + * - Aggregate member initialization (C++14) + - 5.0 + - 3.3 + - Yes + * - Clarifying memory allocation (C++14) + - 5.0 + - 3.4 + - Yes + * - [[deprecated]] attribute (C++14) + - 4.9 + - 3.4 + - **No** (see notes) + * - Sized deallocation (C++14) + - 5.0 + - 3.4 + - **No** (see notes) + * - Concepts (Concepts TS) + - 6.0 + - — + - **No** + * - Inline variables (C++17) + - 7.0 + - 3.9 + - **No** (clang 5 has bugs with inline variables) + * - constexpr_if (C++17) + - 7.0 + - 3.9 + - Yes + * - constexpr lambdas (C++17) + - — + - — + - **No** + * - Structured bindings (C++17) + - 7.0 + - 4.0 + - Yes + * - Separated declaration and condition in ``if``, ``switch`` (C++17) + - 7.0 + - 3.9 + - Yes + * - `Fold expressions <https://en.cppreference.com/w/cpp/language/fold>`__ (C++17) + - 6.0 + - 3.9 + - Yes + * - [[fallthrough]], [[maybe_unused]], [[nodiscard]] (C++17) + - 7.0 + - 3.9 + - Yes + * - Aligned allocation/deallocation (C++17) + - 7.0 + - 4.0 + - **No** (see notes) + * - #pragma once + - 3.4 + - Yes + - **Not** until we `normalize headers <https://groups.google.com/d/msg/mozilla.dev.platform/PgDjWw3xp8k/eqCFlP4Kz1MJ>`__ + * - `Source code information capture <https://en.cppreference.com/w/cpp/experimental/lib_extensions_2#Source_code_information_capture>`__ + - 8.0 + - — + - **No** + +Sources +~~~~~~~ + +* GCC: https://gcc.gnu.org/projects/cxx-status.html +* Clang: https://clang.llvm.org/cxx_status.html + +Notes +~~~~~ + +rvalue references: Implicit move method generation cannot be used. + +Attributes: Several common attributes are defined in +`mozilla/Attributes.h <https://searchfox.org/mozilla-central/source/mfbt/Attributes.h>`__ +or nscore.h. + +Alignment: Some alignment utilities are defined in +`mozilla/Alignment.h <https://searchfox.org/mozilla-central/source/mfbt/Alignment.h>`__. +/!\\ MOZ_ALIGNOF and alignof don't have the same semantics. Be careful +of what you expect from them. + +``[[deprecated]]``: If we have deprecated code, we should be removing it +rather than marking it as such. Marking things as ``[[deprecated]]`` +also means the compiler will warn if you use the deprecated API, which +turns into a fatal error in our automation builds, which is not helpful. + +Sized deallocation: Our compilers all support this (custom flags are +required for GCC and Clang), but turning it on breaks some classes' +``operator new`` methods, and `some +work <https://bugzilla.mozilla.org/show_bug.cgi?id=1250998>`__ would +need to be done to make it an efficiency win with our custom memory +allocator. + +Aligned allocation/deallocation: Our custom memory allocator doesn't +have support for these functions. + +Thread locals: ``thread_local`` is not supported on Android. + + +C++ and Mozilla standard libraries +---------------------------------- + +The Mozilla codebase contains within it several subprojects which follow +different rules for which libraries can and can't be used it. The rules +listed here apply to normal platform code, and assume unrestricted +usability of MFBT or XPCOM APIs. + +.. warning:: + + The rest of this section is a draft for expository and exploratory + purposes. Do not trust the information listed here. + +What follows is a list of standard library components provided by +Mozilla or the C++ standard. If an API is not listed here, then it is +not permissible to use it in Mozilla code. Deprecated APIs are not +listed here. In general, prefer Mozilla variants of data structures to +standard C++ ones, even when permitted to use the latter, since Mozilla +variants tend to have features not found in the standard library (e.g., +memory size tracking) or have more controllable performance +characteristics. + +A list of approved standard library headers is maintained in +`config/stl-headers.mozbuild <https://searchfox.org/mozilla-central/source/config/stl-headers.mozbuild>`__. + + +Data structures +~~~~~~~~~~~~~~~ + +.. list-table:: + :widths: 25 25 25 25 + :header-rows: 1 + + * - Name + - Header + - STL equivalent + - Notes + * - ``nsAutoTArray`` + - ``nsTArray.h`` + - + - Like ``nsTArray``, but will store a small amount as stack storage + * - ``nsAutoTObserverArray`` + - ``nsTObserverArray.h`` + - + - Like ``nsTObserverArray``, but will store a small amount as stack storage + * - ``mozilla::BloomFilter`` + - ``mozilla/BloomFilter.h`` + - + - Probabilistic set membership (see `Wikipedia <https://en.wikipedia.org/wiki/Bloom_filter#Counting_filters>`__) + * - ``nsClassHashtable`` + - ``nsClassHashtable.h`` + - + - Adaptation of nsTHashtable, see `XPCOM hashtable guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Hashtables>`__ + * - ``nsCOMArray`` + - ``nsCOMArray.h`` + - + - Like ``nsTArray<nsCOMPtr<T>>`` + * - ``nsDataHashtable`` + - ``nsClassHashtable.h`` + - ``std::unordered_map`` + - Adaptation of ``nsTHashtable``, see `XPCOM hashtable guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Hashtables>`__ + * - ``nsDeque`` + - ``nsDeque.h`` + - ``std::deque<void *>`` + - + * - ``mozilla::EnumSet`` + - ``mozilla/EnumSet.h`` + - + - Like ``std::set``, but for enum classes. + * - ``mozilla::Hash{Map,Set}`` + - `mozilla/HashTable.h <https://searchfox.org/mozilla-central/source/mfbt/HashTable.h>`__ + - ``std::unordered_{map,set}`` + - A general purpose hash map and hash set. + * - ``nsInterfaceHashtable`` + - ``nsInterfaceHashtable.h`` + - ``std::unordered_map`` + - Adaptation of ``nsTHashtable``, see `XPCOM hashtable guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Hashtables>`__ + * - ``nsJSThingHashtable`` + - ``nsJSThingHashtable.h`` + - + - Adaptation of ``nsTHashtable``, see `XPCOM hashtable guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Hashtables>`__ + * - ``mozilla::LinkedList`` + - ``mozilla/LinkedList.h`` + - ``std::list`` + - Doubly-linked list + * - ``nsRef PtrHashtable`` + - ``nsRefPtrHashtable.h`` + - ``std::unordered_map`` + - Adaptation of ``nsTHashtable``, see `XPCOM hashtable guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Hashtables>`__ + * - ``mozilla::SegmentedVector`` + - ``mozilla/SegmentedVector.h`` + - ``std::deque`` w/o O(1) pop_front + - Doubly-linked list of vector elements + * - ``mozilla::SplayTree`` + - ``mozilla/SplayTree.h`` + - + - Quick access to recently-accessed elements (see `Wikipedia <https://en.wikipedia.org/wiki/Splay_tree>`__) + * - ``nsTArray`` + - ``nsTArray.h`` + - ``std::vector`` + - + * - ``nsTHashtable`` + - ``nsTHashtable.h`` + - ``std::unordered_{map,set}`` + - See `XPCOM hashtable guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Hashtables>`__, you probably want a subclass + * - ``nsTObserverArray`` + - ``nsTObserverArray.h`` + - + - Like ``nsTArray``, but iteration is stable even through mutation + * - ``nsTPriorityQueue`` + - ``nsTPriorityQueue.h`` + - ``std::priority_queue`` + - Unlike the STL class, not a container adapter + * - ``mozilla::Vector`` + - ``mozilla/Vector.h`` + - ``std::vector`` + - + * - ``mozilla::Buffer`` + - ``mozilla/Buffer.h`` + - + - Unlike ``Array``, has a run-time variable length. Unlike ``Vector``, does not have capacity and growth mechanism. Unlike ``Span``, owns its buffer. + + +Safety utilities +~~~~~~~~~~~~~~~~ + +.. list-table:: + :widths: 25 25 25 25 + :header-rows: 1 + + * - Name + - Header + - STL equivalent + - Notes + * - ``mozilla::Array`` + - ``mfbt/Array.h`` + - + - safe array index + * - ``mozilla::AssertedCast`` + - ``mfbt/Casting.h`` + - + - casts + * - ``mozilla::CheckedInt`` + - ``mfbt/CheckedInt.h`` + - + - avoids overflow + * - ``nsCOMPtr`` + - ``xpcom/base/nsCOMPtr.h`` + - ``std::shared_ptr`` + - + * - ``mozilla::EnumeratedArray`` + - ``mfbt/EnumeratedArray.h`` + - ``mozilla::Array`` + - + * - ``mozilla::Maybe`` + - ``mfbt/Maybe.h`` + - ``std::optional`` + - + * - ``mozilla::RangedPtr`` + - ``mfbt/RangedPtr.h`` + - + - like ``mozilla::Span`` but with two pointers instead of pointer and length + * - ``mozilla::RefPtr`` + - ``mfbt/RefPtr.h`` + - ``std::shared_ptr`` + - + * - ``mozilla::Span`` + - ``mozilla/Span.h`` + - ``gsl::span``, ``absl::Span``, ``std::string_view``, ``std::u16string_view`` + - Rust's slice concept for C++ (without borrow checking) + * - ``StaticRefPtr`` + - ``xpcom/base/StaticPtr.h`` + - + - ``nsRefPtr`` w/o static constructor + * - ``mozilla::UniquePtr`` + - ``mfbt/UniquePtr.h`` + - ``std::unique_ptr`` + - + * - ``mozilla::WeakPtr`` + - ``mfbt/WeakPtr.h`` + - ``std::weak_ptr`` + - + * - ``nsWeakPtr`` + - ``xpcom/base/nsWeakPtr.h`` + - ``std::weak_ptr`` + - + + +Strings +~~~~~~~ + +See the `Mozilla internal string +guide <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Guide/Internal_strings>`__ for +usage of ``nsAString`` (our copy-on-write replacement for +``std::u16string``) and ``nsACString`` (our copy-on-write replacement +for ``std::string``). + +Be sure not to introduce further uses of ``std::wstring``, which is not +portable! (Some uses exist in the IPC code.) + + +Algorithms +~~~~~~~~~~ + +.. list-table:: + :widths: 25 25 + + * - ``mozilla::BinarySearch`` + - ``mfbt/BinarySearch.h`` + * - ``mozilla::BitwiseCast`` + - ``mfbt/Casting.h`` (strict aliasing-safe cast) + * - ``mozilla/MathAlgorithms.h`` + - (rotate, ctlz, popcount, gcd, abs, lcm) + * - ``mozilla::RollingMean`` + - ``mfbt/RollingMean.h`` () + + +Concurrency +~~~~~~~~~~~ + +.. list-table:: + :widths: 25 25 25 25 + :header-rows: 1 + + * - Name + - Header + - STL/boost equivalent + - Notes + * - ``mozilla::Atomic`` + - mfbt/Atomic.h + - ``std::atomic`` + - + * - ``mozilla::CondVar`` + - xpcom/threads/CondVar.h + - ``std::condition_variable`` + - + * - ``mozilla::DataMutex`` + - xpcom/threads/DataMutex.h + - ``boost::synchronized_value`` + - + * - ``mozilla::Monitor`` + - xpcom/threads/Monitor.h + - + - + * - ``mozilla::Mutex`` + - xpcom/threads/Mutex.h + - ``std::mutex`` + - + * - ``mozilla::ReentrantMonitor`` + - xpcom/threads/ReentrantMonitor.h + - + - + * - ``mozilla::StaticMutex`` + - xpcom/base/StaticMutex.h + - ``std::mutex`` + - Mutex that can (and in fact, must) be used as a global/static variable. + + +Miscellaneous +~~~~~~~~~~~~~ + +.. list-table:: + :widths: 25 25 25 25 + :header-rows: 1 + + * - Name + - Header + - STL/boost equivalent + - Notes + * - ``mozilla::AlignedStorage`` + - mfbt/Alignment.h + - ``std::aligned_storage`` + - + * - ``mozilla::MaybeOneOf`` + - mfbt/MaybeOneOf.h + - ``std::optional<std::variant<T1, T2>>`` + - ~ ``mozilla::Maybe<union {T1, T2}>`` + * - ``mozilla::Pair`` + - mfbt/Pair.h + - ``std::tuple<T1, T2>`` + - minimal space! + * - ``mozilla::TimeStamp`` + - xpcom/ds/TimeStamp.h + - ``std::chrono::time_point`` + - + * - + - mozilla/TypeTraits.h + - ``<type_traits>`` + - + * - + - mozilla/PodOperations.h + - + - C++ versions of ``memset``, ``memcpy``, etc. + * - + - mozilla/ArrayUtils.h + - + - + * - + - mozilla/Compression.h + - + - + * - + - mozilla/Endian.h + - + - + * - + - mozilla/FloatingPoint.h + - + - + * - + - mozilla/HashFunctions.h + - ``std::hash`` + - + * - + - mozilla/Move.h + - ``std::move``, ``std::swap``, ``std::forward`` + - + + +Mozilla data structures and standard C++ ranges and iterators +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some Mozilla-defined data structures provide STL-style +`iterators <https://en.cppreference.com/w/cpp/named_req/Iterator>`__ and +are usable in `range-based for +loops <https://en.cppreference.com/w/cpp/language/range-for>`__ as well +as STL `algorithms <https://en.cppreference.com/w/cpp/algorithm>`__. + +Currently, these include: + +.. list-table:: + :widths: 16 16 16 16 16 + :header-rows: 1 + + * - Name + - Header + - Bug(s) + - Iterator category + - Notes + * - ``nsTArray`` + - ``xpcom/ds/n sTArray.h`` + - `1126552 <https://bugzilla.mozilla.org/show_bug.cgi?id=1126552>`__ + - Random-access + - Also reverse-iterable. Also supports remove-erase pattern via RemoveElementsAt method. Also supports back-inserting output iterators via ``MakeBackInserter`` function. + * - ``nsBaseHashtable`` and subclasses: ``nsClassHashtable`` ``nsDataHashtable`` ``nsInterfaceHashtable`` ``nsJSThingHashtable`` ``nsRefPtrHashtable`` + - ``xpcom/ds/nsBaseHashtable.h`` ``xpcom/ds/nsClassHashtable.h`` ``xpcom/ds/nsDataHashtable.h`` ``xpcom/ds/nsInterfaceHashtable.h`` ``xpcom/ds/nsJSThingHashtable.h`` ``xpcom/ds/nsRefPtrHashtable.h`` + - `1575479 <https://bugzilla.mozilla.org/show_bug.cgi?id=1575479>`__ + - Forward + - + * - ``nsCOMArray`` + - ``xpcom/ds/nsCOMArray.h`` + - `1342303 <https://bugzilla.mozilla.org/show_bug.cgi?id=1342303>`__ + - Random-access + - Also reverse-iterable. + * - ``Array`` ``EnumerationArray`` ``RangedArray`` + - ``mfbt/Array.h`` ``mfbt/EnumerationArray.h`` ``mfbt/RangedArray.h`` + - `1216041 <https://bugzilla.mozilla.org/show_bug.cgi?id=1216041>`__ + - Random-access + - Also reverse-iterable. + * - ``Buffer`` + - ``mfbt/Buffer.h`` + - `1512155 <https://bugzilla.mozilla.org/show_bug.cgi?id=1512155>`__ + - Random-access + - Also reverse-iterable. + * - ``DoublyLinkedList`` + - ``mfbt/DoublyLinkedList.h`` + - `1277725 <https://bugzilla.mozilla.org/show_bug.cgi?id=1277725>`__ + - Forward + - + * - ``EnumeratedRange`` + - ``mfbt/EnumeratedRange.h`` + - `1142999 <https://bugzilla.mozilla.org/show_bug.cgi?id=1142999>`__ + - *Missing* + - Also reverse-iterable. + * - ``IntegerRange`` + - ``mfbt/IntegerRange.h`` + - `1126701 <https://bugzilla.mozilla.org/show_bug.cgi?id=1126701>`__ + - *Missing* + - Also reverse-iterable. + * - ``SmallPointerArray`` + - ``mfbt/SmallPointerArray.h`` + - `1331718 <https://bugzilla.mozilla.org/show_bug.cgi?id=1331718>`__ + - Random-access + - + * - ``Span`` + - ``mfbt/Span.h`` + - `1295611 <https://bugzilla.mozilla.org/show_bug.cgi?id=1295611>`__ + - Random-access + - Also reverse-iterable. + +Note that if the iterator category is stated as "missing", the type is +probably only usable in range-based for. This is most likely just an +omission, which could be easily fixed. + +Useful in this context are also the class template ``IteratorRange`` +(which can be used to construct a range from any pair of iterators) and +function template ``Reversed`` (which can be used to reverse any range), +both defined in ``mfbt/ReverseIterator.h`` + + +Further C++ rules +----------------- + + +Don't use static constructors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(You probably shouldn't be using global variables to begin with. Quite +apart from the weighty software-engineering arguments against them, +globals affect startup time! But sometimes we have to do ugly things.) + +Non-portable example: + +.. code-block:: c++ + + FooBarClass static_object(87, 92); + + void + bar() + { + if (static_object.count > 15) { + ... + } + } + +Once upon a time, there were compiler bugs that could result in +constructors not being called for global objects. Those bugs are +probably long gone by now, but even with the feature working correctly, +there are so many problems with correctly ordering C++ constructors that +it's easier to just have an init function: + +.. code-block:: c++ + + static FooBarClass* static_object; + + FooBarClass* + getStaticObject() + { + if (!static_object) + static_object = + new FooBarClass(87, 92); + return static_object; + } + + void + bar() + { + if (getStaticObject()->count > 15) { + ... + } + } + + +Don't use exceptions +~~~~~~~~~~~~~~~~~~~~ + +See the introduction to the "C++ language features" section at the start +of this document. + + +Don't use Run-time Type Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See the introduction to the "C++ language features" section at the start +of this document. + +If you need runtime typing, you can achieve a similar result by adding a +``classOf()`` virtual member function to the base class of your +hierarchy and overriding that member function in each subclass. If +``classOf()`` returns a unique value for each class in the hierarchy, +you'll be able to do type comparisons at runtime. + + +Don't use the C++ standard library (including iostream and locale) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See the section "C++ and Mozilla standard libraries". + + +Use C++ lambdas, but with care +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +C++ lambdas are supported across all our compilers now. Rejoice! We +recommend explicitly listing out the variables that you capture in the +lambda, both for documentation purposes, and to double-check that you're +only capturing what you expect to capture. + + +Use namespaces +~~~~~~~~~~~~~~ + +Namespaces may be used according to the style guidelines in :ref:`C++ Coding style`. + + +Don't mix varargs and inlines +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +What? Why are you using varargs to begin with?! Stop that at once! + + +Make header files compatible with C and C++ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Non-portable example: + +.. code-block:: c++ + + /*oldCheader.h*/ + int existingCfunction(char*); + int anotherExistingCfunction(char*); + + /* oldCfile.c */ + #include "oldCheader.h" + ... + + // new file.cpp + extern "C" { + #include "oldCheader.h" + }; + ... + +If you make new header files with exposed C interfaces, make the header +files work correctly when they are included by both C and C++ files. + +(If you need to include a C header in new C++ files, that should just +work. If not, it's the C header maintainer's fault, so fix the header if +you can, and if not, whatever hack you come up with will probably be +fine.) + +Portable example: + +.. code-block:: c++ + + /* oldCheader.h*/ + PR_BEGIN_EXTERN_C + int existingCfunction(char*); + int anotherExistingCfunction(char*); + PR_END_EXTERN_C + + /* oldCfile.c */ + #include "oldCheader.h" + ... + + // new file.cpp + #include "oldCheader.h" + ... + +There are number of reasons for doing this, other than just good style. +For one thing, you are making life easier for everyone else, doing the +work in one common place (the header file) instead of all the C++ files +that include it. Also, by making the C header safe for C++, you document +that "hey, this file is now being included in C++". That's a good thing. +You also avoid a big portability nightmare that is nasty to fix... + + +Use override on subclass virtual member functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``override`` keyword is supported in C++11 and in all our supported +compilers, and it catches bugs. + + +Always declare a copy constructor and assignment operator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many classes shouldn't be copied or assigned. If you're writing one of +these, the way to enforce your policy is to declare a deleted copy +constructor as private and not supply a definition. While you're at it, +do the same for the assignment operator used for assignment of objects +of the same class. Example: + +.. code-block:: c++ + + class Foo { + ... + private: + Foo(const Foo& x) = delete; + Foo& operator=(const Foo& x) = delete; + }; + +Any code that implicitly calls the copy constructor will hit a +compile-time error. That way nothing happens in the dark. When a user's +code won't compile, they'll see that they were passing by value, when +they meant to pass by reference (oops). + + +Be careful of overloaded methods with like signatures +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It's best to avoid overloading methods when the type signature of the +methods differs only by one "abstract" type (e.g. ``PR_Int32`` or +``int32``). What you will find as you move that code to different +platforms, is suddenly on the Foo2000 compiler your overloaded methods +will have the same type-signature. + + +Type scalar constants to avoid unexpected ambiguities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Non-portable code: + +.. code-block:: c++ + + class FooClass { + // having such similar signatures + // is a bad idea in the first place. + void doit(long); + void doit(short); + }; + + void + B::foo(FooClass* xyz) + { + xyz->doit(45); + } + +Be sure to type your scalar constants, e.g., ``uint32_t(10)`` or +``10L``. Otherwise, you can produce ambiguous function calls which +potentially could resolve to multiple methods, particularly if you +haven't followed (2) above. Not all of the compilers will flag ambiguous +method calls. + +Portable code: + +.. code-block:: c++ + + class FooClass { + // having such similar signatures + // is a bad idea in the first place. + void doit(long); + void doit(short); + }; + + void + B::foo(FooClass* xyz) + { + xyz->doit(45L); + } + + +Use nsCOMPtr in XPCOM code +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See the ``nsCOMPtr`` `User +Manual <https://developer.mozilla.org/en-US/docs/Using_nsCOMPtr>`__ for +usage details. + + +Don't use identifiers that start with an underscore +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This rule occasionally surprises people who've been hacking C++ for +decades. But it comes directly from the C++ standard! + +According to the C++ Standard, 17.4.3.1.2 Global Names +[lib.global.names], paragraph 1: + +Certain sets of names and function signatures are always reserved to the +implementation: + +- Each name that contains a double underscore (__) or begins with an + underscore followed by an uppercase letter (2.11) is reserved to the + implementation for any use. +- **Each name that begins with an underscore is reserved to the + implementation** for use as a name in the global namespace. + + +Stuff that is good to do for C or C++ +------------------------------------- + + +Avoid conditional #includes when possible +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Don't write an ``#include`` inside an ``#ifdef`` if you could instead +put it outside. Unconditional includes are better because they make the +compilation more similar across all platforms and configurations, so +you're less likely to cause stupid compiler errors on someone else's +favorite platform that you never use. + +Bad code example: + +.. code-block:: c++ + + #ifdef MOZ_ENABLE_JPEG_FOUR_BILLION + #include <stdlib.h> // <--- don't do this + #include "jpeg4e9.h" // <--- only do this if the header really might not be there + #endif + +Of course when you're including different system files for different +machines, you don't have much choice. That's different. + + +Every .cpp source file should have a unique name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Every object file linked into libxul needs to have a unique name. Avoid +generic names like nsModule.cpp and instead use nsPlacesModule.cpp. + + +Turn on warnings for your compiler, and then write warning free code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +What generates a warning on one platform will generate errors on +another. Turn warnings on. Write warning-free code. It's good for you. +Treat warnings as errors by adding +``ac_add_options --enable-warnings-as-errors`` to your mozconfig file. + + +Use the same type for all bitfields in a ``struct`` or ``class`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some compilers do not pack the bits when different bitfields are given +different types. For example, the following struct might have a size of +8 bytes, even though it would fit in 1: + +.. code-block:: c++ + + struct { + char ch: 1; + int i: 1; + }; + + +Don't use an enum type for a bitfield +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The classic example of this is using ``PRBool`` for a boolean bitfield. +Don't do that. ``PRBool`` is a signed integer type, so the bitfield's +value when set will be ``-1`` instead of ``+1``, which---I know, +*crazy*, right? The things C++ hackers used to have to put up with... + +You shouldn't be using ``PRBool`` anyway. Use ``bool``. Bitfields of +type ``bool`` are fine. + +Enums are signed on some platforms (in some configurations) and unsigned +on others and therefore unsuitable for writing portable code when every +bit counts, even if they happen to work on your system. diff --git a/docs/code-quality/index.rst b/docs/code-quality/index.rst new file mode 100644 index 0000000000..97939211df --- /dev/null +++ b/docs/code-quality/index.rst @@ -0,0 +1,184 @@ +Code quality +============ + +Because Firefox is a complex piece of software, a lot of tools are +executed to identify issues at development phase. +In this document, we try to list these all tools. + + +.. toctree:: + :maxdepth: 1 + :glob: + + static-analysis.rst + lint/index.rst + coding-style/index.rst + +.. list-table:: C/C++ + :header-rows: 1 + :widths: 20 20 20 20 20 + + * - Tools + - Has autofixes + - Meta bug + - More info + - Upstream + * - Custom clang checker + - + - + - `Source <https://searchfox.org/mozilla-central/source/build/clang-plugin>`_ + - + * - Clang-Tidy + - Yes + - `bug 712350 <https://bugzilla.mozilla.org/show_bug.cgi?id=712350>`__ + - :ref:`Static analysis <Mach static analysis>` + - https://clang.llvm.org/extra/clang-tidy/checks/list.html + * - Clang analyzer + - + - `bug 712350 <https://bugzilla.mozilla.org/show_bug.cgi?id=712350>`__ + - + - https://clang-analyzer.llvm.org/ + * - Coverity + - + - `bug 1230156 <https://bugzilla.mozilla.org/show_bug.cgi?id=1230156>`__ + - + - + * - cpp virtual final + - + - + - :ref:`cpp virtual final` + - + * - Semmle/LGTM + - + - `bug 1458117 <https://bugzilla.mozilla.org/show_bug.cgi?id=1458117>`__ + - + - + * - clang-format + - Yes + - `bug 1188202 <https://bugzilla.mozilla.org/show_bug.cgi?id=1188202>`__ + - :ref:`Formatting C++ Code With clang-format` + - https://clang.llvm.org/docs/ClangFormat.html + +.. list-table:: JavaScript + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Tools + - Has autofixes + - Meta bug + - More info + - Upstream + * - Eslint + - Yes + - `bug 1229856 <https://bugzilla.mozilla.org/show_bug.cgi?id=1229856>`__ + - :ref:`ESLint` + - https://eslint.org/ + * - Mozilla ESLint + - + - `bug 1229856 <https://bugzilla.mozilla.org/show_bug.cgi?id=1229856>`__ + - :ref:`Mozilla ESLint Plugin` + - + * - Prettier + - Yes + - `bug 1558517 <https://bugzilla.mozilla.org/show_bug.cgi?id=1558517>`__ + - :ref:`JavaScript Coding style` + - https://prettier.io/ + + + +.. list-table:: Python + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Tools + - Has autofixes + - Meta bug + - More info + - Upstream + * - Flake8 + - Yes (with `autopep8 <https://github.com/hhatto/autopep8>`_) + - `bug 1155970 <https://bugzilla.mozilla.org/show_bug.cgi?id=1155970>`__ + - :ref:`Flake8` + - http://flake8.pycqa.org/ + * - black + - Yes + - `bug 1555560 <https://bugzilla.mozilla.org/show_bug.cgi?id=1555560>`__ + - :ref:`black` + - https://black.readthedocs.io/en/stable + * - pylint + - + - `bug 1623024 <https://bugzilla.mozilla.org/show_bug.cgi?id=1623024>`__ + - :ref:`pylint` + - https://www.pylint.org/ + * - Python 2/3 compatibility check + - + - `bug 1496527 <https://bugzilla.mozilla.org/show_bug.cgi?id=1496527>`__ + - :ref:`Python 2/3 compatibility check` + - + + +.. list-table:: Rust + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Tools + - Has autofixes + - Meta bug + - More info + - Upstream + * - Rustfmt + - Yes + - `bug 1454764 <https://bugzilla.mozilla.org/show_bug.cgi?id=1454764>`__ + - :ref:`Rustfmt` + - https://github.com/rust-lang/rustfmt + * - Clippy + - + - `bug 1361342 <https://bugzilla.mozilla.org/show_bug.cgi?id=1361342>`__ + - :ref:`clippy` + - https://github.com/rust-lang/rust-clippy + +.. list-table:: Java + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Tools + - Has autofixes + - Meta bug + - More info + - Upstream + * - Infer + - + - `bug 1175203 <https://bugzilla.mozilla.org/show_bug.cgi?id=1175203>`__ + - + - https://github.com/facebook/infer + +.. list-table:: Others + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Tools + - Has autofixes + - Meta bug + - More info + - Upstream + * - shellcheck + - + - + - + - https://www.shellcheck.net/ + * - rstchecker + - + - + - :ref:`RST Linter` + - https://github.com/myint/rstcheck + * - Typo detection + - Yes + - + - :ref:`Codespell` + - https://github.com/codespell-project/codespell + * - YAML linter + - + - + - + - https://github.com/adrienverge/yamllint + diff --git a/docs/code-quality/lint/create.rst b/docs/code-quality/lint/create.rst new file mode 100644 index 0000000000..066cb5eeef --- /dev/null +++ b/docs/code-quality/lint/create.rst @@ -0,0 +1,329 @@ +Adding a New Linter to the Tree +=============================== + +Linter Requirements +------------------- + +For a linter to be integrated into the mozilla-central tree, it needs to have: + +* Any required dependencies should be installed as part of ``./mach bootstrap`` +* A ``./mach lint`` interface +* Running ``./mach lint`` command must pass (note, linters can be disabled for individual directories) +* Taskcluster/Treeherder integration +* In tree documentation (under ``docs/code-quality/lint``) to give a basic summary, links and any other useful information +* Unit tests (under ``tools/lint/test``) to make sure that the linter works as expected and we don't regress. + +The review group in Phabricator is ``#linter-reviewers``. + +Linter Basics +------------- + +A linter is a yaml file with a ``.yml`` extension. Depending on how the type of linter, there may +be python code alongside the definition, pointed to by the 'payload' attribute. + +Here's a trivial example: + +no-eval.yml + +.. code-block:: yaml + + EvalLinter: + description: Ensures the string eval doesn't show up. + extensions: ['js'] + type: string + payload: eval + +Now ``no-eval.yml`` gets passed into :func:`LintRoller.read`. + + +Linter Types +------------ + +There are four types of linters, though more may be added in the future. + +1. string - fails if substring is found +2. regex - fails if regex matches +3. external - fails if a python function returns a non-empty result list +4. structured_log - fails if a mozlog logger emits any lint_error or lint_warning log messages + +As seen from the example above, string and regex linters are very easy to create, but they +should be avoided if possible. It is much better to use a context aware linter for the language you +are trying to lint. For example, use eslint to lint JavaScript files, use flake8 to lint python +files, etc. + +Which brings us to the third and most interesting type of linter, +external. External linters call an arbitrary python function which is +responsible for not only running the linter, but ensuring the results +are structured properly. For example, an external type could shell out +to a 3rd party linter, collect the output and format it into a list of +:class:`Issue` objects. The signature for this python +function is ``lint(files, config, **kwargs)``, where ``files`` is a list of +files to lint and ``config`` is the linter definition defined in the ``.yml`` +file. + +Structured log linters are much like external linters, but suitable +for cases where the linter code is using mozlog and emits +``lint_error`` or ``lint_warning`` logging messages when the lint +fails. This is recommended for writing novel gecko-specific lints. In +this case the signature for lint functions is ``lint(files, config, logger, +**kwargs)``. + + +Linter Definition +----------------- + +Each ``.yml`` file must have at least one linter defined in it. Here are the supported keys: + +* description - A brief description of the linter's purpose (required) +* type - One of 'string', 'regex' or 'external' (required) +* payload - The actual linting logic, depends on the type (required) +* include - A list of file paths that will be considered (optional) +* exclude - A list of file paths or glob patterns that must not be matched (optional) +* extensions - A list of file extensions to be considered (optional) +* setup - A function that sets up external dependencies (optional) +* support-files - A list of glob patterns matching configuration files (optional) +* find-dotfiles - If set to ``true``, run on dot files (.*) (optional) +* ignore-case - If set to ``true`` and ``type`` is regex, ignore the case (optional) + +In addition to the above, some ``.yml`` files correspond to a single lint rule. For these, the +following additional keys may be specified: + +* message - A string to print on infraction (optional) +* hint - A string with a clue on how to fix the infraction (optional) +* rule - An id string for the lint rule (optional) +* level - The severity of the infraction, either 'error' or 'warning' (optional) + +For structured_log lints the following additional keys apply: + +* logger - A StructuredLog object to use for logging. If not supplied + one will be created (optional) + + +Example +------- + +Here is an example of an external linter that shells out to the python flake8 linter, +let's call the file ``flake8_lint.py`` (`in-tree version <https://searchfox.org/mozilla-central/source/tools/lint/python/flake8.py>`_): + +.. code-block:: python + + import json + import os + import subprocess + from collections import defaultdict + + from mozlint import result + + try: + from shutil import which + except ImportError: + from shutil_which import which + + + FLAKE8_NOT_FOUND = """ + Could not find flake8! Install flake8 and try again. + """.strip() + + + def lint(files, config, **lintargs): + binary = os.environ.get('FLAKE8') + if not binary: + binary = which('flake8') + if not binary: + print(FLAKE8_NOT_FOUND) + return 1 + + # Flake8 allows passing in a custom format string. We use + # this to help mold the default flake8 format into what + # mozlint's Issue object expects. + cmdargs = [ + binary, + '--format', + '{"path":"%(path)s","lineno":%(row)s,"column":%(col)s,"rule":"%(code)s","message":"%(text)s"}', + ] + files + + proc = subprocess.Popen(cmdargs, stdout=subprocess.PIPE, env=os.environ) + output = proc.communicate()[0] + + # all passed + if not output: + return [] + + results = [] + for line in output.splitlines(): + # res is a dict of the form specified by --format above + res = json.loads(line) + + # parse level out of the id string + if 'code' in res and res['code'].startswith('W'): + res['level'] = 'warning' + + # result.from_linter is a convenience method that + # creates a Issue using a LINTER definition + # to populate some defaults. + results.append(result.from_config(config, **res)) + + return results + +Now here is the linter definition that would call it: + +.. code-block:: yaml + + flake8: + description: Python linter + include: ['.'] + extensions: ['py'] + type: external + payload: py.flake8:lint + support-files: + - '**/.flake8' + +Notice the payload has two parts, delimited by ':'. The first is the module +path, which ``mozlint`` will attempt to import. The second is the object path +within that module (e.g, the name of a function to call). It is up to consumers +of ``mozlint`` to ensure the module is in ``sys.path``. Structured log linters +use the same import mechanism. + +The ``support-files`` key is used to list configuration files or files related +to the running of the linter itself. If using ``--outgoing`` or ``--workdir`` +and one of these files was modified, the entire tree will be linted instead of +just the modified files. + +Result definition +----------------- + +When generating the list of results, the following values are available. + +.. csv-table:: + :header: "Name", "Description", "Optional" + :widths: 20, 40, 10 + + "linter", "Name of the linter that flagged this error", "" + "path", "Path to the file containing the error", "" + "message", "Text describing the error", "" + "lineno", "Line number that contains the error", "" + "column", "Column containing the error", "" + "level", "Severity of the error, either 'warning' or 'error' (default 'error')", "Yes" + "hint", "Suggestion for fixing the error", "Yes" + "source", "Source code context of the error", "Yes" + "rule", "Name of the rule that was violated", "Yes" + "lineoffset", "Denotes an error spans multiple lines, of the form (<lineno offset>, <num lines>)", "Yes" + "diff", "A diff describing the changes that need to be made to the code", "Yes" + + +Automated testing +----------------- + +Every new checker must have tests associated. + +They should be pretty easy to write as most of the work is managed by the Mozlint +framework. The key declaration is the ``LINTER`` variable which must match +the linker declaration. + +As an example, the `Flake8 test <https://searchfox.org/mozilla-central/source/tools/lint/test/test_flake8.py>`_ looks like the following snippet: + +.. code-block:: python + + import mozunit + LINTER = 'flake8' + + def test_lint_single_file(lint, paths): + results = lint(paths('bad.py')) + assert len(results) == 2 + assert results[0].rule == 'F401' + assert results[1].rule == 'E501' + assert results[1].lineno == 5 + + if __name__ == '__main__': + mozunit.main() + +As always with tests, please make sure that enough positive and negative cases are covered. + +To run the tests: + +.. code-block:: shell + + $ ./mach python-test --python 3 --subsuite mozlint + + +More tests can be `found in-tree <https://searchfox.org/mozilla-central/source/tools/lint/test>`_. + + + +Bootstrapping Dependencies +-------------------------- + +Many linters, especially 3rd party ones, will require a set of dependencies. It +could be as simple as installing a binary from a package manager, or as +complicated as pulling a whole graph of tools, plugins and their dependencies. + +Either way, to reduce the burden on users, linters should strive to provide +automated bootstrapping of all their dependencies. To help with this, +``mozlint`` allows linters to define a ``setup`` config, which has the same +path object format as an external payload. For example (`in-tree version <https://searchfox.org/mozilla-central/source/tools/lint/flake8.yml>`_): + +.. code-block:: yaml + + flake8: + description: Python linter + include: ['.'] + extensions: ['py'] + type: external + payload: py.flake8:lint + setup: py.flake8:setup + +The setup function takes a single argument, the root of the repository being +linted. In the case of ``flake8``, it might look like: + +.. code-block:: python + + import subprocess + from distutils.spawn import find_executable + + def setup(root, **lintargs): + # This is a simple example. Please look at the actual source for better examples. + if not find_executable('flake8'): + subprocess.call(['pip', 'install', 'flake8']) + +The setup function will be called implicitly before running the linter. This +means it should return fast and not produce any output if there is no setup to +be performed. + +The setup functions can also be called explicitly by running ``mach lint +--setup``. This will only perform setup and not perform any linting. It is +mainly useful for other tools like ``mach bootstrap`` to call into. + + +Adding the linter to the CI +--------------------------- + +First, the job will have to be declared in Taskcluster. + +This should be done in the `mozlint Taskcluster configuration <https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/mozlint.yml>`_. +You will need to define a symbol, how it is executed and on what kind of change. + +For example, for flake8, the configuration is the following: + +.. code-block:: yaml + + py-flake8: + description: flake8 run over the gecko codebase + treeherder: + symbol: py(f8) + run: + mach: lint -l flake8 -f treeherder -f json:/builds/worker/mozlint.json + when: + files-changed: + - '**/*.py' + - '**/.flake8' + # moz.configure files are also Python files. + - '**/*.configure' + +If the linter requires an external program, you will have to install it in the `setup script <https://searchfox.org/mozilla-central/source/taskcluster/docker/lint/system-setup.sh>`_ +and maybe install the necessary files in the `Docker configuration <https://searchfox.org/mozilla-central/source/taskcluster/docker/lint/Dockerfile>`_. + +.. note:: + + If the defect found by the linter is minor, make sure that it is run as `tier 2 <https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Overview_of_the_Job_Visibility_Tiers>`_. + This prevents the tree from closing because of a tiny issue. + For example, the typo detection is run as tier-2. diff --git a/docs/code-quality/lint/index.rst b/docs/code-quality/lint/index.rst new file mode 100644 index 0000000000..1c282f60b4 --- /dev/null +++ b/docs/code-quality/lint/index.rst @@ -0,0 +1,38 @@ +Linting +======= + +Linters are used in mozilla-central to help enforce coding style and avoid bad practices. Due to the +wide variety of languages in use, this is not always an easy task. In addition, linters should be runnable from editors, from the command line, from review tools +and from continuous integration. It's easy to see how the complexity of running all of these +different kinds of linters in all of these different places could quickly balloon out of control. + +``Mozlint`` is a library that accomplishes several goals: + +1. It provides a standard method for adding new linters to the tree, which can be as easy as + defining a config object in a ``.yml`` file. This helps keep lint related code localized, and + prevents different teams from coming up with their own unique lint implementations. +2. It provides a streamlined interface for running all linters at once. Instead of running N + different lint commands to test your patch, a single ``mach lint`` command will automatically run + all applicable linters. This means there is a single API surface that other tools can use to + invoke linters. +3. With a simple taskcluster configuration, Mozlint provides an easy way to execute all these jobs + at review phase. + +``Mozlint`` isn't designed to be used directly by end users. Instead, it can be consumed by things +like mach, phabricator and taskcluster. + +Getting Help +------------ + +If you need help or have questions, please don’t hesitate to contact us via Matrix +in the "Lint and Formatting" room +(`#lint:mozilla.org <https://chat.mozilla.org/#/room/#lint:mozilla.org>`_). + +.. toctree:: + :caption: Linting User Guide + :maxdepth: 1 + :glob: + + usage + create + linters/* diff --git a/docs/code-quality/lint/linters/black.rst b/docs/code-quality/lint/linters/black.rst new file mode 100644 index 0000000000..da4c1a52a2 --- /dev/null +++ b/docs/code-quality/lint/linters/black.rst @@ -0,0 +1,36 @@ +Black +===== + +`Black <https://black.readthedocs.io/en/stable/>`__ is a opinionated python code formatter. + + +Run Locally +----------- + +The mozlint integration of black can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter black <file paths> + +Alternatively, omit the ``--linter black`` and run all configured linters, which will include +black. + + +Configuration +------------- + +To enable black on new directory, add the path to the include +section in the `black.yml <https://searchfox.org/mozilla-central/source/tools/lint/black.yml>`_ file. + +Autofix +------- + +The black linter provides a ``--fix`` option. + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/black.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/python/black.py>`_ diff --git a/docs/code-quality/lint/linters/clang-format.rst b/docs/code-quality/lint/linters/clang-format.rst new file mode 100644 index 0000000000..2913c7440e --- /dev/null +++ b/docs/code-quality/lint/linters/clang-format.rst @@ -0,0 +1,35 @@ +clang-format +============ + +`clang-format <https://clang.llvm.org/docs/ClangFormat.html>`__ is a tool to reformat C/C++ to the right coding style. + +Run Locally +----------- + +The mozlint integration of clang-format can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter clang-format <file paths> + + +Configuration +------------- + +To enable clang-format on new directory, add the path to the include +section in the `clang-format.yml <https://searchfox.org/mozilla-central/source/tools/lint/clang-format.yml>`_ file. + +While excludes: will work, this linter will read the ignore list from `.clang-format-ignore file <https://searchfox.org/mozilla-central/source/.clang-format-ignore>`_ +at the root directory. This because it is also used by the ./mach clang-format -p command. + +Autofix +------- + +clang-format can reformat the code with the option `--fix` (based on the upstream option `-i`). +To highlight the results, we are using the ``--dry-run`` option (from clang-format 10). + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/clang-format.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/clang-format/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/clippy.rst b/docs/code-quality/lint/linters/clippy.rst new file mode 100644 index 0000000000..728b38b6b4 --- /dev/null +++ b/docs/code-quality/lint/linters/clippy.rst @@ -0,0 +1,42 @@ +clippy +====== + +`clippy`_ is the tool for Rust static analysis. + +Run Locally +----------- + +The mozlint integration of clippy can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter clippy <file paths> + +.. note:: + + clippy expects a path or a .rs file. It doesn't accept Cargo.toml + as it would break the mozlint workflow. + +Configuration +------------- + +To enable clippy on new directory, add the path to the include +section in the `clippy.yml <https://searchfox.org/mozilla-central/source/tools/lint/clippy.yml>`_ file. + +Autofix +------- + +This linter provides a ``--fix`` option. It requires using nightly +which can be installed with: + +.. code-block:: shell + + $ rustup component add clippy --toolchain nightly-x86_64-unknown-linux-gnu + + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/clippy.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/clippy/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/codespell.rst b/docs/code-quality/lint/linters/codespell.rst new file mode 100644 index 0000000000..e5804d30bb --- /dev/null +++ b/docs/code-quality/lint/linters/codespell.rst @@ -0,0 +1,36 @@ +Codespell +========= + +`codespell <https://github.com/codespell-project/codespell/>`__ is a popular tool to look for typical typos in the source code. + +It is enabled mostly for the documentation and English locale files. + +Run Locally +----------- + +The mozlint integration of codespell can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter codespell <file paths> + + +Configuration +------------- + +To enable codespell on new directory, add the path to the include +section in the `codespell.yml <https://searchfox.org/mozilla-central/source/tools/lint/codespell.yml>`_ file. + +This job is configured as `tier 2 <https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Overview_of_the_Job_Visibility_Tiers>`_. + +Autofix +------- + +Codespell provides a ``--fix`` option. It is based on the ``-w`` option provided by upstream. + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/codespell.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/spell/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/cpp-virtual-final.rst b/docs/code-quality/lint/linters/cpp-virtual-final.rst new file mode 100644 index 0000000000..5353b6b4fe --- /dev/null +++ b/docs/code-quality/lint/linters/cpp-virtual-final.rst @@ -0,0 +1,30 @@ +cpp virtual final +================= + +This linter detects the virtual function declarations with multiple specifiers. + +It matches our coding style: +Method declarations must use at most one of the following keywords: virtual, override, or final. + +As this linter uses some simple regular expression, it can miss some declarations. + +Run Locally +----------- + +This linter can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter cpp-virtual-final <file paths> + + +Configuration +------------- + +This linter is enabled on all C family files. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/cpp-virtual-final.yml>`_ + diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla.rst new file mode 100644 index 0000000000..bd844f752a --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla.rst @@ -0,0 +1,243 @@ +===================== +Mozilla ESLint Plugin +===================== + +This is the documentation of Mozilla ESLint PLugin. + +Environments +============ + +The plugin implements the following environments: + + +.. toctree:: + :maxdepth: 2 + + eslint-plugin-mozilla/environment + +Rules +===== + +The plugin implements the following rules: + +.. toctree:: + :maxdepth: 1 + + eslint-plugin-mozilla/avoid-Date-timing + eslint-plugin-mozilla/no-define-cc-etc + eslint-plugin-mozilla/no-throw-cr-literal + eslint-plugin-mozilla/no-useless-parameters + eslint-plugin-mozilla/use-chromeutils-import + +avoid-removeChild +----------------- + +Rejects using element.parentNode.removeChild(element) when element.remove() +can be used instead. + +balanced-listeners +------------------ + +Checks that for every occurrence of 'addEventListener' or 'on' there is an +occurrence of 'removeEventListener' or 'off' with the same event name. + +consistent-if-bracing +--------------------- + +Checks that if/elseif/else bodies are braced consistently, so either all bodies +are braced or unbraced. Doesn't enforce either of those styles though. + +import-browser-window-globals +----------------------------- + +For scripts included in browser-window, this will automatically inject the +browser-window global scopes into the file. + +import-content-task-globals +--------------------------- + +For files containing ContentTask.spawn calls, this will automatically declare +the frame script variables in the global scope. ContentTask is only available +to test files, so by default the configs only specify it for the mochitest based +configurations. + +This saves setting the file as a mozilla/frame-script environment. + +Note: due to the way ESLint works, it appears it is only easy to declare these +variables on a file global scope, rather than function global. This may mean that +they are incorrectly allowed, but given they are test files, this should be +detected during testing. + +import-globals +-------------- + +Checks the filename of imported files e.g. ``Cu.import("some/path/Blah.jsm")`` +adds Blah to the global scope. + +Note: uses modules.json for a list of globals listed in each file. + + +import-globals-from +------------------- + +Parses a file for globals defined in various unique Mozilla ways. + +When a "import-globals-from <path>" comment is found in a file, then all globals +from the file at <path> will be imported in the current scope. This will also +operate recursively. + +This is useful for scripts that are loaded as <script> tag in a window and rely +on each other's globals. + +If <path> is a relative path, then it must be relative to the file being +checked by the rule. + + +import-headjs-globals +--------------------- + +Import globals from head.js and from any files that were imported by +head.js (as far as we can correctly resolve the path). + +The following file import patterns are supported: + +- ``Services.scriptloader.loadSubScript(path)`` +- ``loader.loadSubScript(path)`` +- ``loadSubScript(path)`` +- ``loadHelperScript(path)`` +- ``import-globals-from path`` + +If path does not exist because it is generated e.g. +``testdir + "/somefile.js"`` we do our best to resolve it. + +The following patterns are supported: + +- ``Cu.import("resource://devtools/client/shared/widgets/ViewHelpers.jsm");`` +- ``loader.lazyImporter(this, "name1");`` +- ``loader.lazyRequireGetter(this, "name2"`` +- ``loader.lazyServiceGetter(this, "name3"`` +- ``XPCOMUtils.defineLazyModuleGetter(this, "setNamedTimeout", ...)`` +- ``loader.lazyGetter(this, "toolboxStrings"`` +- ``XPCOMUtils.defineLazyGetter(this, "clipboardHelper"`` + + +mark-test-function-used +----------------------- + +Simply marks `test` (the test method) or `run_test` as used when in mochitests +or xpcshell tests respectively. This avoids ESLint telling us that the function +is never called. + + +no-aArgs +-------- + +Checks that function argument names don't start with lowercase 'a' followed by +a capital letter. This is to prevent the use of Hungarian notation whereby the +first letter is a prefix that indicates the type or intended use of a variable. + +no-compare-against-boolean-literals +----------------------------------- + +Checks that boolean expressions do not compare against literal values +of `true` or `false`. This is to prevent overly verbose code such as +`if (isEnabled == true)` when `if (isEnabled)` would suffice. + +no-useless-removeEventListener +------------------------------ + +Reject calls to removeEventListener where {once: true} could be used instead. + +no-useless-run-test +------------------- + +Designed for xpcshell-tests. Rejects definitions of ``run_test()`` where the +function only contains a single call to ``run_next_test()``. xpcshell's head.js +already defines a utility function so there is no need for duplication. + +reject-importGlobalProperties +----------------------------- + +Rejects calls to ``Cu.importGlobalProperties``. Use of this function is +undesirable in some parts of the tree. + + +reject-some-requires +-------------------- + +This takes an option, a regular expression. Invocations of +``require`` with a string literal argument are matched against this +regexp; and if it matches, the ``require`` use is flagged. + + +use-cc-etc +---------- + +This requires using ``Cc`` rather than ``Components.classes``, and the same for +``Components.interfaces``, ``Components.results`` and ``Components.utils``. This has +a slight performance advantage by avoiding the use of the dot. + +use-default-preference-values +----------------------------- + +Require providing a second parameter to get*Pref methods instead of +using a try/catch block. + +use-ownerGlobal +--------------- + +Require .ownerGlobal instead of .ownerDocument.defaultView. + +use-includes-instead-of-indexOf +------------------------------- + +Use .includes instead of .indexOf to check if something is in an array or string. + +use-returnValue +--------------- + +Warn when idempotent methods are called and their return value is unused. + +use-services +------------ + +Requires the use of Services.jsm rather than Cc[].getService() where a service +is already defined in Services.jsm. + +var-only-at-top-level +--------------------- + +Marks all var declarations that are not at the top level invalid. + +Tests +===== + +The tests for eslint-plugin-mozilla are run via `mochajs`_ on top of node. Most +of the tests use the `ESLint Rule Unit Test framework`_. + +.. _mochajs: https://mochajs.org/ +.. _ESLint Rule Unit Test Framework: http://eslint.org/docs/developer-guide/working-with-rules#rule-unit-tests + +Running Tests +------------- + +The tests for eslint-plugin-mozilla are run via `mochajs`_ on top of node. Most +of the tests use the `ESLint Rule Unit Test framework`_. + +The rules have some self tests, these can be run via: + +.. code-block:: shell + + $ cd tools/lint/eslint/eslint-plugin-mozilla + $ npm install + $ npm run test + +Disabling tests +--------------- + +In the unlikely event of needing to disable a test, currently the only way is +by commenting-out. Please file a bug if you have to do this. Bugs should be filed +in the *Testing* product under *Lint*. + +.. _mochajs: https://mochajs.org/ +.. _ESLint Rule Unit Test Framework: http://eslint.org/docs/developer-guide/working-with-rules#rule-unit-tests diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla/avoid-Date-timing.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla/avoid-Date-timing.rst new file mode 100644 index 0000000000..378b153f03 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla/avoid-Date-timing.rst @@ -0,0 +1,31 @@ +================= +avoid-Date-timing +================= + +Rejects grabbing the current time via Date.now() or new Date() for timing +purposes when the less problematic performance.now() can be used instead. + +The performance.now() function returns milliseconds since page load. To +convert that to milliseconds since the epoch, use: + +.. code-block:: js + + performance.timing.navigationStart + performance.now() + +Often timing relative to the page load is adequate and that conversion may not +be necessary. + +Examples of incorrect code for this rule: +----------------------------------------- + +.. code-block:: js + + Date.now() + +Examples of correct code for this rule: +--------------------------------------- + +.. code-block:: js + + new Date('2017-07-11'); + performance.now() diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla/environment.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla/environment.rst new file mode 100644 index 0000000000..34c4b20343 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla/environment.rst @@ -0,0 +1,33 @@ +=========== +Environment +=========== + +These environments are available by specifying a comment at the top of the file, +e.g. + +.. code-block:: js + + /* eslint-env mozilla/chrome-worker */ + +There are also built-in ESLint environments available as well. Find them here: http://eslint.org/docs/user-guide/configuring#specifying-environments + +browser-window +-------------- + +Defines the environment for scripts that are in the main browser.xhtml scope. + +chrome-worker +------------- + +Defines the environment for chrome workers. This differs from normal workers by +the fact that `ctypes` can be accessed as well. + +frame-script +------------ + +Defines the environment for frame scripts. + +jsm +--- + +Defines the environment for jsm files (javascript modules). diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-define-cc-etc.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-define-cc-etc.rst new file mode 100644 index 0000000000..cb55dabac0 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-define-cc-etc.rst @@ -0,0 +1,24 @@ +================ +no-define-cc-etc +================ + +Disallows old-style definitions for Cc/Ci/Cu/Cr. These are now defined globally +for all chrome contexts. + +Examples of incorrect code for this rule: +----------------------------------------- + +.. code-block:: js + + var Cc = Components.classes; + var Ci = Components.interfaces; + var {Ci: interfaces, Cc: classes, Cu: utils} = Components; + var Cr = Components.results; + + +Examples of correct code for this rule: +--------------------------------------- + +.. code-block:: js + + const CC = Components.Constructor; diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-throw-cr-literal.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-throw-cr-literal.rst new file mode 100644 index 0000000000..ce73c72b36 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-throw-cr-literal.rst @@ -0,0 +1,39 @@ +=================== +no-throw-cr-literal +=================== + +This is similar to the ESLint built-in rule no-throw-literal. It disallows +throwing Components.results code directly. + +Throwing bare literals is inferior to throwing Exception objects, which provide +stack information. Cr.ERRORs should be be passed as the second argument to +``Components.Exception()`` to create an Exception object with stack info, and +the correct result property corresponding to the NS_ERROR that other code +expects. +Using a regular ``new Error()`` to wrap just turns it into a string and doesn't +set the result property, so the errors can't be recognised. + +This option can be autofixed (``--fix``). + +.. code-block:: js + + performance.timing.navigationStart + performance.now() + +Often timing relative to the page load is adequate and that conversion may not +be necessary. + +Examples of incorrect code for this rule: +----------------------------------------- + +.. code-block:: js + + throw Cr.NS_ERROR_UNEXPECTED; + throw Components.results.NS_ERROR_ABORT; + throw new Error(Cr.NS_ERROR_NO_INTERFACE); + +Examples of correct code for this rule: +--------------------------------------- + +.. code-block:: js + + throw Components.Exception("Not implemented", Cr.NS_ERROR_NOT_IMPLEMENTED); diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-useless-parameters.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-useless-parameters.rst new file mode 100644 index 0000000000..2783746198 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla/no-useless-parameters.rst @@ -0,0 +1,27 @@ +===================== +no-useless-parameters +===================== + +Reject common XPCOM methods called with useless optional parameters (eg. +``Services.io.newURI(url, null, null)``, or non-existent parameters (eg. +``Services.obs.removeObserver(name, observer, false)``). + +This option can be autofixed (``--fix``). + +Examples of incorrect code for this rule: +----------------------------------------- + +.. code-block:: js + + elt.addEventListener('click', handler, false); + Services.io.newURI('http://example.com', null, null); + Services.obs.notifyObservers(obj, 'topic', null); + +Examples of correct code for this rule: +--------------------------------------- + +.. code-block:: js + + elt.addEventListener('click', handler); + Services.io.newURI('http://example.com'); + Services.obs.notifyObservers(obj, 'topic'); diff --git a/docs/code-quality/lint/linters/eslint-plugin-mozilla/use-chromeutils-import.rst b/docs/code-quality/lint/linters/eslint-plugin-mozilla/use-chromeutils-import.rst new file mode 100644 index 0000000000..7cef9d19ad --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-mozilla/use-chromeutils-import.rst @@ -0,0 +1,25 @@ +====================== +use-chromeutils-import +====================== + +Require use of ``ChromeUtils.import`` and ``ChromeUtils.defineModuleGetter`` +rather than ``Components.utils.import`` and +``XPCOMUtils.defineLazyModuleGetter``. + +Examples of incorrect code for this rule: +----------------------------------------- + +.. code-block:: js + + Components.utils.import("resource://gre/modules/Services.jsm", this); + XPCOMUtils.defineLazyModuleGetter(this, "Services", "resource://gre/modules/Services.jsm"); + +Examples of correct code for this rule: +--------------------------------------- + +.. code-block:: js + + ChromeUtils.import("resource://gre/modules/Services.jsm", this); + ChromeUtils.defineModuleGetter(this, "Services", "resource://gre/modules/Services.jsm"); + // 4 argument version of defineLazyModuleGetter is allowed. + XPCOMUtils.defineLazyModuleGetter(this, "Services","resource://gre/modules/Service.jsm","Foo"); diff --git a/docs/code-quality/lint/linters/eslint-plugin-spidermonkey-js.rst b/docs/code-quality/lint/linters/eslint-plugin-spidermonkey-js.rst new file mode 100644 index 0000000000..35ff88dd59 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint-plugin-spidermonkey-js.rst @@ -0,0 +1,15 @@ +============================== +Mozilla ESLint SpiderMonkey JS +============================== + +This plugin only creates one item at the moment - a processor for the SpiderMonkey +JS code. + +Processors +========== + +The processor is used to pre-process all `*.js` files and deals with the macros +that SpiderMonkey uses. + +Note: Currently the ESLint option --fix is disabled when the preprocessor is +enabled. diff --git a/docs/code-quality/lint/linters/eslint.rst b/docs/code-quality/lint/linters/eslint.rst new file mode 100644 index 0000000000..5803163751 --- /dev/null +++ b/docs/code-quality/lint/linters/eslint.rst @@ -0,0 +1,60 @@ +ESLint +====== + +`ESLint <http://eslint.org/>`__ is a popular linter for JavaScript. + +Run Locally +----------- + +The mozlint integration of `ESLint <http://eslint.org/>`__ can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter eslint <file paths> + +Alternatively, omit the ``--linter eslint`` and run all configured linters, which will include +ESLint. + + +Configuration +------------- + +The ESLint mozilla-central integration uses a skip list to exclude certain directories from being +linted. This lives in ``topsrcdir/.eslintignore``. If you don't wish your directory to be linted, it +must be added here. + +The global configuration file lives in ``topsrcdir/.eslintrc``. This global configuration can be +overridden by including an ``.eslintrc`` in the appropriate subdirectory. For an overview of the +supported configuration, see `ESLint's documentation <http://eslint.org/docs/user-guide/configuring>`__. + + +Autofix +------- + +The eslint linter provides a ``--fix`` option. It is based on the upstream option. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/eslint.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/eslint/__init__.py>`_ + + +ESLint Plugin Mozilla +--------------------- + +In addition to default ESLint rules, there are several Mozilla-specific rules that are defined in +the :doc:`Mozilla ESLint Plugin <eslint-plugin-mozilla>`. + +ESLint Plugin SpiderMonkey JS +----------------------------- + +In addition to default ESLint rules, there is an extra processor for SpiderMonkey +code :doc:`Mozilla ESLint SpiderMonkey JS <eslint-plugin-spidermonkey-js>`. + + +.. toctree:: + :hidden: + + eslint-plugin-mozilla + eslint-plugin-spidermonkey-js diff --git a/docs/code-quality/lint/linters/file-perm.rst b/docs/code-quality/lint/linters/file-perm.rst new file mode 100644 index 0000000000..5c3a02fa1b --- /dev/null +++ b/docs/code-quality/lint/linters/file-perm.rst @@ -0,0 +1,42 @@ +File permission +=============== + +This linter verifies if a file has unnecessary permissions. +If a file has execution permissions (+x), file-perm will +generate a warning. + +It will ignore files starting with ``#!`` for types of files +that typically have shebang lines (such as python, node or +shell scripts). + +This linter does not have any affect on Windows. + + +Run Locally +----------- + +This mozlint linter can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter file-perm <file paths> + + +Configuration +------------- + +This linter is enabled on the whole code base. + +This job is configured as `tier 2 <https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Overview_of_the_Job_Visibility_Tiers>`_. + +Autofix +------- + +This linter provides a ``--fix`` option. The python script is doing the change itself. + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/file-perm.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/file-perm/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/file-whitespace.rst b/docs/code-quality/lint/linters/file-whitespace.rst new file mode 100644 index 0000000000..fd134b32ef --- /dev/null +++ b/docs/code-quality/lint/linters/file-whitespace.rst @@ -0,0 +1,38 @@ +Trailing whitespaces +==================== + +This linter verifies if a file has: + +* unnecessary trailing whitespaces, +* Windows carriage return, +* empty lines at the end of file, +* if file ends with a newline or not + + +Run Locally +----------- + +This mozlint linter can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter file-whitespace <file paths> + + +Configuration +------------- + +This linter is enabled on most of the code base. + +This job is configured as `tier 2 <https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Overview_of_the_Job_Visibility_Tiers>`_. + +Autofix +------- + +This linter provides a ``--fix`` option. The python script is doing the change itself. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/file-whitespace.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/file-whitespace/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/flake8.rst b/docs/code-quality/lint/linters/flake8.rst new file mode 100644 index 0000000000..0e46d33ab0 --- /dev/null +++ b/docs/code-quality/lint/linters/flake8.rst @@ -0,0 +1,46 @@ +Flake8 +====== + +`Flake8 <https://flake8.pycqa.org/en/latest/index.html>`__ is a popular lint wrapper for python. Under the hood, it runs three other tools and +combines their results: + +* `pep8 <http://pep8.readthedocs.io/en/latest/>`__ for checking style +* `pyflakes <https://github.com/pyflakes/pyflakes>`__ for checking syntax +* `mccabe <https://github.com/pycqa/mccabe>`__ for checking complexity + + +Run Locally +----------- + +The mozlint integration of flake8 can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter flake8 <file paths> + +Alternatively, omit the ``--linter flake8`` and run all configured linters, which will include +flake8. + + +Configuration +------------- + +Path configuration is defined in the root `.flake8`_ file. Please update this file rather than +``tools/lint/flake8.yml`` if you need to exclude a new path. For an overview of the supported +configuration, see `flake8's documentation`_. + +.. _.flake8: https://searchfox.org/mozilla-central/source/.flake8 +.. _flake8's documentation: https://flake8.pycqa.org/en/latest/user/configuration.html + +Autofix +------- + +The flake8 linter provides a ``--fix`` option. It is based on `autopep8 <https://github.com/hhatto/autopep8>`__. +Please note that autopep8 does NOT fix all issues reported by flake8. + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/flake8.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/python/flake8.py>`_ diff --git a/docs/code-quality/lint/linters/l10n.rst b/docs/code-quality/lint/linters/l10n.rst new file mode 100644 index 0000000000..f8beb246d1 --- /dev/null +++ b/docs/code-quality/lint/linters/l10n.rst @@ -0,0 +1,45 @@ +L10n +==== + +The l10n linter checks for mistakes and problems in the localizable files. +Most of the code lives inside the +`compare-locales <https://pypi.org/project/compare-locales/>`_ +package, and is shipping as the ``moz-l10n-lint`` command. + +The linter checks for fundamental issues like parsing errors, but it also +finds more subtle mistakes like duplicated messages. It also warns if you're +trying to change a string without changing the ID, or to add a string that's +still in use in a stable channel with a different value. + +The warnings on string ID changes get reported on phabricator, but they're +not making the build fail. To find out when to change IDs and when not to, +read the :ref:`Lifecycle & Workflow <Localization>` section in the +localization documentation. + +Run Locally +----------- + +The can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter l10n <file paths> + +Alternatively, omit the ``--linter l10n`` and run all configured linters, which +will include the l10n linter. + + +Updating the Reference +---------------------- + +The linter checks out the cross-channel localization files into your +``.mozbuild`` state directory. By default this is updated automatically after +48 hours. There might be new strings anyway, if you want to ensure an +updated clone, remove the marker file in +``~/.mozbuild/gecko-strings/.hg/l10n_pull_marker``. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/l10n.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/python/l10n_lint.py>`_ diff --git a/docs/code-quality/lint/linters/license.rst b/docs/code-quality/lint/linters/license.rst new file mode 100644 index 0000000000..0533333218 --- /dev/null +++ b/docs/code-quality/lint/linters/license.rst @@ -0,0 +1,39 @@ +License +======= + +This linter verifies if a file has a known license header. + +By default, Firefox uses MPL-2 license with the `appropriate headers <https://www.mozilla.org/en-US/MPL/headers/>`_. +In some cases (thirdpardy code), a file might have a different header file. +If this is the case, one of the significant line of the header should be listed in the list `of valid licenses +<https://searchfox.org/mozilla-central/source/tools/lint/license/valid-licenses.txt>`_. + +Run Locally +----------- + +This mozlint linter can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter license <file paths> + + +Configuration +------------- + +This linter is enabled on most of the whole code base. + +Autofix +------- + +This linter provides a ``--fix`` option. The python script is doing the change itself +and will use the right header MPL-2 header depending on the language. +It will add the license at the right place in case the file is a script (ie starting with ``!#`` +or a XML file ``<?xml>``). + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/license.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/license/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/lintpref.rst b/docs/code-quality/lint/linters/lintpref.rst new file mode 100644 index 0000000000..1b2236e10a --- /dev/null +++ b/docs/code-quality/lint/linters/lintpref.rst @@ -0,0 +1,31 @@ +Lintpref +======== + +The lintpref linter is a simple linter for libpref files to check for duplicate +entries between ``modules/libpref/init/all.js`` and ``modules/libpref/init/StaticPrefList.yaml``. +If a duplicate is found, lintpref will raise an error and emit the ``all.js`` line +where you can find the duplicate entry. + + +Running Locally +--------------- + +The linter can be run using mach: + + .. parsed-literal:: + + $ mach lint --linter lintpref + + +Fixing Lintpref Errors +---------------------- + +In most cases, duplicate entries should be avoided and the duplicate removed +from ``all.js``. If for any reason a pref should exist in both files, the pref +should be added to ``IGNORE_PREFS`` in ``tools/lint/libpref/__init__.py``. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/lintpref.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/libpref/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/mingw-capitalization.rst b/docs/code-quality/lint/linters/mingw-capitalization.rst new file mode 100644 index 0000000000..df82a9c59d --- /dev/null +++ b/docs/code-quality/lint/linters/mingw-capitalization.rst @@ -0,0 +1,28 @@ +MinGW capitalization +==================== + +This linter verifies that Windows include file are lowercase. +It might break the mingw build otherwise. + + +Run Locally +----------- + +This mozlint linter can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter mingw-capitalization <file paths> + + +Configuration +------------- + +This linter is enabled on the whole code base except WebRTC + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/mingw-capitalization.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/cpp/mingw-capitalization.py>`_ diff --git a/docs/code-quality/lint/linters/perfdocs.rst b/docs/code-quality/lint/linters/perfdocs.rst new file mode 100644 index 0000000000..851c920987 --- /dev/null +++ b/docs/code-quality/lint/linters/perfdocs.rst @@ -0,0 +1,84 @@ +PerfDocs +======== + +`PerfDocs`_ is a tool that checks to make sure all performance tests are documented in tree. + +At the moment, it is only used for this documentation verification, but in the future it will also auto-generate documentation from these descriptions that will be displayed in the source-docs documentation page (rather than the wiki, which is where they currently reside). + +Run Locally +----------- + +The mozlint integration of PerfDocs can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter perfdocs + + +Configuration +------------- + +There are no configuration options available for this linter. It scans the full source tree under ``testing``, looking for folders named ``perfdocs`` and then validates their content. This has only been implemented for Raptor so far, but Talos will be added in the future. We also hope to expand this to search outside the ``testing`` directory. + +The ``perfdocs`` folders, there needs to be an ``index.rst`` file and it needs to contain the string ``{documentation}`` in some location in the file which is where the test documentation will be placed. The folders must also have a ``config.yml`` file following this schema: + +.. code-block:: python + + CONFIG_SCHEMA = { + "type": "object", + "properties": { + "name": {"type": "string"}, + "manifest": {"type": "string"}, + "suites": { + "type": "object", + "properties": { + "suite_name": { + "type": "object", + "properties": { + "tests": { + "type": "object", + "properties": { + "test_name": {"type": "string"}, + } + }, + "description": {"type": "string"}, + }, + "required": [ + "description" + ] + } + } + } + }, + "required": [ + "name", + "manifest", + "suites" + ] + } + +Here is an example of a configuration file for the Raptor framework: + +.. parsed-literal:: + + name: raptor + manifest: testing/raptor/raptor/raptor.ini + suites: + desktop: + description: "Desktop tests." + tests: + raptor-tp6: "Raptor TP6 tests." + mobile: + description: "Mobile tests" + benchmarks: + description: "Benchmark tests." + tests: + wasm: "All wasm tests." + +Note that there needs to be a FrameworkGatherer implemented for the framework being documented since each of them may have different ways of parsing test manifests for the tests. See `RaptorGatherer <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs/framework_gatherers.py>`_ for an example gatherer that was implemented for Raptor. + +Sources +------- + +* `Configuration <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs.yml>`__ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs>`__ diff --git a/docs/code-quality/lint/linters/pylint.rst b/docs/code-quality/lint/linters/pylint.rst new file mode 100644 index 0000000000..603f80516a --- /dev/null +++ b/docs/code-quality/lint/linters/pylint.rst @@ -0,0 +1,33 @@ +pylint +====== + +`pylint <https://www.pylint.org/>`__ is a popular linter for python. It is now the default python +linter in VS Code. + +Please note that we also have :ref:`Flake8` available as a linter. + +Run Locally +----------- + +The mozlint integration of pylint can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter pylint <file paths> + + + +Configuration +------------- + +To enable pylint on new directory, add the path to the include +section in the `pylint.yml <https://searchfox.org/mozilla-central/source/tools/lint/pylint.yml>`_ file. + +We enabled the same Pylint rules as `VS Code <https://code.visualstudio.com/docs/python/linting#_pylint>`_. +See in `pylint.py <https://searchfox.org/mozilla-central/source/tools/lint/python/pylint.py>`_ for the full list + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/pylint.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/python/pylint.py>`_ diff --git a/docs/code-quality/lint/linters/rejected-words.rst b/docs/code-quality/lint/linters/rejected-words.rst new file mode 100644 index 0000000000..c952b420da --- /dev/null +++ b/docs/code-quality/lint/linters/rejected-words.rst @@ -0,0 +1,27 @@ +Rejected words +============== + +Reject some words we don't want to use in the code base for various reasons. + +Run Locally +----------- + +The mozlint integration of codespell can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter rejected-words <file paths> + + +Configuration +------------- + +This linter is enabled on the whole code base. Issues existing in the code base are listed in the exclude list in + the `rejected-words.yml <https://searchfox.org/mozilla-central/source/tools/lint/rejected-words.yml>`_ file. + +New words can be added in the `payload` section. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/rejected-words.yml>`_ diff --git a/docs/code-quality/lint/linters/rstlinter.rst b/docs/code-quality/lint/linters/rstlinter.rst new file mode 100644 index 0000000000..6ed4f1befe --- /dev/null +++ b/docs/code-quality/lint/linters/rstlinter.rst @@ -0,0 +1,32 @@ +RST Linter +========== + +`rstcheck`_ is a popular linter for restructuredtext. + + +Run Locally +----------- + +The mozlint integration of rst linter can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter rst <file paths> + + +Configuration +------------- + +All directories will have rst linter run against them. +If you wish to exclude a subdirectory of an included one, you can add it to the ``exclude`` +directive. + + +.. _rstcheck: https://github.com/myint/rstcheck + + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/rst.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/rst/__init__.py>`_ diff --git a/docs/code-quality/lint/linters/rustfmt.rst b/docs/code-quality/lint/linters/rustfmt.rst new file mode 100644 index 0000000000..7ba3cd6c34 --- /dev/null +++ b/docs/code-quality/lint/linters/rustfmt.rst @@ -0,0 +1,33 @@ +Rustfmt +======= + +`rustfmt <https://github.com/rust-lang/rustfmt>`__ is the tool for Rust coding style. + +Run Locally +----------- + +The mozlint integration of rustfmt can be run using mach: + +.. parsed-literal:: + + $ mach lint --linter rustfmt <file paths> + + +Configuration +------------- + +To enable rustfmt on new directory, add the path to the include +section in the `rustfmt.yml <https://searchfox.org/mozilla-central/source/tools/lint/rustfmt.yml>`_ file. + + +Autofix +------- + +Rustfmt is reformatting the code by default. To highlight the results, we are using +the ``--check`` option. + +Sources +------- + +* `Configuration (YAML) <https://searchfox.org/mozilla-central/source/tools/lint/rustfmt.yml>`_ +* `Source <https://searchfox.org/mozilla-central/source/tools/lint/rustfmt/__init__.py>`_ diff --git a/docs/code-quality/lint/usage.rst b/docs/code-quality/lint/usage.rst new file mode 100644 index 0000000000..9c128383c1 --- /dev/null +++ b/docs/code-quality/lint/usage.rst @@ -0,0 +1,117 @@ +Running Linters Locally +======================= + +Using the Command Line +---------------------- + +You can run all the various linters in the tree using the ``mach lint`` command. Simply pass in the +directory or file you wish to lint (defaults to current working directory): + +.. parsed-literal:: + + ./mach lint path/to/files + +Multiple paths are allowed: + +.. parsed-literal:: + + ./mach lint path/to/foo.js path/to/bar.py path/to/dir + +To force execution on a directory that would otherwise be excluded: + +.. parsed-literal:: + + ./mach lint -n path/in/the/exclude/list + +``Mozlint`` will automatically determine which types of files exist, and which linters need to be run +against them. For example, if the directory contains both JavaScript and Python files then mozlint +will automatically run both ESLint and Flake8 against those files respectively. + +To restrict which linters are invoked manually, pass in ``-l/--linter``: + +.. parsed-literal:: + + ./mach lint -l eslint path/to/files + +You can see a list of the available linters by running: + +.. parsed-literal:: + + ./mach lint --list + +Finally, ``mozlint`` can lint the files touched by outgoing revisions or the working directory using +the ``-o/--outgoing`` and ``-w/--workdir`` arguments respectively. These work both with mercurial and +git. In the case of ``--outgoing``, the default remote repository the changes would be pushed to is +used as the comparison. If desired, a remote can be specified manually. In git, you may only want to +lint staged commits from the working directory, this can be accomplished with ``--workdir=staged``. +Examples: + +.. parsed-literal:: + + ./mach lint --workdir + ./mach lint --workdir=staged + ./mach lint --outgoing + ./mach lint --outgoing origin/master + ./mach lint -wo + + +Using a VCS Hook +---------------- + +There are also both pre-commit and pre-push version control hooks that work in +either hg or git. To enable a pre-push hg hook, add the following to hgrc: + +.. parsed-literal:: + + [hooks] + pre-push.lint = python:./tools/lint/hooks.py:hg + + +To enable a pre-commit hg hook, add the following to hgrc: + +.. parsed-literal:: + + [hooks] + pretxncommit.lint = python:./tools/lint/hooks.py:hg + + +To enable a pre-push git hook, run the following command: + +.. parsed-literal:: + + $ ln -s /path/to/gecko/tools/lint/hooks.py .git/hooks/pre-push + + +To enable a pre-commit git hook, run the following command: + +.. parsed-literal:: + + $ ln -s /path/to/gecko/tools/lint/hooks.py .git/hooks/pre-commit + + +Fixing Lint Errors +================== + +``Mozlint`` has a best-effort ability to fix lint errors: + +.. parsed-literal:: + + $ ./mach lint --fix + +Not all linters support fixing, and even the ones that do can not usually fix +all types of errors. Any errors that cannot be automatically fixed, will be +printed to stdout like normal. In that case, you can also fix errors manually: + +.. parsed-literal:: + + $ ./mach lint --edit + +This requires the $EDITOR environment variable be defined. For most editors, +this will simply open each file containing errors one at a time. For vim (or +neovim), this will populate the `quickfix list`_ with the errors. + +The ``--fix`` and ``--edit`` arguments can be combined, in which case any +errors that can be fixed automatically will be, and the rest will be opened +with your $EDITOR. + +.. _quickfix list: http://vimdoc.sourceforge.net/htmldoc/quickfix.html diff --git a/docs/code-quality/static-analysis.rst b/docs/code-quality/static-analysis.rst new file mode 100644 index 0000000000..21565e6969 --- /dev/null +++ b/docs/code-quality/static-analysis.rst @@ -0,0 +1,251 @@ +Static analysis +=============== + +This document is split in two parts. The first part will focus on the +modern and robust way of static-analysis and the second part will +present the build-time static-analysis. + +For linting, please see the `linting documentation </code-quality/lint/>`_. + +For reviews, use the `#static-analysis-reviewers review group <https://phabricator.services.mozilla.com/project/view/120/>`__. +Ask questions on `#static-analysis:mozilla.org <https://chat.mozilla.org/#/room/#static-analysis:mozilla.org>`__. + + +Clang-Tidy static analysis +-------------------------- + +Our current static-analysis infrastructure is based on +`clang-tidy <http://clang.llvm.org/extra/clang-tidy/>`__. It uses +checkers in order to assert different programming errors present in the +code. The checkers that we use are split into 3 categories: + +#. `Firefox specific checkers <https://searchfox.org/mozilla-central/source/build/clang-plugin>`_. They detect incorrect Gecko programming + patterns which could lead to bugs or security issues. +#. `Clang-tidy checkers <https://clang.llvm.org/extra/clang-tidy/checks/list.html>`_. They aim to suggest better programming practices + and to improve memory efficiency and performance. +#. `Clang-analyzer checkers <https://clang-analyzer.llvm.org/>`_. These checks are more advanced, for example + some of them can detect dead code or memory leaks, but as a typical + side effect they have false positives. Because of that, we have + disabled them for now, but will enable some of them in the near + future. + +In order to simplify the process of static-analysis we have focused on +integrating this process with Phabricator and mach. A list of some +checkers that are used during automated scan can be found +`here <https://searchfox.org/mozilla-central/source/tools/clang-tidy/config.yaml>`__. + +We are also running Coverity at review phase and a few times per day +(when autoland is merged into mozilla-central). + +Static analysis at review phase +------------------------------- + +We created a TaskCluster bot that runs clang static analysis on every +patch submitted to Phabricator. It then quickly reports any code defects +directly on the review platform, thus preventing bad patches from +landing until all their defects are fixed. Currently, its feedback is +posted in about 10 minutes after a patch series is published on the +review platform. + +As part of the process, the various linting jobs are also executed +using try. This can be also used to add new jobs, see: :ref:`attach-job-review`. +An example of automated review can be found `on +phabricator <https://phabricator.services.mozilla.com/D2066>`__. + + +Mach static analysis +-------------------- + +It is supported on all Firefox built platforms. During the first run it +automatically installs all of its dependencies like clang-tidy +executable in the .mozbuild folder thus making it very easy to use. The +resources that are used are provided by toolchain artifacts clang-tidy +target. + +This is used through ``mach static-analysis`` command that has the +following parameters: + +- ``check`` - Runs the checks using the installed helper tool from + ~/.mozbuild. +- ``--checks, -c`` - Checks to enabled during the scan. The checks + enabled + `in the yaml file <https://searchfox.org/mozilla-central/source/tools/clang-tidy/config.yaml>`__ + are used by default. +- ``--fix, -f`` - Try to autofix errors detected by the checkers. + Depending on the checker, this option might not do anything. + The list of checkers with autofix can be found on the `clang-tidy website <https://clang.llvm.org/extra/clang-tidy/checks/list.html>`__. +- ``--header-filter, -h-f`` - Regular expression matching the names of + the headers to output diagnostic from.Diagnostic from the main file + of each translation unit are always displayed. + +As an example we run static-analysis through mach on +``dom/presentation/Presentation.cpp`` with +``google-readability-braces-around-statements`` check and autofix we +would have: + +.. code-block:: shell + + ./mach static-analysis check --checks="-*, google-readability-braces-around-statements" --fix dom/presentation/Presentation.cpp + +If you want to use a custom clang-tidy binary this can be done by using +the ``install`` subcommand of ``mach static-analysis``, but please note +that the archive that is going to be used must be compatible with the +directory structure clang-tidy from toolchain artifacts. + +.. code-block:: shell + + ./mach static-analysis install clang.tar.gz + + +Regression Testing +------------------ + +In order to prevent regressions in our clang-tidy based static analysis, +we have created a +`task <https://searchfox.org/mozilla-central/source/taskcluster/ci/static-analysis-autotest/kind.yml>`__ +on automation. This task runs on each commit and launches a test suite +that is integrated into mach. + +The test suite implements the following: + +- Downloads the necessary clang-tidy artifacts. +- Reads the + `configuration <https://searchfox.org/mozilla-central/source/tools/clang-tidy/config.yaml>`__ + file. +- For each checker reads the test file plus the expected result. A + sample of test and expected result can be found + `in the test file <https://searchfox.org/mozilla-central/source/tools/clang-tidy/test/clang-analyzer-deadcode.DeadStores.cpp>`__ + and + `the json file <https://searchfox.org/mozilla-central/source/tools/clang-tidy/test/clang-analyzer-deadcode.DeadStores.json>`__. + +This testing suit can be run locally by doing the following: + +.. code-block:: shell + + ./mach static-analysis autotest + +If we want to test only a specific checker, let's say +modernize-raw-string-literal, we can run: + +.. code-block:: shell + + ./mach static-analysis autotest modernize-raw-string-literal + +If we want to add a new checker we need to generated the expected result +file, by doing: + +.. code-block:: shell + + ./mach static-analysis autotest modernize-raw-string-literal -d + + +Build-time static-analysis +-------------------------- + +If you want to build with the Firefox Clang plug-in +(located in ``/build/clang-plugin`` and associated with +``MOZ_CLANG_PLUGIN`` and the attributes in ``/mfbt/Attributes.h``) +just add ``--enable-clang-plugin`` to your mozconfig! +If you want to also have our experimental checkers that will produce ``warnings`` as +diagnostic messages also add ``--enable-clang-plugin-alpha``. +This requires to build Firefox using Clang. + +Configuring the build environment +--------------------------------- + +Once you have your Clang build in place, you will need to set up tools +to use it. +A full working .mozconfig for the desktop browser is: + +.. code-block:: shell + + . $topsrcdir/browser/config/mozconfig + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-ff-dbg + + ac_add_options --enable-debug + +Attempts to use ``ccache`` will likely result in failure to compile. It +is also necessary to avoid optimized builds, as these will modify macros +which will result in many false positives. + +At this point, your Firefox build environment should be configured to +compile via the Clang static analyzer! + + +Performing scanning builds +-------------------------- + +It is not enough to simply start the build like normal. Instead, you +need to run the build through a Clang utility script which will keep +track of all produced analysis and consolidate it automatically. + +Reports are published daily on +`https://sylvestre.ledru.info/reports/fx-scan-build/ <http://sylvestre.ledru.info/reports/fx-scan-build/>`__ +Many of the defects reported as sources for Good First Bug. + +That script is scan-build. You can find it in +``$clang_source/tools/scan-build/scan-build``. + +Try running your build through ``scan-build``: + +.. code-block:: shell + + $ cd /path/to/mozilla/source + + # Blow away your object directory because incremental builds don't make sense + $ rm -rf obj-dir + + # To start the build: + scan-build --show-description ./mach build -v + + # The above should execute without any errors. However, it should take longer than + # normal because all compilation will be executing through Clang's static analyzer, + # which adds overhead. + +If things are working properly, you should see a bunch of console spew, +just like any build. + +The first time you run scan-build, CTRL+C after a few files are +compiled. You should see output like: + +.. code-block:: shell + + scan-build: 3 bugs found. + scan-build: Run 'scan-view /Users/gps/tmp/mcsb/2011-12-15-3' to examine bug reports. + +If you see a message like: + +.. code-block:: shell + + scan-build: Removing directory '/var/folders/s2/zc78dpsx2rz6cpc_21r9g5hr0000gn/T/scan-build-2011-12-15-1' because it contains no reports. + +either no static analysis results were available yet or your environment +is not configured properly. + +By default, ``scan-build`` writes results to a folder in a +pseudo-temporary location. You can control where results go by passing +the ``-o /path/to/output`` arguments to ``scan-build``. + +You may also want to run ``scan-build --help`` to see all the options +available. For example, it is possible to selectively enable and disable +individual analyzers. + + +Analyzing the output +-------------------- + +Once the build has completed, ``scan-build`` will produce a report +summarizing all the findings. This is called ``index.html`` in the +output directory. You can run ``scan-view`` (from +``$clang_source/tools/scan-view/scan-view``) as ``scan-build's`` output +suggests; this merely fires up a local HTTP server. Or you should be +able to open the ``index.html`` directly with your browser. + + +False positives +--------------- + +By definition, there are currently false positives in the static +analyzer. A lot of these are due to the analyzer having difficulties +following the relatively complicated error handling in various +preprocessor macros. diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000000..16683256f7 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,127 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. + +from __future__ import absolute_import, unicode_literals + +import os +import sys + +from recommonmark.transform import AutoStructify + +# Set up Python environment to load build system packages. +OUR_DIR = os.path.dirname(__file__) +topsrcdir = os.path.normpath(os.path.join(OUR_DIR, "..")) + +# Escapes $, [, ] and 3 dots in copy button +copybutton_prompt_text = r">>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: " +copybutton_prompt_is_regexp = True + +EXTRA_PATHS = ( + "layout/tools/reftest", + "python/mach", + "python/mozbuild", + "python/mozversioncontrol", + "testing/mozbase/manifestparser", + "testing/mozbase/mozfile", + "testing/mozbase/mozprocess", + "third_party/python/futures", + "third_party/python/jsmin", + "third_party/python/which", + "toolkit/components/glean/sphinx", +) + +sys.path[:0] = [os.path.join(topsrcdir, p) for p in EXTRA_PATHS] + +sys.path.insert(0, OUR_DIR) + +extensions = [ + "sphinx.ext.autodoc", + "sphinx.ext.autosectionlabel", + "sphinx.ext.doctest", + "sphinx.ext.graphviz", + "sphinx.ext.napoleon", + "sphinx.ext.todo", + "mozbuild.sphinx", + "sphinx_js", + "sphinxcontrib.mermaid", + "recommonmark", + "sphinx_copybutton", + "sphinx_markdown_tables", + "glean", +] + +# JSDoc must run successfully for dirs specified, so running +# tree-wide (the default) will not work currently. +js_source_path = [ + "../browser/components/extensions", + "../browser/components/uitour", + "../testing/marionette", + "../toolkit/components/extensions", + "../toolkit/components/extensions/parent", + "../toolkit/components/featuregates", + "../toolkit/mozapps/extensions", + "../toolkit/components/prompts/src", +] +root_for_relative_js_paths = ".." +jsdoc_config_path = "jsdoc.json" + +templates_path = ["_templates"] +source_suffix = [".rst", ".md"] +master_doc = "index" +project = "Firefox Source Docs" +html_logo = os.path.join( + topsrcdir, "browser/branding/nightly/content/firefox-wordmark.svg" +) +html_favicon = os.path.join(topsrcdir, "browser/branding/nightly/firefox.ico") + +exclude_patterns = ["_build", "_staging", "_venv"] +pygments_style = "sphinx" + +# We need to perform some adjustment of the settings and environment +# when running on Read The Docs. +on_rtd = os.environ.get("READTHEDOCS", None) == "True" + +if on_rtd: + # SHELL isn't set on RTD and mach.mixin.process's import raises if a + # shell-related environment variable can't be found. Set the variable here + # to hack us into working on RTD. + assert "SHELL" not in os.environ + os.environ["SHELL"] = "/bin/bash" +else: + # We only need to set the RTD theme when not on RTD because the RTD + # environment handles this otherwise. + import sphinx_rtd_theme + + html_theme = "sphinx_rtd_theme" + html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] + + +html_static_path = ["_static"] +htmlhelp_basename = "MozillaTreeDocs" + +moz_project_name = "main" + +html_show_copyright = False + +# Only run autosection for the page title. +# Otherwise, we have a huge number of duplicate links. +# For example, the page https://firefox-source-docs.mozilla.org/code-quality/lint/ +# is called "Linting" +# just like https://firefox-source-docs.mozilla.org/remote/CodeStyle.html +autosectionlabel_maxdepth = 1 + + +def setup(app): + app.add_config_value( + "recommonmark_config", + { + # Crashes with sphinx + "enable_inline_math": False, + # We use it for testing/web-platform/tests + "enable_eval_rst": True, + }, + True, + ) + app.add_stylesheet("custom_theme.css") + app.add_transform(AutoStructify) diff --git a/docs/config.yml b/docs/config.yml new file mode 100644 index 0000000000..dcfb107fa9 --- /dev/null +++ b/docs/config.yml @@ -0,0 +1,83 @@ +--- + +# The order of the main categories are defined in index.rst +# Sub categories orders are preserved +categories: + setup_doc: + - setup + contributing_doc: + - contributing + - bug-mgmt + source_doc: + - browser + - dom + - editor + - layout + - gfx + - devtools + - toolkit + - js + - mobile/android/geckoview + - dom/bindings/webidl + - modules/libpref + - remote + - services/common/services + - uriloader + - widget/cocoa + - code-quality + - writing-rust-code + - tools/profiler + build_doc: + - mach + - tools/try + - build/buildsystem + - taskcluster + - tools/moztreedocs + testing_doc: + - testing/testing-policy + - testing/ci-configs + - testing/marionette + - testing/geckodriver + - web-platform + - tools/fuzzing + - tools/sanitizer + - testing/perfdocs + - tools/code-coverage + - testing-rust-code + l10n_doc: + - intl + - l10n + python_doc: + - mozbase + - python + fennec_doc: + - mobile/android + metrics_doc: + - metrics + +redirects: + browser/browser: browser + contributing/how_to_contribute_firefox.html: contributing/contribution_quickref.html + contributing/artifact_builds.html: contributing/build/artifact_builds.html + contributing/linux_build.html: setup/linux_build.html + contributing/build/linux_build.html: setup/linux_build.html + contributing/mercurial.html: contributing/vcs/mercurial.html + contributing/mercurial_bundles.html: contributing/vcs/mercurial_bundles.html + dom/dom: dom + layout/layout: layout + gfx/gfx: gfx + intl/l10n/l10n: l10n + modules/libpref/libpref: modules/libpref + python/mach: mach + python/python: python + taskcluster/taskcluster: taskcluster + testing/geckodriver/geckodriver: testing/geckodriver + testing/marionette/marionette: testing/marionette + toolkit/components/telemetry/telemetry: toolkit/components/telemetry + tools/compare-locales/index.html: build/buildsystem/locales.html + tools/docs/index.html: tools/moztreedocs/index.html + tools/docs/contribute/how_to_contribute_firefox.html: contributing/how_to_contribute_firefox.html + tools/docs/contribute/directory_structure.html: contributing/directory_structure.html + tools/lint: code-quality/lint + tools/lint/coding-style: code-quality/coding-style + tools/static-analysis/index.html: code-quality/static-analysis.html diff --git a/docs/contributing/Code_Review_FAQ.rst b/docs/contributing/Code_Review_FAQ.rst new file mode 100644 index 0000000000..18fe85a6e2 --- /dev/null +++ b/docs/contributing/Code_Review_FAQ.rst @@ -0,0 +1,93 @@ +Code Review FAQ +=============== + +What is the purpose of code review? +----------------------------------- + +Code review is our basic mechanism for validating the design and +implementation of patches. It also helps us maintain a level of +consistency in design and implementation practices across the many +hackers and among the various modules of Mozilla. + +Of course, code review doesn't happen instantaneously, and so there is +some latency built into the system. We're always looking for ways to +reduce the wait, while simultaneously allowing reviewers to do a good +chunk of hacking themselves. We don't have a perfect system, and we +never will. It's still evolving, so let us know if you have suggestions. + +Mozilla used to have the concept of "super-review", but `a consensus was +reached in +2018 <https://groups.google.com/forum/#!topic/mozilla.governance/HHU0h-44NDo>`__ +to retire this process. + +Who must review my code? +------------------------ + +You must have an approval ("r={{ mediawiki.external('name') }}") from +the module owner or designated "peer" of the module where the code will +be checked in. If your code affects several modules, then generally you +should have an "r={{ mediawiki.external('name') }}" from the owner or +designated peer of each affected module. We try to be reasonable here, +so we don't have an absolute rule on when every module owner must +approve. For example, tree-wide changes such as a change to a string +class or a change to text that is displayed in many modules generally +doesn't get reviewed by every module owner. + +You may wish to ask others as well. + + +What do reviewers look for? +--------------------------- + +A review is focused on a patch's design, implementation, usefulness in +fixing a stated problem, and fit within its module. A reviewer should be +someone with domain expertise in the problem area. A reviewer may also +utilize other areas of his or her expertise and comment on other +possible improvements. There are no inherent limitations on what +comments a reviewer might make about improving the code. + +Reviewers will probably look at the following areas of the code: + +- “goal” review: is the issue being fixed actually a bug? Does the + patch fix the fundamental problem? +- API/design review. Because APIs define the interactions between + modules, they need special care. Review is especially important to + keep APIs balanced and targeted, and not too specific or + overdesigned. There are a `WebIDL review + checklist <https://wiki.mozilla.org/WebAPI/WebIDL_Review_Checklist>`__. + There are also templates for emails that should be sent when APIs are + going to be exposed to the Web and general guidance around naming on + `this wiki + page <https://wiki.mozilla.org/WebAPI/ExposureGuidelines>`__. +- Maintainability review. Code which is unreadable is impossible to + maintain. If the reviewer has to ask questions about the purpose of a + piece of code, then it is probably not documented well enough. Does + the code follow the :ref:`Coding style` ? Be careful when + reviewing code using modern C++ features like auto. +- Security review. Does the design use security concepts such as input + sanitizers, wrappers, and other techniques? Does this code need + additional security testing such as fuzz-testing or static analysis? +- Integration review. Does this code work properly with other modules? + Is it localized properly? Does it have server dependencies? Does it + have user documentation? +- Testing review. Are there tests for correct function? Are there tests + for error conditions and incorrect inputs which could happen during + operation? +- Performance review. Has this code been profiled? Are you sure it's + not negatively affecting performance of other code? +- License review. Does the code follow the `code licensing + rules <http://www.mozilla.org/hacking/committer/committers-agreement.pdf>`__? + + +How can I tell the status of reviews? +------------------------------------- + +When a patch has passed review you'll see "Accepted" in green at the top +of a Phabricator revision, under the title. In Bugzilla (which is +deprecated in favour of Phabricator), this is indicated by "{{ +mediawiki.external('name') }}:review+" in the attachment table in the +bug report. If it has failed review then you'll see "Needs Revision" in +red at the top of the revision, or, in Bugzilla, "{{ +mediawiki.external('name') }}:review-". Most of the time that a reviewer +sets a review flag, they will also add a comment to the bug explaining +the review. diff --git a/docs/contributing/build/artifact_builds.rst b/docs/contributing/build/artifact_builds.rst new file mode 100644 index 0000000000..9c6d1a9daa --- /dev/null +++ b/docs/contributing/build/artifact_builds.rst @@ -0,0 +1,174 @@ +Understanding Artifact Builds +============================= + +Firefox for Desktop and Android supports a **fast build mode** called +*artifact mode*. The resulting builds are called *artifact builds*. +Artifact mode downloads pre-built C++ components rather than building them +locally, trading bandwidth for time. + +Artifact builds will be useful to many developers who are not working +with compiled code (see "Restrictions" below). Artifacts are typically +fetched from `mozilla-central <https://hg.mozilla.org/mozilla-central/>`__. + +To automatically download and use pre-built binary artifacts, add the +following lines into your :ref:`mozconfig <Configuring Build Options>` +file: + +.. code-block:: shell + + # Automatically download and use compiled C++ components: + ac_add_options --enable-artifact-builds + + # Write build artifacts to: + mk_add_options MOZ_OBJDIR=./objdir-frontend + +To automatically download and use the debug version of the pre-built +binary artifact (currently supported for Linux, OSX and Windows +artifacts), add ``ac_add_options --enable-debug`` to your mozconfig file +(with artifact builds option already enabled): + +.. code-block:: shell + + # Enable debug versions of the pre-built binary artifacts: + ac_add_options --enable-debug + + # Automatically download and use compiled C++ components: + ac_add_options --enable-artifact-builds + + # Download debug info so that stack traces refers to file and columns rather than library and Hex address + ac_add_options --enable-artifact-build-symbols + + # Write build artifacts to: + mk_add_options MOZ_OBJDIR=./objdir-frontend-debug-artifact + + +Prerequisites +------------- + +Artifact builds are supported for users of Mercurial and Git. Git +artifact builds require a mozilla-central clone made with the help of +`git-cinnabar <https://github.com/glandium/git-cinnabar>`__. Please +follow the instructions on the git-cinnabar project page to install +git-cinnabar. Further information about using git-cinnabar to interact +with Mozilla repositories can be found on `the project +wiki <https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development>`__. + +Building +-------- + +If you've added ``--enable-artifact-builds`` to your ``mozconfig``, each +time you run ``mach build`` and ``mach build path/to/subdirectory`` the +build system will determine what the best pre-built binary artifacts +available are, download them, and put them in place for you. The +computations are cached, so the additional calculations should be very +fast after the up-to-date artifacts are downloaded -- just a second or +two on modern hardware. Most Desktop developers should find that + +.. code-block:: shell + + ./mach build + ./mach run + +just works. + +To only rebuild local changes (to avoid re-checking for pushes and/or +unzipping the downloaded cached artifacts after local commits), you can +use: + +.. code-block:: shell + + ./mach build faster + +which only "builds" local JS, CSS and packaged (e.g. images and other +asset) files. + +Most Firefox for Android developers should find that + +.. code-block:: shell + + ./mach build + ./mach package + ./mach install + +just works. + +Pulling artifacts from a try build +---------------------------------- + +To only accept artifacts from a specific revision (such as a try build), +set ``MOZ_ARTIFACT_REVISION`` in your environment to the value of the +revision that is at the head of the desired push. Note that this will +override the default behavior of finding a recent candidate build with +the required artifacts, and will cause builds to fail if the specified +revision does not contain the required artifacts. + +Restrictions +------------ + +Oh, so many. Artifact builds are rather delicate: any mismatch between +your local source directory and the downloaded binary artifacts can +result in difficult to diagnose incompatibilities, including unexplained +crashes and catastrophic XPCOM initialization and registration +failures. These are rare, but do happen. + +Things that are supported +------------------------- + +- Modifying JavaScript, (X)HTML, and CSS resources; and string + properties and DTD and FTL files. +- Modifying Android Java code, resources, and strings. +- Running mochitests and xpcshell tests. +- Modifying ``Scalars.yaml`` to add Scalar Telemetry (since {{ + Bug("1425909") }}, except artifact builds on try). +- Modifying ``Events.yaml`` to add Event Telemetry (since {{ + Bug("1448945") }}, except artifact builds on try). + +Essentially everything updated by ``mach build faster`` should work with +artifact builds. + +Things that are not supported +----------------------------- + +- Support for products other than Firefox for Desktop and + Android are not supported and are unlikely to ever be supported. + Other projects like Thunderbird may provide + <a href="https://developer.thunderbird.net/thunderbird-development/building-thunderbird/artifact-builds">their own support</a> + for artifact builds. +- You cannot modify C, C++, or Rust source code anywhere in the tree. + If it’s compiled to machine code, it can't be changed. +- You cannot modify ``histograms.json`` to add Telemetry histogram + definitions.(But see `Bug 1206117 <https://bugzilla.mozilla.org/show_bug.cgi?id=1206117>`__). +- Modifying build system configuration and definitions does not work in + all situations. + +Things that are not **yet** supported +------------------------------------- + +- Tests other than mochitests, xpcshell, and Marionette-based tests. + There aren’t inherent barriers here, but these are not known to work. +- Modifying WebIDL definitions, even ones implemented in JavaScript. + +Troubleshooting +--------------- + +There are two parts to artifact mode: +the ``--disable-compile-environment`` option, and the ``mach artifact`` +command that implements the downloading and caching. Start by running + +.. code-block:: shell + + ./mach artifact install --verbose + +to see what the build system is trying to do. There is some support for +querying and printing the cache; run ``mach artifact`` to see +information about those commands. + +Downloaded artifacts are stored in +``$MOZBUILD_STATE_PATH/package-frontend``, which is almost always +``~/.mozbuild/package-frontend``. + +Discussion is best started on the `dev-builds mailing +list <https://lists.mozilla.org/listinfo/dev-builds>`__. Questions are +best raised in `#build <https://chat.mozilla.org/#/room/#build:mozilla.org>`__ on `Matrix <https://chat.mozilla.org/>`__. Please +file bugs in *Firefox Build System :: General*, blocking `Bug 901840 <https://bugzilla.mozilla.org/show_bug.cgi?id=901840>`__ + diff --git a/docs/contributing/build/building_mobile_firefox.rst b/docs/contributing/build/building_mobile_firefox.rst new file mode 100644 index 0000000000..a8826321f1 --- /dev/null +++ b/docs/contributing/build/building_mobile_firefox.rst @@ -0,0 +1,30 @@ +Firefox for Mobile Devices +-------------------------- + +We have several different mobile products aimed at different tasks, +devices, and audiences: + +- `Building Firefox for + Android <https://geckoview.dev/contributor/geckoview-quick-start>`_. + (Codename: Fennec) +- `Building Firefox for iOS <https://developer.mozilla.org/docs/Mozilla/Firefox_for_iOS>`_, + our general-purpose browser for iOS with desktop sync built-in. +- `Building Firefox + Focus <https://github.com/mozilla-mobile/focus>`_, our + privacy-focused browser for + `iOS <https://github.com/mozilla-mobile/focus-ios>`_ and + `Android <https://github.com/mozilla-mobile/focus-android>`_. + +For both Desktop and Mobile development, please bear the following in +mind: + +- While you can build Firefox on older hardware it can take quite a bit + of time to compile on slower machines. Having at least 8GB of RAM is + recommended, and more is always better. The build process is both CPU + and I/O intensive, so building on a machine with an SSD is also + strongly preferred. +- Fast broadband internet is strongly recommended as well. Both the + development environment and the source code repository are quite + large. +- Though you can build Firefox to run on 32-bit machines, the build + process for almost all of our products requires a 64-bit OS. diff --git a/docs/contributing/committing_rules_and_responsibilities.rst b/docs/contributing/committing_rules_and_responsibilities.rst new file mode 100644 index 0000000000..4c8c5f615d --- /dev/null +++ b/docs/contributing/committing_rules_and_responsibilities.rst @@ -0,0 +1,198 @@ +Committing rules and responsibilities +===================================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Preparation +----------- + +There are things you need to be sure of before you even attempt to check +in: + +- Your code must + :ref:`compile <Building Firefox On Linux>` and `pass all the automated tests <https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing>`__ + before you consider pushing changes. If you are at all unsure, verify + your changes with the + `mozilla-central <https://wiki.mozilla.org/Build:TryServer>`__. + try server, as appropriate. +- You need :ref:`code review <Code Review FAQ>`. +- Depending on the stage of the development process, you may need + `approval <https://wiki.mozilla.org/Tree_Rules>`__. Commits to trees + where approval is required must have "a=" in the commit message + followed by the name of the approver. +- Code should be factored in such a way such that we can disable + features which cause regressions, either by backout or via a kill + switch/preference. Be especially careful when landing features which + depend on other new features which may be disabled. Ask + mozilla.dev.planning for assistance if there are any questions. + +Checkin comment +--------------- + +The checkin comment for the change you push should include the bug +number, the names of the reviewers, and a clear explanation of the fix. +Please say what changes are made, not what problem was fixed, e.g.: + +Good: "Bug 123456 - Null-check presentation shell so we don't crash when a +button removes itself during its own onclick handler. r=paul, a=ringo." + +Bad: "Bug 123456 - crash clicking button on www.example.com" + +If you are not the author of the code, use ``hg commit -u`` to specify +the actual author in the Mercurial changeset: + +:: + + hg commit -u "Pat Chauthor <pat@chauthor.com>" + +Commit message restrictions +--------------------------- + +The purpose of these new restrictions, implemented via a mercurial hook, +is to prevent commit messages that do not have a bug number. We will +still allow a small set of special commits lacking bugs numbers, like +merges and backouts. + +This hook will be enabled on mozilla-central and every major branch that +directly merges into it, such as autoland or integration +branches, team branches, or established project branches. + +An example for a passing commit message would be, + +:: + + Bug 577872 - Create WebM versions of Ogg reftests. r=kinetik + +Note the *Bug ####*, you at least need that. You also can't commit +bustage-fixes without a bug number anymore. This is intentional to keep +track of the bug which caused it. + +Allowed are: + +- Commit messages containing "bug" or "b=" followed by a bug number +- Commit messages containing "no bug" (please use this sparingly) +- Commit message indicating backout of a given 12+ digit changeset ID, + starting with (back out|backing out|backed out|backout)( of)? + (rev|changeset|cset)s? [0-9a-f]{12} +- Commit messages that start with "merge" or "merging" and are actually + for a merge changeset. + +Special exceptions: + +- Commits by the special users "ffxbld", "seabld", "tbirdbld", or + "cltbld". +- When the commit is older then some date shortly after the hook has + been enabled, to allow merges from other branches. This exception + will be lifted after a short period of time (probably a few months) + after the hooks is enabled. +- You can also specify "IGNORE BAD COMMIT MESSAGES" in the tip (latest) + commit message to override all the restrictions. This is an extreme + measure, so you should only do this if you have a very good reason. + +Explicitly disallowed: + +- Commit messages containing "try: " to avoid unintentional commits + that were meant for the try server. + +All tests for allowed or excluded messages are case-insensitive. The +hook, +`commit-message.py <https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hghooks/mozhghooks/commit-message.py>`__, +was added in `bug 506949 <https://bugzilla.mozilla.org/show_bug.cgi?id=506949>`__. + + +Check the tree +-------------- + +TaskCluster is a continuous build system that builds and tests every change +checked into autoland/mozilla-central and related source trees. +`Treeherder <https://treeherder.mozilla.org/>`__ displays the progress +and results of all the build and test jobs for a given tree. For a +particular job, green means all is well, orange means tests have failed, +and red means the build itself broke. Purple means that a test was +interrupted, possibly by a problem with the build system or the +network. Blue means that a test was interrupted in a known way and will +be automatically restarted. You can click on the "Help" link in the top +right corner of Treeherder for a legend to help you decode all the other +colors and letters. + +If the tree is green, it is okay to check in. If some builds are orange +or red, you can either wait, or make sure all the failures are +classified with annotations/comments that reference bug numbers or +fixes. + +If the tree is marked as "closed", or if you have questions about any +oranges or reds, you should contact the sheriff before checking in. + + +Failures and backouts +--------------------- + +Patches which cause unit test failures (on :ref:`tier 1 +platforms <Supported build targets>`) will be backed out. +Regressions on tier-2 platforms and in performance are not cause for a +direct backout, but you will be expected to help fix them if quickly. + +*Note: Performance regressions require future data points to ensure a +sustained regression and can take anywhere from 3 hours to 30 hours +depending on the volume of the tree and build frequency. All regression +alerts do get briefly investigated and bugs are filed if necessary.* + + +Dealing with test failures +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If a build or a test job fails, you can click on the red or orange or +purple symbol for the job on Treeherder to display more information. +The information will appear in the footer, including a summary of any +error messages, a "+" icon to re-trigger the job (schedule it to run +again), and links to the log files and to possibly-related bugs. + +Here are some steps you can follow to figure out what is causing most +failures, `and "star" them +appropriately <http://ehsanakhgari.org/blog/2010-04-09/assisted-starring-oranges>`__: + +#. Click on the failing job to see a list of suggested bugs. If the + failure clearly matches a known bug, **click on the star** next to + that bug and then click "Add a comment" and then submit the comment. + This is referred to as "starring the build;" you'll see this phrase + or ones like it in IRC a lot. +#. If the failure might match a known bug but you are not sure, click + the bug number to open the Bugzilla report, and click the failing job + to open its log. If the log and the bug do match, add a comment as + in step 1 (above). +#. If the summary does not seem to match any suggested bugs, search + Bugzilla for the name of the failing test or the error message. If + you find a matching bug, add a comment in the bug in Bugzilla, and + another to the job in Treeherder. +#. If you can't figure out whether a known bug exists (for example, + because you can't figure out what part of the log you should search + for), look on Treeherder to see if there are other similar failures + nearby, or ask on #developers to see if anyone recognizes it as a + known failure. For example, many Android tests fail frequently in + ways that do not produce useful log messages. You can often find the + appropriate bug just by looking at other Android failures that are + already starred. +#. If there is no matching bug, you can back out the change (if you + suspect the failure was caused by your changeset) or re-trigger the + job (if you suspect it's an unrelated intermittent failure). After + more test runs it should become clear whether it is a new regression + or just an unknown intermittent failure. +#. If it turns out to be an unknown intermittent failure, file a new bug + with "intermittent-failure" in the keywords. Include the name of the + test file and an one-line summary of the log messages in the Summary + field. In the description, include an excerpt of the error messages + from the log, and a link to the log file itself. + +At any point if you are not sure or can't figure out what to do, ask for +advice or help in `#developers <https://chat.mozilla.org>`__. +If a large number of jobs are failing and you suspect an infrastructure problem, you can also ask +about it in `#releng <https://chat.mozilla.org>`__. + + +Dealing with performance regressions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Under some circumstances, if your patch causes a performance regression +that is not acceptable, it will get backed out. diff --git a/docs/contributing/contribution_quickref.rst b/docs/contributing/contribution_quickref.rst new file mode 100644 index 0000000000..d4e11050d8 --- /dev/null +++ b/docs/contributing/contribution_quickref.rst @@ -0,0 +1,290 @@ +Firefox Contributors' Quick Reference +===================================== + +Some parts of this process, including cloning and compiling, can take a long time even on modern hardware. +If at any point you get stuck, please don't hesitate to ask at `https://chat.mozilla.org <https://chat.mozilla.org>`__ +in the `#introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`__ channel. + +Clone the sources +----------------- + +You can use either mercurial or git. `Mercurial <https://www.mercurial-scm.org/downloads>`__ is the canonical version control system. + +.. code-block:: shell + + $ hg clone https://hg.mozilla.org/mozilla-central/ + +For git, see the `git cinnabar documentation <https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development>`__ + +The clone can take from 40 minutes to two hours (depending on your connection) and +the repository should be less than 5GB (~ 20GB after the build). + +If you have any network connection issues and cannot clone with command, try :ref:`Mercurial bundles <Mercurial bundles>`. + +:ref:`More information <Mercurial Overview>` + +Install dependencies (non-Windows) +---------------------------------- + +Firefox provides a mechanism to install all dependencies; in the source tree: + +.. code-block:: shell + + $ ./mach bootstrap + +The default options are recommended. +If you're not planning to write C++ or Rust code, select :ref:`Artifact Mode <Understanding Artifact Builds>` +and follow the instructions at the end of the bootstrap for creating a mozconfig file. + +More information :ref:`for Linux <Building Firefox On Linux>` and :ref:`for MacOS <Building Firefox On MacOS>` + +Windows dependencies +-------------------- + +#. You need 64-bit version of Windows 7 or later. +#. Download and install `Visual Studio Community Edition. <https://visualstudio.microsoft.com/downloads/>`__ +#. Finally download the `MozillaBuild Package. <https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe>`__ Installation directory should be: + + .. code-block:: shell + + $ c:\mozilla-build\ + +#. Before moving on to the next steps, make sure to fulfill the :ref:`Windows prerequisites <Building Firefox On Windows>` + +.. note:: + + All the commands of this tutorial must be run in the shell provided with the MozillaBuild Package (start-shell.bat) + +:ref:`More information <Building Firefox On Windows>` + +To build & run +-------------- + +Once all the dependencies have been installed, run: + +.. code-block:: shell + + $ ./mach build + +which will check for dependencies and start the build. +This will take a while; a few minutes to a few hours depending on your hardware. + +.. note:: + + The default build is a compiled build with optimizations. Check out the + :ref:`mozconfig file documentation <Configuring Build Options>` + to see other build options. If you don't plan to change C++ or Rust code, + an :ref:`artifact build <Understanding Artifact Builds>` will be faster. + +To run it: + +.. code-block:: shell + + $ ./mach run + +:ref:`More information about Linux <Building Firefox On Linux>` / :ref:`More information about MacOS <Building Firefox On MacOS>` + + +To write a patch +---------------- + +Make the changes you need in the codebase. You can look up UI text in `Searchfox <https://searchfox.org>`__ to find the right file. + +Then: + +.. code-block:: shell + + # Mercurial + $ hg commit + + # Git + $ git commit + +.. _Commit message: + +The commit message should look like: + +.. code-block:: + + Bug xxxx - Short description of your change. r?reviewer + + Optionally, a longer description of the change. + +**Make sure you include the bug number and at least one reviewer (or reviewer group) in this format.** + +To :ref:`find a reviewer or a review group <Getting reviews>`, the easiest way is to run +``hg log <modified-file>`` (or ``git log <modified-file>``, if +you're using git) on the relevant files, and look who usually is +reviewing the actual changes (ie not reformat, renaming of variables, etc). + + +To visualize your patch in the repository, run: + +.. code-block:: shell + + # Mercurial + $ hg wip + + # Git + $ git show + +:ref:`More information on how to work with stack of patches <Working with stack of patches Quick Reference>` + +:ref:`More information <Mercurial Overview>` + +To make sure the change follows the coding style +------------------------------------------------ + +To detect coding style violations, use mach lint: + +.. code-block:: shell + + $ ./mach lint path/to/the/file/or/directory/you/changed + + # To get the autofix, add --fix: + $ ./mach lint path/to/the/file/or/directory/you/changed --fix + +:ref:`More information <Code quality>` + +To test a change locally +------------------------ + +To run the tests, use mach test with the path. However, it isn’t +always easy to parse the results. + +.. code-block:: shell + + $ ./mach test dom/serviceworkers + +`More information <https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing>`__ + +To test a change remotely +------------------------- + +Running all the tests for Firefox takes a very long time and requires multiple +operating systems with various configurations. To build Firefox and run its +tests on continuous integration servers (CI), two commands are available: + +.. code-block:: shell + + $ ./mach try chooser + +To select jobs running a fuzzy search: + +.. code-block:: shell + + $ ./mach try fuzzy + +From `Treeherder <https://treeherder.mozilla.org/>`__ (our continuous integration system), it is also possible to attach new jobs. As every review has +a try CI run associated, it makes this work easier. See :ref:`attach-job-review` for +more information. + +.. note:: + + This requires `level 1 commit access <https://www.mozilla.org/about/governance/policies/commit/access-policy/>`__. + + You can ask your reviewer to submit the patch for you if you don't have that + level of access. + +:ref:`More information <Try Server>` + + +To submit a patch +----------------- + +To submit a patch for review, we use a tool called `moz-phab <https://pypi.org/project/MozPhab/>`__. +To install it, run: + +.. code-block:: shell + + $ ./mach install-moz-phab + +Once you want to submit your patches (make sure you :ref:`use the right commit message <Commit message>`), run: + +.. code-block:: shell + + $ moz-phab + +It will publish all the currently applied patches to Phabricator and inform the reviewer. + +If you wrote several patches on top of each other: + +.. code-block:: shell + + $ moz-phab submit <first_revision>::<last_revision> + +`More +information <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`__ + +To update a submitted patch +--------------------------- + +It is rare that a reviewer will accept the first version of patch. Moreover, +as the code review bot might suggest some improvements, changes to your patch +may be required. + +Run: + +.. code-block:: shell + + # Mercurial + $ hg commit --amend + + # Git + $ git commit --amend + +After amending the patch, you will need to submit it using moz-phab again. + +If you wrote many changes, you can squash or edit commits with the +command: + +.. code-block:: shell + + # Mercurial + $ hg histedit + + # Git + $ git rebase -i + +The submission step is the same as for the initial patch. + +:ref:`More information on how to work with stack of patches <Working with stack of patches Quick Reference>` + +Retrieve new changes from the repository +---------------------------------------- + +To pull changes from the repository, run: + +.. code-block:: shell + + # Mercurial + $ hg pull --rebase + + # Git + $ git pull --rebase + +To push a change in the code base +--------------------------------- + +Once the change has been accepted and you've fixed any remaining issues +the reviewer identified, add the *Check-in Needed* tag to the review +(use the *Edit Revision* option on the top right). + +The landing procedure will automatically close the review and the bug. + +:ref:`More information <How to submit a patch>` + +Contributing to GeckoView +------------------------- + +Note that the GeckoView setup and contribution processes are different from those of Firefox; +GeckoView setup and contribution docs live in `geckoview.dev <https://geckoview.dev>`__. + +More documentation about contribution +------------------------------------- + +https://developer.mozilla.org/docs/Mozilla/Developer_guide/Introduction + +https://mozilla-version-control-tools.readthedocs.io/en/latest/devguide/contributing.html + +https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html diff --git a/docs/contributing/debugging/capturing_minidump.rst b/docs/contributing/debugging/capturing_minidump.rst new file mode 100644 index 0000000000..bd3d089e9f --- /dev/null +++ b/docs/contributing/debugging/capturing_minidump.rst @@ -0,0 +1,95 @@ +Capturing a minidump +==================== + +*Minidumps* are files created by various Windows tools which record the +complete state of a program as it's running, or as it was at the moment +of a crash. Small minidumps are created by the Breakpad :ref:`crash +reporting <Crash Reporter>` tool, but sometimes that's not +sufficient to diagnose a problem. For example, if the application is +hanging (not responding to input, but hasn't crashed) then Breakpad is +not triggered, and it can be difficult to determine where the problem +lies. Sometimes a more complete form of minidump is needed to see +additional details about a crash, in which case manual capture of a +minidump is desired. + +This page describes how to capture these minidumps on Windows, to permit +better debugging. + + +Privacy and minidumps +--------------------- + +.. warning:: + + **Warning!** Unlike the minidumps submitted by Breakpad, these + minidumps contain the **complete** contents of program memory. They + are therefore much more likely to contain private information, if + there is any in the browser. For this reason, you may prefer to + generate minidumps against a `clean + profile <http://support.mozilla.com/en-US/kb/Managing%20profiles>`__ + where possible. + + +Capturing a minidump: application crash +--------------------------------------- + +To capture a full minidump for an application crash, you can use a tool +called windbg. + + +Install debugging tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Microsoft distributes the Debugging Tools for Windows for free, those +include WinDbg which you will need here. Download it from `Install +Debugging Tools for +Windows <http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx>`__. +(*You'll want the 32-bit version of WinDbg only if you are using a +32*-bit version of Firefox) Then install it, the standard settings in +the installation process are fine. + + +Capture a minidump +~~~~~~~~~~~~~~~~~~ + +#. Connect Firefox to the debugger. + + a. If Firefox is not already running, then open WinDbg from the Start + menu (Start->All Programs->Debugging Tools for Windows->WinDbg). + Next, open the **"File"** menu and choose **"Open + Executable..."**. In the file chooser window that appears, open + the firefox.exe executable in your Firefox program folder + (C:\Program Files\Mozilla Firefox). + + b. If Firefox is already running, open WinDbg from the Start menu + (Start->All Programs->Debugging Tools for Windows->WinDbg). Next, + open the **"File"** menu and choose **"Attach to a Process..."**. + In the file chooser window that appears, find the firefox.exe + executable process with the lowest PID. + +#. You should now see a "Command" text window with debug output at the + top and an input box at the bottom. From the menu, select + ``Debug → Go``, and Firefox should start. If the debugger spits out + some text right away and Firefox doesn't come up, select + ``Debug → Go`` again. + +#. When the program is about to crash, WinDbg will spit out more data, + and the prompt at the bottom will change from saying "``*BUSY*``" to + having a number in it. At this point, you should type + "``.dump /ma c:\temp\firefoxcrash.dmp``" -- without the quotes, but + don't forget the dot at the beginning. Once it completes, which can + take a fair while, you will have a very large file at + ``c:\temp\firefoxcrash.dmp`` that can be used to help debug your + problem. File size will depend on this size of Firefox running in + your environment, which could several GB. + +#. Ask in the relevant bug or thread how best to share this very large + file! + + +Capturing a minidump: application hang +-------------------------------------- + +On Windows Vista and Windows 7, you can follow `these +instructions <http://support.microsoft.com/kb/931673>`__ to capture a +dump file and locate it after it's been saved. diff --git a/docs/contributing/debugging/debugging_a_hang_on_macos.rst b/docs/contributing/debugging/debugging_a_hang_on_macos.rst new file mode 100644 index 0000000000..ee2bed16b2 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_hang_on_macos.rst @@ -0,0 +1,10 @@ +Debugging A Hang On macOS +========================= + +See `How to Report a Hung +Firefox <https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox>`_. + +See also +~~~~~~~~ + +`Debugging on macOS <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_macOS>`__ diff --git a/docs/contributing/debugging/debugging_a_minidump.rst b/docs/contributing/debugging/debugging_a_minidump.rst new file mode 100644 index 0000000000..44fd73f207 --- /dev/null +++ b/docs/contributing/debugging/debugging_a_minidump.rst @@ -0,0 +1,191 @@ +Debugging A Minidump +==================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +The +`minidump <http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx>`__ +file format contains data about a crash on Windows. It is used by +`Breakpad <https://wiki.mozilla.org/Breakpad>`__ and also by various +Windows debugging tools. Each minidump includes the following data. + +- Details about the exception which led to the crash. +- Information about each thread in the process: the address which was + executing and the register state at the time the process stopped. +- A list of shared libraries loaded into the process at the time of the + crash. +- The stack memory of each thread. +- The memory right around the crashing address. +- (Optional) Other memory regions, if requested by the application. +- (Optional) Other platform-specific data. + +Accessing minidumps from crash reports +-------------------------------------- + +Minidumps are not available to everyone. For details on how to gain +access and where to find minidump files for crash reports, consult the +:ref:`crash report documentation <Understanding Crash Reports>` + +Using the MS Visual Studio debugger +----------------------------------- + +#. Set up the debugger to :ref:`use the Mozilla symbol + server <Using The Mozilla Symbol Server>` and + :ref:`source server <Using The Mozilla Source Server>`. +#. Double-click on the minidump file to open it in the debugger. +#. When it loads, click the green icon in the visual studio debugger + toolbar that looks like a play button. + +For Firefox releases older than Firefox 41, you will also need to +install the relevant release of Firefox (for example from +`here <https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/>`__), +and add the directory it is in (e.g., "C:\Program Files\Mozilla +Firefox 3.6 Beta 1\") to the same dialog in which you set up the +symbol server (in case the binary location in the minidump is not the +same as the one on your machine). Note that you can install the +relevant release anywhere. Just make sure to configure the symbol +server to the directory where you installed it. For releases from 41 +onward, the binaries are available on the symbol server. + +If this doesn't work, downloading the exact build and crashreporter +symbols full files. These can be found in treeherder / build folder. +Load Visual Studio, and go to file -> open -> minidump location. Click +on "Run Native", and Visual Studio will ask for the corresponding symbol +files. For each .dll you wish to have symbols for, you must go to a +console and go to the corresponding directory. E.g. (xul.dll should go +to xul.pdf in the crashreporter symbols directory). Each directory will +have a .pd\_ file. In a command shell run: "expand /r foo.pd\_". Then +point Visual Studio to this directory. + +Then you'll be able to examine: + ++------------------+-------------------------------------------------------------------------+ +| stack trace | The debugger shows the stack trace. You can right-click on any frame | +| | in the stack, and then choose "go to disassembly" or "go to source". | +| | (Choosing "go to disassembly" from the source view might not get you | +| | to the right place due to optimizations.) When looking at the | +| | source, beware that the debugging information will associate all | +| | inlined functions as part of the line into which they were inlined, | +| | and compiler (with PGO) inlines *very* aggressively (including | +| | inlining virtual functions). You can often figure out where you | +| | *really* are by reading the disassembly. | ++------------------+-------------------------------------------------------------------------+ +| registers | In the Registers tab (Debug->Windows->Registers if you don't have | +| | it open), you can look at the registers associated with each stack | +| | frame, but only at the current state (i.e., the time of the crash). | +| | Registers that Visual Studio can't figure out will be grayed-out and | +| | have the value 00000000. | ++------------------+-------------------------------------------------------------------------+ +| stack memory | You open a window (Memory 1, etc.) that shows contiguous segments of | +| | memory using the Debug->Windows->Memory menu item. You can then | +| | enter the address of the stack pointer (ESP register) in this window | +| | and look at the memory on the stack. (The minidump doesn't have the | +| | memory on the heap.) It's a good idea to change the "width" dropdown | +| | in the top right corner of the window from its default "Auto" to | +| | either "8" or "16" so that the memory display is word-aligned. If | +| | you're interested in pointers, which is usually the case, you can | +| | right click in this window and change the display to show 4-byte | +| | words (so that you don't have to reverse the order due to | +| | little-endianness). This view, combined with the disassembly, can | +| | often be used to reconstruct information beyond what in shown the | +| | function parameters. | ++------------------+-------------------------------------------------------------------------+ +| local variables | In the Watch 1 (etc.) window (which, if you don't have open, you can | +| | iget from Debug->Windows->Watch), you can type an expression | +| | (e.g., the name of a local variable) and the debugger will show you | +| | its value (although it sometimes gets confused). If you're looking | +| | at a pointer to a variable that happens to be on the stack, you can | +| | even examine member variables by just typing expressions. If Visual | +| | Studio can't figure something out from the minidump, it might show | +| | you 00000000 (is this true?). | ++------------------+-------------------------------------------------------------------------+ + +Using minidump-2-core on Linux +------------------------------ + +The `Breakpad +source <https://chromium.googlesource.com/breakpad/breakpad/+/master/>`__ +contains a tool called +`minidump-2-core <https://chromium.googlesource.com/breakpad/breakpad/+/master/src/tools/linux/md2core/>`__, +which converts Linux minidumps into core files. If you checkout and +build Breakpad, the binary will be at +``src/tools/linux/md2core/minidump-2-core``. Running the binary with the +path to a Linux minidump will generate a core file on stdout which can +then be loaded in gdb as usual. You will need to manually download the +matching Firefox binaries, but then you can use the :ref:`GDB Python +script <Downloading_symbols_on_Linux_Mac_OS_X>` to download symbols. + +The ``minidump-2-core`` source does not currently handle processing +minidumps from a different CPU architecture than the system it was +built for. If you want to use it on an ARM dump, for example, you may +need to build the tool for ARM and run it under QEMU. + +Using other tools to inspect minidump data +------------------------------------------ + +Breakpad includes a tool called ``minidump_dump`` built alongside +``minidump_stackwalk`` which will verbosely print the contents of a +minidump. This can sometimes be useful for finding specific information +that is not exposed on crash-stats. + +Ted has a few tools that can be built against an already-built copy of +Breakpad to do more targeted inspection. All of these tools assume you +have checked out their source in a directory next to the breakpad +checkout, and that you have built Breakpad in an objdir named +``obj-breakpad`` at the same level. + +- `stackwalk-http <https://hg.mozilla.org/users/tmielczarek_mozilla.com/stackwalk-http/>`__ + is a version of minidump_stackwalk that can fetch symbols over HTTP, + and also has the Mozilla symbol server URL baked in. If you run it + like ``stackwalk /path/to/dmp /tmp/syms`` it will print the stack + trace and save the symbols it downloaded in ``/tmp/syms``. Note that + symbols are only uploaded to the symbol server for nightly and + release builds, not per-change builds. +- `dump-lookup <https://hg.mozilla.org/users/tmielczarek_mozilla.com/dump-lookup/>`__ + takes a minidump and prints values on the stack that are potential + return addresses. This is useful when a stack trace looks truncated + or otherwise wrong. It needs symbol files to produce useful output, + so you will generally want to have run ``stackwalk-http`` to download + them first. +- `get-minidump-instructions <https://hg.mozilla.org/users/tmielczarek_mozilla.com/get-minidump-instructions/>`__ + retrieves and displays the memory range surrounding the faulting + instruction pointer from a minidump. You will almost always want to + run it with the ``--disassemble`` option, which will make it send the + bytes through ``objdump`` to display the disassembled instructions. + If you also give it a path to symbols (see ``stackwalk-http`` above) + it can download the matching source files from hg.mozilla.org and + display source interleaved with the disassembly. +- `minidump-modules <http://hg.mozilla.org/users/tmielczarek_mozilla.com/minidump-modules>`__ + takes a minidump and prints the list of modules from the crashed + process. It will print the full path to each module, whereas the + Socorro UI only prints the filename for each module for privacy + reasons. It also accepts a -v option to print the debug ID for each + module, and a -d option to print relative paths to the symbol files + that would be used instead of the module filenames. + +Getting a stack trace from a crashed B2G process +------------------------------------------------ + +#. Get the minidump file in the phone at + /data/b2g/mozilla/\*.default/minidump/. You can use `adb + pull <http://developer.android.com/tools/help/adb.html>`__ for that. +#. Build the debug symbols using the command ./build.sh buildsymbols + inside the B2G tree. The symbol files will be generated in + $OBJDIR/dist/crashreporter-symbols. +#. Build and install + `google-breakpad <https://code.google.com/p/google-breakpad/>`__. +#. Use the + `minidump_stackwalk <https://code.google.com/p/google-breakpad/wiki/LinuxStarterGuide>`__ + breakpad tool to get the stack trace. + +.. code:: bash + + Example: + + $ cd B2G + $ adb pull /data/b2g/mozilla/*.default/minidump/*.dmp . + $ls *.dmp + 71788789-197e-d769-67167423-4e7aef32.dmp + $ minidump_stackwalk 71788789-197e-d769-67167423-4e7aef32.dmp objdir-debug/dist/crashreporter-symbols/ diff --git a/docs/contributing/debugging/debugging_firefox_with_gdb.rst b/docs/contributing/debugging/debugging_firefox_with_gdb.rst new file mode 100644 index 0000000000..cafad0cc8b --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_gdb.rst @@ -0,0 +1,504 @@ +Debugging Firefox with GDB +========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This page details how you can more easily debug Firefox and work around +some GDB problems. + +Use GDB 5, or higher. A more recent version of GDB can be obtained from +`sourceware <https://sourceware.org/gdb/>`__ or your Linux distro repo. +If you are running less than 256 MB of RAM, be sure to see `Using gdb on +wimpy computers <https://developer.mozilla.org/en/Using_gdb_on_wimpy_computers>`__. + +Where can I find general gdb documentation? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using GDB is beyond the scope of this document. Documentation is likely +available on your system if you have GDB installed, in the form of +**info,** **man** pages, or the gnome help browser. Additionally, you +can use a graphical front-end to GDB like +`ddd <https://www.gnu.org/software/ddd/>`__ or +`insight <https://sourceware.org/insight/>`__. For more information see +https://sourceware.org/gdb/current/onlinedocs/gdb/ + +How do I run Firefox under gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The preferred method, is using the +:ref:`mach` command-line tool to run the +debugger, which can bypass several optional defaults. Use "mach help +run" to get more details. If inside the source directory, you would use +"./mach". If you have previously :ref:`added mach to your path <Adding_mach_to_your_shell>`, +then just use "mach". Please note that :ref:`mach is aware of mozconfigs <mach_and_mozconfigs>`. + +.. code:: bash + + $ ./mach run --debug [arguments to pass to firefox] + +If you need to direct arguments to gdb, you can use '--debugger-args' +options via the command line parser, taking care to adhere to shell +splitting rules. For example, if you wanted to run the command 'show +args' when gdb starts, you would use: + +.. code:: bash + + $ ./mach run --debug --debugger-args "-ex 'show args'" + +Alternatively, you can run gdb directly against Firefox. However, you +won't get some of the more useful capabilities this way. For example, +mach sets an environment variable (see below) to stop the JS engine from +generating synthetic segfaults to support the slower script dialoging +mechanism. + +.. code:: + + (gdb) OBJDIR/dist/bin/firefox + +How do I pass arguments in prun? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the arguments in GDB before calling prun. Here's an example on how +to do that: + +.. code:: + + (gdb) set args https://www.mozilla.org + (gdb) prun + +How do I set a breakpoint in a library that hasn't been loaded? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +GDB 6.1 and above has support for "pending breakpoints". This is +controlled by the "``set breakpoint pending``" setting, and is enabled +by default. If a breakpoint cannot be immediately resolved, it will be +re-checked each time a shared library is loaded, by the process being +debugged. If your GDB is older than this, you should upgrade. + +In older versions, there isn't a way to set breakpoints in a library +that has not yet been loaded. See more on `setting a breakpoint when a +component is +loaded <#How_do_I_set_a_breakpoint_when_a_component_is_loaded.3F>`__. If +you have to set a breakpoint you can set a breakpoint in ``_dl_open``. +This function is called when a new library is loaded, when you can +finally set your breakpoint. + +How do I set a breakpoint when a component is loaded? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Firefox Version 57 (and possibly earlier) XPCOM_BREAK_ON_LOAD does +not seem to exist. + +There's a facility in XPCOM which allows you to set an environment +variable to drop into the debugger when loading a certain component. You +have to set ``XPCOM_BREAK_ON_LOAD`` variable before you run Firefox, +setting it to a string containing the names of libraries you want to +load. For example, if you wish to stop when a library named ``raptor`` +or ``necko`` is loaded, you set the variable to ``raptor:necko``. Here's +an example: + +.. code:: + + (gdb) set env XPCOM_BREAK_ON_LOAD raptor:necko + (gdb) prun + +Why can't I set a breakpoint? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You probably can't set a breakpoint because its library hasn't been +loaded. Most Firefox functionality is in libraries loaded mid-way +through the ``main()``\ function. If you break on ``main(),``\ and step +through until the libraries are loaded, with a call to +``InitCOMGlue()``, you should be able to set breakpoints on many more +symbols, source files, and continue running. + +.. code:: + + (gdb) break main + (gdb) run + Breakpoint 1, main(argc=4, argv=0x7fffffffde98, envp=0x7ffffffffdec0) ..... + 256 { + (gdb) next + ... + 293 nsresult rv = InitXPCOMGlue() + (gdb) next + +If you still can't set the breakpoints, you need to confirm the library +has loaded. You can't proceed until the library loads. See more on +`loading shared libraries <#How_do_I_load_shared_libraries.3F>`__. If +you wish to break as soon as the library is loaded, see the section on +`breaking when a component is +loaded <#How_do_I_set_a_breakpoint_when_a_component_is_loaded.3F>`__ and +`breaking on a library +load <#How_do_I_set_a_breakpoint_when_a_component_is_loaded.3F>`__. + +How do I display PRUnichar's? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One suggestion is this: + +.. code:: + + (gdb) print ((PRUnichar*)uri.mBuffer)[0]@16 + $47 = {114, 100, 102, 58, 110, 117, 108, 108, 0, 0, 8, 0, 0, 0, 37432, + 16514} + + + +.. code:: + + (gdb) print aURI + $1 = (const PRUnichar *) 0x855e6e0 + (gdb) x/32ch aURI + 0x855e6e0: 104 'h' 116 't' 116 't' 112 'p' 58 ':' 47 '/' 47 '/' 119 'w' + 0x855e6f0: 119 'w' 119 'w' 46 '.' 109 'm' 111 'o' 122 'z' 105 'i' 108 'l' + 0x855e700: 108 'l' 97 'a' 46 '.' 111 'o' 114 'r' 103 'g' 47 '/' 115 's' + 0x855e710: 116 't' 97 'a' 114 'r' 116 't' 47 '/' 0 '\0' 25 '\031' 0 '\0' + (gdb) + +- Define helper functions in your .gdbinit + +.. code:: + + # Define a "pu" command to display PRUnichar * strings (100 chars max) + # Also allows an optional argument for how many chars to print as long as + # it's less than 100. + def pu + set $uni = $arg0 + if $argc == 2 + set $limit = $arg1 + if $limit > 100 + set $limit = 100 + end + else + set $limit = 100 + end + # scratch array with space for 100 chars plus null terminator. Make + # sure to not use ' ' as the char so this copy/pastes well. + set $scratch = "____________________________________________________________________________________________________" + set $i = 0 + set $scratch_idx = 0 + while (*$uni && $i++ < $limit) + if (*$uni < 0x80) + set $scratch[$scratch_idx++] = *(char*)$uni++ + else + if ($scratch_idx > 0) + set $scratch[$scratch_idx] = '\0' + print $scratch + set $scratch_idx = 0 + end + print /x *(short*)$uni++ + end + end + if ($scratch_idx > 0) + set $scratch[$scratch_idx] = '\0' + print $scratch + end + end + + # Define a "ps" command to display subclasses of nsAC?String. Note that + # this assumes strings as of Gecko 1.9 (well, and probably a few + # releases before that as well); going back far enough will get you + # to string classes that this function doesn't work for. + def ps + set $str = $arg0 + if (sizeof(*$str.mData) == 1 && ($str.mFlags & 1) != 0) + print $str.mData + else + pu $str.mData $str.mLength + end + end + +`This is hard. Give me a .gdbinit that already has the +functions. <#This_is_hard._Give_me_a_.gdbinit_that_works.>`__ + +- Define a small helper function "punichar" in #ifdef NS_DEBUG code + somewhere. + +How do I display an nsString? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can call the ToNewCString() method on the nsString. It leaks a +little memory but it shouldn't hurt anything if you only do it a few +times in one gdb session. (via akkana@netscape.com) + +.. code:: + + (gdb) p string.ToNewCString() + +Another method (via bent) is the following (replace ``n`` with: the +returned length of your string): + +.. code:: + + (gdb) p string.Length() + $1 = n + (gdb) x/ns string.BeginReading() + +You can of course use any of the above unichar-printing routines instead +of x/s. + +This is hard. Give me a .gdbinit that works. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See `Boris Zbarsky's +.gdbinit <http://web.mit.edu/bzbarsky/www/gdbinit>`__. It contained +several function definitions including: + +- "prun" to start the browser and disable library loading. +- "pu" which will display a (PRUnichar \*) string. +- "ps" which will display a nsString. + +How do I determine the concrete type of an object pointed to by an interface pointer? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can determine the concrete type of any object pointed to, by an +XPCOM interface pointer, by looking at the mangled name of the symbol +for the object's vtable: + +.. code:: + + (gdb) p aKidFrame + $1 = (nsIFrame *) 0x85058d4 + (gdb) x/wa *(void**)aKidFrame + 0x4210d380 <__vt_14nsRootBoxFrame>: 0x0 + (gdb) p *(nsRootBoxFrame*)aKidFrame + [ all the member variables of aKidFrame ] + +If you're using gcc 3.x, the output is slightly different from the gcc +2.9x output above. Pay particular attention to the vtable symbol, in +this case ``__vt_14nsRootBoxFrame``. You won't get anything useful if +the shared library containing the object is not loaded. See `How do I +load shared libraries? <#How_do_I_load_shared_libraries.3F>`__ and `How +do I see what libraries I already have +loaded? <#How_do_I_see_what_libraries_I_already_have_loaded.3F>`__ + +Or use the gdb command ``set print object on``. + +How can I debug JavaScript from gdb? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have JavaScript Engine code on the stack, you'll probably want a +JS stack in addition to the C++ stack. + +.. code:: + + (gdb) call DumpJSStack() + +See `https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_JavaScript <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_JavaScript>`__ +for more JS debugging tricks. + +How can I debug race conditions and/or how can I make something different happen at NS_ASSERTION time? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +| [submitted by Dan Mosedale] +| As Linux is unable to generate useful core files for multi-threaded + applications, tracking down race-conditions which don't show up under + the debugger can be a bit tricky. Unless you've given the + ``--enable-crash-on-assert`` switch to ``configure``, you can now + change the behavior of ``NS_ASSERTION`` (nsDebug::Break) using the + ``XPCOM_DEBUG_BREAK`` environment variable. + +How do I run the debugger in emacs/xemacs? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Emacs and XEmacs contain modes for doing visual debugging. However, you +might want to set up environment variables, specifying the loading of +symbols and components. The easiest way to set up these is to use the +``run-mozilla.sh`` script, located in the dist/bin directory of your +build. This script sets up the environment to run the editor, shell, +debugger, or defining a preferred setup and running any commands you +wish. For example: + +.. code:: bash + + $ ./run-mozilla.sh /bin/bash + MOZILLA_FIVE_HOME=/home/USER/src/mozilla/build/dist/bin + LD_LIBRARY_PATH=/home/USER/src/mozilla/build/dist/bin + LIBRARY_PATH=/home/USER/src/mozilla/build/dist/bin + SHLIB_PATH=/home/USER/src/mozilla/build/dist/bin + LIBPATH=/home/USER/src/mozilla/build/dist/bin + ADDON_PATH=/home/USER/src/mozilla/build/dist/bin + MOZ_PROGRAM=/bin/bash + MOZ_TOOLKIT= + moz_debug=0 + moz_debugger= + +GDB 5 used to work for me, but now Firefox won't start. What can I do? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A recent threading change (see `bug +57051 <https://bugzilla.mozilla.org/show_bug.cgi?id=57051>`__ for +details) caused a problem on some systems. Firefox would get part-way +through its initialization, then stop before showing a window. A recent +change to gdb has fixed this. Download and build `the latest version of +Insight <https://sources.redhat.com/insight/>`__, or if you don't want a +GUI, `the latest version of gdb <https://sources.redhat.com/gdb/>`__. + +"run" or "prun" in GDB fails with "error in loading shared libraries." +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Running mozilla-bin inside GDB fails with an error message like: + +.. code:: + + Starting program: + /u/dmose/s/mozilla/mozilla-all/mozilla/dist/bin/./mozilla-bin + /u/dmose/s/mozilla/mozilla-all/mozilla/dist/bin/./mozilla-bin: error + in loading shared libraries: libraptorgfx.so: cannot open shared + object file: No such file or directory + +Your LD_LIBRARY_PATH is probably being reset by your .cshrc or .profile. +From the GDB manual: + +*\*Warning:\* GDB runs your program using the shell indicated by your +'SHELL' environment variable if it exists (or '/bin/sh' if not). If your +'SHELL' variable names a shell that runs an initialization file -- such +as '.cshrc' for C-shell, or '.bashrc' for BASH--any variables you set in +that file affect your program. You may wish to move the setting of +environment variables to files that are only run when you sign on, such +as '.login' or '.profile'.* + +Debian's GDB doesn't work. What do I do? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Debian's unstable distribution currently uses glibc 2.1 and GDB 4.18. +However, there is no package of GDB for Debian with the appropriate +threads patches that will work with glibc 2.1. I was able to get this to +work by getting the GDB 4.18 RPM from Red Hat's rawhide server and +installing that. It has all of the patches necessary for debugging +threaded software. These fixes are expected to be merged into GDB, which +will fix the problem for Debian Linux. (via `Bruce +Mitchener <mailto:bruce@cybersight.com>`__) + +Firefox is aborting. Where do I set a breakpoint to find out where it is exiting? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On Linux there are two possible symbols that are causing this: +``PR_ASSERT()`` and ``NS_ASSERTION()``. To see where it's asserting you +can stop at two places: + +.. code:: + + (gdb) b abort + (gdb) b exit + +I keep getting a SIGSEGV in JS/JIT code under gdb even though there is no crash when gdb is not attached. How do I fix it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set the JS_DISABLE_SLOW_SCRIPT_SIGNALS environment variable (in FF33, +the shorter and easier-to-remember JS_NO_SIGNALS). For an explanation, +read `Jan's blog +post <https://www.jandemooij.nl/blog/2014/02/18/using-segfaults-to-interrupt-jit-code/>`__. + +I keep getting a SIG32 in the debugger. How do I fix it? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are getting a SIG32 while trying to debug Firefox you might have +turned off shared library loading before the pthreads library was +loaded. For example, ``set auto-solib-add 0`` in your ``.gdbinit`` file. +In this case, you can either: + +- Remove it and use the method explained in the section about `GDB's + memory + usage <#The_debugger_uses_a_lot_of_memory._How_do_I_fix_it.3F>`__ +- Use ``handle SIG32 noprint`` either in gdb or in your ``.gdbinit`` + file + +Alternatively, the problem might lie in your pthread library. If this +library has its symbols stripped, then GDB can't hook into thread +events, and you end up with SIG32 signals. You can check if your +libpthread is stripped in ``file /lib/libpthread*`` and looking for +``'stripped'.``\ To fix this problem on Gentoo Linux, you can re-emerge +glibc after adding ``"nostrip"`` to your ``FEATURES`` in +``/etc/make.conf``. + +How do I get useful stack traces inside system libraries? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many Linux distributions provide separate packages with debugging +information for system libraries, such as gdb, Valgrind, profiling +tools, etc., to give useful stack traces via system libraries. + +Fedora +^^^^^^ + +On Fedora, you need to enable the debuginfo repositories, as the +packages are in separate repositories. Enable them permanently, so when +you get updates you also get security updates for these packages. A way +to do this is edit ``/etc/yum.repos.d/fedora.repo`` and +``fedora-updates.repo`` to change the ``enabled=0`` line in the +debuginfo section to ``enabled=1``. This may then flag a conflict when +upgrading to a new distribution version. You would the need to perform +this edit again. + +You can finally install debuginfo packages with yum or other package +management tools. The best way is install the ``yum-utils`` package, and +then use the ``debuginfo-install`` command to install all the debuginfo: + +.. code:: bash + + $ yum install yum-utils + $ debuginfo-install firefox + +This can be done manually using: + +.. code:: bash + + $ yum install GConf2-debuginfo ORBit2-debuginfo atk-debuginfo \ + cairo-debuginfo dbus-debuginfo dbus-glib-debuginfo expat-debuginfo \ + fontconfig-debuginfo freetype-debuginfo gcc-debuginfo glib2-debuginfo \ + glibc-debuginfo gnome-vfs2-debuginfo gtk2-debuginfo gtk2-engines-debuginfo \ + hal-debuginfo libX11-debuginfo libXcursor-debuginfo libXext-debuginfo \ + libXfixes-debuginfo libXft-debuginfo libXi-debuginfo libXinerama-debuginfo \ + libXrender-debuginfo libbonobo-debuginfo libgnome-debuginfo \ + libselinux-debuginfo pango-debuginfo popt-debuginfo scim-bridge-debuginfo + +Debugging electrolysis (e10s) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``mach run`` and ``mach test`` both accept a ``--disable-e10s`` +argument. Some debuggers can't catch child-process crashes without it. + +You can find some (outdated) information on +https://wiki.mozilla.org/Electrolysis/Debugging. You may also like to +read +https://mikeconley.ca/blog/2014/04/25/electrolysis-debugging-child-processes-of-content-for-make-benefit-glorious-browser-of-firefox +for a more up-to-date blog post. + +To get the child process id use: + +.. code:: + + MOZ_DEBUG_CHILD_PROCESS=1 mach run + +See also +~~~~~~~~~ + +- `Debugging <https://developer.mozilla.org/En/Debugging>`__ +- `Performance tools <https://wiki.mozilla.org/Performance:Tools>`__ +- `Fun with + gdb <https://blog.mozilla.com/sfink/2011/02/22/fun-with-gdb/>`__ by + Steve Fink +- `Archer pretty printers for + SpiderMonkey <https://hg.mozilla.org/users/jblandy_mozilla.com/archer-mozilla>`__ + (`blog + post <https://itcouldbesomuchbetter.wordpress.com/2010/12/20/debugging-spidermonkey-with-archer-2/>`__) +- `More pretty + printers <https://hg.mozilla.org/users/josh_joshmatthews.net/archer-mozilla/>`__ + for Gecko internals (`blog + post <https://www.joshmatthews.net/blog/2011/06/nscomptr-has-never-been-so-pretty/>`__) + +.. container:: originaldocinfo + + .. rubric:: Original Document Information + :name: Original_Document_Information + + - `History <http://bonsai-www.mozilla.org/cvslog.cgi?file=mozilla-org/html/unix/debugging-faq.html&rev=&root=/www/>`__ + - Copyright Information: © 1998-2008 by individual mozilla.org + contributors; content available under a `Creative Commons + license <https://www.mozilla.org/foundation/licensing/website-content.html>`__ + + diff --git a/docs/contributing/debugging/debugging_firefox_with_lldb.rst b/docs/contributing/debugging/debugging_firefox_with_lldb.rst new file mode 100644 index 0000000000..c3b0c4eb2f --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_lldb.rst @@ -0,0 +1,80 @@ +Debugging Firefox with LLDB +=========================== + +See http://lldb.llvm.org/index.html. + +Mozilla-specific lldb settings +------------------------------ + +There's an +``.lldbinit`` `file <http://mxr.mozilla.org/mozilla-central/source/.lldbinit>`_ +in the Mozilla source tree, which applies recommended settings and +includes a few type summaries and Mozilla-specific debugging commands +via the lldbutils module (see +`python/lldbutils/README.txt <http://mxr.mozilla.org/mozilla-central/source/python/lldbutils/README.txt>`__). +For information about available features see the links above and the `Using +LLDB to debug Gecko <http://mcc.id.au/blog/2014/01/lldb-gecko>`__ blog +post. + +The in-tree ``.lldbinit`` should be loaded automatically in most cases +when running lldb from the command line (e.g. using +:ref:`mach`), but **not** +when using Xcode. See :ref:`Debugging on macOS` for information on setting up +Xcode. + +.. warning:: + + LLDB warning: Xcode 5 only comes with lldb (gdb is gone). The + introduction and use of UNIFIED_SOURCES in the source starting around + November 2013 has broken the default LLDB configuration so that it + will not manage to resolve breakpoints in files that are build using + UNIFIED_SOURCES (the breakpoints will be listed as "pending", and + lldb will not stop at them). To fix this add the following to your + $HOME/.lldbinit file: + + .. code:: + + # Mozilla's use of UNIFIED_SOURCES to include multiple source files into a + # single compiled file breaks lldb breakpoint setting. This works around that. + # See http://lldb.llvm.org/troubleshooting.html for more. + settings set target.inline-breakpoint-strategy always + + Restart Xcode/lldb and restart your debugging session. If that still + doesn't fix things then try closing Xcode/lldb, doing a clobber + build, reopening Xcode/lldb, and restarting your debugging session. + +Starting a debugging session +---------------------------- + +Attaching to an existing process +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can attach to Firefox with following command: + +.. code:: + + (lldb) process attach --name firefox + +Some versions of lldb causes crashes after attaching to Firefox. + +Running a new process +~~~~~~~~~~~~~~~~~~~~~ + +To start Firefox under the debugger, run ``lldb`` followed by "--", +followed by the command line you'd like to run, like this: + +.. code:: bash + + $ lldb -- obj-ff-dbg/dist/Nightly.app/Contents/MacOS/firefox-bin -no-remote -profile /path/to/profile + +Then set breakpoints you need and start the process: + +.. code:: + + (lldb) breakpoint set --name nsInProcessTabChildGlobal::InitTabChildGlobal + Breakpoint created: 1: name = 'nsInProcessTabChildGlobal::InitTabChildGlobal', locations = 0 (pending) + WARNING: Unable to resolve breakpoint to any actual locations. + + (lldb) r + Process 7602 launched: '/.../obj-ff-opt/dist/Nightly.app/Contents/MacOS/firefox-bin' (x86_64) + 1 location added to breakpoint 1 diff --git a/docs/contributing/debugging/debugging_firefox_with_rr.rst b/docs/contributing/debugging/debugging_firefox_with_rr.rst new file mode 100644 index 0000000000..7f2025596b --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_rr.rst @@ -0,0 +1,75 @@ +Debugging Firefox with rr +========================= + +This page is intended to help Firefox/Gecko developers get started using rr to debug Firefox. + +Prerequisites +------------- + +You must have Linux installed with a recent kernel. If you're not running Linux already, an option is set up a virtual machine in which to record Firefox. Be forewarned though that + + * rr requires a VM hypervisor that virtualizes CPU performance counters. VMWare Workstation supports that. + * there's a 20% or so performance hit from running in a VM; generally speaking recorder overhead increases from ~1.2x to ~1.4x. (It's a feather in the cap of the hypervisor authors that the hit is that small, though!) + * Some features (reverse execution) `may not work well in VMWare <https://robert.ocallahan.org/2014/09/vmware-cpuid-conditional-branch.html>`__ due to a VMWare optimization that can be disabled `this way <http://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html>`__. + +Ensure that you've `installed <http://rr-project.org/>`__ or `built <https://github.com/mozilla/rr/wiki/Building-And-Installing>`__ rr and have `used it successfully <https://github.com/mozilla/rr/wiki/Usage>`__. + +Firefox developers are strongly encouraged to build rr from source. If your Firefox patch triggers a bug in rr, rr developers will fix that bug with high priority. You might be able to pull a fix within a few hours or days instead of waiting for the next release. + +Recording Firefox +----------------- + + +To record Firefox running normally, simply launch it under rr as you would if running it under valgrind or gdb + +.. code:: bash + + $ rr $ff-objdir/dist/bin/firefox ... + +This will save a trace to your working directory as described in the `usage instructions <https://github.com/mozilla/rr/wiki/Usage>`__. Please refer to `those instructions <https://github.com/mozilla/rr/wiki/Usage>`__ for details on how to debug the recording, which isn't covered in this document. + +SIGSYS +------ + +When recording and replaying Firefox running with the Linux sandbox, you will get SIGSYS signals frequently. This is expected behavior caused by the sandbox. In gdb, use handle SIGSYS noprint nostop to suppress the signals. + +Recording test suites +--------------------- + +You can use the test runners' --debugger feature to punch rr down through the layers of python script to where Firefox is launched. This is used in the same way you would use --debugger to run valgrind or gdb, for example: + +.. code:: bash + + $ ./mach mochitest --debugger=rr ... + +The test harnesses disable the slow-script timeout when the --debugger argument is passed. That's usually sensible, because you don't want those warnings being generated while Firefox is stopped in gdb. However, this has been `observed to change Gecko behavior <https://bugzilla.mozilla.org/show_bug.cgi?id=986673>`__. rr doesn't need to have the slow-script timeout disabled, so to avoid those kinds of pitfalls, pass the --slowscript argument to the test harness. + +To run rr in chaos mode: + +.. code:: bash + + $ ./mach mochitest --debugger=rr --debugger-args="record --chaos" + +You can also run the entire test harness in rr: + +.. code:: bash + + $ rr ./mach mochitest ... + +The trace will contain many processes, so to debug the correct one, you'll want to use rr ps or rr replay -p firefox etc. + +Recording Firefox in e10s mode +------------------------------ + +rr should work out of the box with multi-process Firefox. Once you have a recording you can use rr ps to show all the process that were recorded and rr replay -p <pid> to attach to a particular process. + +You can combine that with the -M and -g flags to jump to a particular point in a particular process's lifetime. +Get help! + +If you encounter a problem with rr, please `file an issue <https://github.com/mozilla/rr/issues>`__. Firefox bugs are high priority, so usually your issue can be fixed very quickly. + +If you want to chat with rr developers, because you need more help or want to contribute or want to complain, we hang out in the `#rr channel <https://chat.mozilla.org/#/room/#rr:mozilla.org>`__. + +You also may find `these debugging protips <https://github.com/mozilla/rr/wiki/Debugging-protips>`__ helpful, though many are for rr developers, not users. + +Happy debugging! diff --git a/docs/contributing/debugging/debugging_firefox_with_valgrind.rst b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst new file mode 100644 index 0000000000..b6b2676249 --- /dev/null +++ b/docs/contributing/debugging/debugging_firefox_with_valgrind.rst @@ -0,0 +1,177 @@ +Debugging Firefox with Valgrind +=============================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This page describes how to use Valgrind (specifically, its Memcheck +tool) to find memory errors. + +Supported platforms +------------------- + +Valgrind runs desktop Firefox fine on Linux, especially on x86 and +x86-64. Firefox for Android and Firefox OS on ARMv7 should also run, +though perhaps not as smoothly. The other architectures supported by +Valgrind on Linux (AARCH64, PPC{32,64}, MIPS{32,64}, S390X) should also +work, in theory. + +MacOS X 10.10 (Yosemite), 64-bit only, works, although it can be a bit +of a rough ride. + +- Expect lower performance and a somewhat higher false positive error + rate than on Linux. +- Valgrind's handling of malloc zones on Yosemite is imperfect. Regard + leak reports with caution. +- Valgrind has been known to cause kernel panics, for unknown reasons. + +Where to get Valgrind +--------------------- + +Linux: Download `Valgrind <http://valgrind.org/>`__ directly, or use +your distribution's package manager (if it has a recent enough version). + +MacOSX: `Get Valgrind trunk from +SVN <http://valgrind.org/downloads/repository.html>`__ and build it. +Don't use 3.10.x or any other tarball. + +Make sure you have Valgrind 3.14 or later, version 3.16.1 is known to work, +3.13.0 did not. Newer versions tend to have better compatibility with both +Firefox's JITs and newer toolchain components (compiler, libc and linker +versions). + +Basics +------ + +Build +~~~~~ + +Build Firefox with the following options, which maximize speed and +accuracy. + +.. code:: + + ac_add_options --disable-jemalloc + ac_add_options --disable-strip + ac_add_options --enable-valgrind + ac_add_options --enable-optimize="-g -O2" + ac_add_options --disable-sandbox + +Run +~~~ + +Note that programs run *much* more slowly under Valgrind than they do +natively. Slow-downs of 20x or 30x aren't unexpected, and it's slower on +Mac than on Linux. Don't try this on an underpowered machine. + +Linux +^^^^^ + +On Linux, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --read-inline-info=yes + +The ``--smc-check`` and ``--vex-iropt-register-updates`` options are +necessary to avoid crashes in JIT-generated code. + +The ``--show-mismatched-frees`` option is necessary due to inconsistent +inlining of ``new`` and ``delete`` -- i.e. one gets inlined but the +other doesn't -- which lead to false-positive mismatched-free errors. + +The ``--read-inline-info`` option improves stack trace readability in +the presence of inlining. + +Also, run with the following environment variable set. + +.. code:: + + G_SLICE=always-malloc + +This is necessary to get the Gnome system libraries to use plain +``malloc`` instead of pool allocators. + +Mac +^^^ + +On Mac, run Valgrind with the following options. + +.. code:: + + --smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access --show-mismatched-frees=no --dsymutil=yes + +The ``--dsymutil`` option ensures line number information is present in +stack traces. + +Advanced usage +-------------- + +Shared suppression files +~~~~~~~~~~~~~~~~~~~~~~~~ + +`/build/valgrind/ <http://mxr.mozilla.org/mozilla-central/source/build/valgrind/>`__ +contains the suppression files used by the periodic Valgrind jobs on +Tinderbox. Some of these files are platform-specific. + +Running mochitests under Valgrind? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To run a mochitest under Valgrind, use the following command. + +.. code:: bash + + $ ./mach mochitest-plain --debugger="valgrind" --debugger-args="$VALGRIND_OPTIONS" relative/path/to/tests + +Where ``$VALGRIND_OPTIONS`` are the options described +:ref:`above <Debugging Firefox With Valgrind>`. You might also +need ``--trace-children=yes`` to trace into child processes. + +As of December 2014 it is possible to do a complete run of +mochitests-plain on Valgrind in about 8 CPU hours on a Core i4910 +(Haswell) machine. Maximum process size is 5.4G, of which about 80% is +in memory. Runs of small subsets of mochitests take far less memory. + +Bits and pieces +~~~~~~~~~~~~~~~ + +For un-released Linux distros (Fedora Rawhide, etc.) you'll need to use +a version of Valgrind trunk build, because fixes for the latest gcc and +glibc versions appear there first. Without them you'll be flooded with +false errors from Memcheck, and have debuginfo reading problems. + +On Linux, code compiled by LLVM at high optimisation levels can cause +Memcheck to report false uninitialised value errors. See +`here <https://bugs.kde.org/show_bug.cgi?id=242137#c3>`__ for an easy +workaround. On Mac, Valgrind has this workaround built in. + +You can make stack traces easier to read by asking for source file names +to be given relative to the root of your source tree. Do this by using +``--fullpath-after=`` to specify the rightmost part of the absolute path +that you don't want to see. For example, if your source tree is rooted +at ``/home/sewardj/MC-20-12-2014``, use ``--fullpath-after=2014/`` to +get path names relative to the source directory. + +The ``--track-origins=yes`` slows down Valgrind greatly, so don't use it +unless you are hunting down a specific uninitialised value error. But if +you are hunting down such an error, it's extremely helpful and worth +waiting for. + +Additional help +--------------- + +The `Valgrind Quick Start +Guide <http://www.valgrind.org/docs/manual/quick-start.html>`__ is short +and worth reading. The `User +Manual <http://valgrind.org/docs/manual/manual.html>`__ is also useful. + +If Valgrind asserts, crashes, doesn't do what you expect, or otherwise +acts up, first of all read this page and make sure you have both Firefox +and Valgrind correctly configured. If that's all OK, try using the +`Valgrind trunk from +SVN <http://www.valgrind.org/downloads/repository.html>`__. Oftentimes +bugs are fixed in the trunk before most users fall across them. If that +doesn't help, consider `filing a bug +report <http://www.valgrind.org/support/bug_reports.html>`__, and/or +mailing Julian Seward or Nick Nethercote. diff --git a/docs/contributing/debugging/debugging_on_macos.rst b/docs/contributing/debugging/debugging_on_macos.rst new file mode 100644 index 0000000000..ac0686067c --- /dev/null +++ b/docs/contributing/debugging/debugging_on_macos.rst @@ -0,0 +1,359 @@ +Debugging On macOS +================== + +This document explains how to debug Gecko-based applications such as +Firefox, Thunderbird, and SeaMonkey on macOS using Xcode. If you want to +debug from the terminal see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. For specific +information on a way to debug hangs, see :ref:`Debugging a hang on macOS <Debugging A Hang On macOS>`. + +Creating a debuggable build +--------------------------- + +First, you need to build the application you're going to debug using +this in your .mozconfig + +.. code:: + + --disable-optimize + --enable-debug-symbols + +you can also add this flag if you want assertions etc. compiled in + +.. code:: + + --enable-debug + +See :ref:`Building Firefox for macOS <Building Firefox On MacOS>` +if you need help creating your own build. + +Debugging Firefox on macOS 10.14+ +--------------------------------- + +macOS 10.14 introduced Notarization and Hardened Runtime features for +improved application security. macOS 10.15 went further, requiring +applications to be Notarized with Hardened Runtime enabled in order to +launch (ignoring workarounds). When run on earlier macOS versions, +Notarization and Hardened Runtime settings have no effect. + +Official Builds +~~~~~~~~~~~~~~~ + +At this time, official builds of Firefox 69 and later are Notarized. +**As a result, it is not possible to attach a debugger to these official +Firefox releases on macOS 10.14+ without disabling System Integrity +Protection (SIP).** This is due to Notarization requiring Hardened +Runtime to be enabled with the ``com.apple.security.get-task-allow`` +entitlement disallowed. **Rather than disabling SIP (which has security +implications), it is recommended to debug with try builds or local +builds. The differences are explained below.** + +try Server Builds +~~~~~~~~~~~~~~~~~ + +In most cases, developers needing to debug a build as close as possible +to the production environment should use a :ref:`try +build <Try Server>`. These +builds enable Hardened Runtime and only differ from production builds in +that they are not Notarized which should not otherwise affect +functionality, (other than the ability to easily launch the browser on +macOS 10.15+ -- see quarantine note below). At this time, developers can +obtain a Hardened Runtime build with the +``com.apple.security.get-task-allow`` entitlement allowed by submitting +a try build and downloading the dmg generated by the "Rpk" shippable +build job. A debugger can be attached to Firefox processes of these +builds. try builds use the ``developer.entitlements.xml`` file from the +source tree while production builds use ``production.entitlements.xml``. +**On macOS 10.15+, downloaded try builds will not launch by default +because Notarization is required. To workaround this problem, remove the +quarantine extended attribute from the downloaded Nightly:** + + ``$ xattr -r -d com.apple.quarantine /Path/to/Nightly.app`` + +Local Builds +~~~~~~~~~~~~ + +Local builds of mozilla-central do not enable Hardened Runtime and hence +do not have debugging restrictions. As a result, some functionality will +be permitted on local builds, but blocked on production builds which +have Hardened Runtime enabled. `Bug +1522409 <https://bugzilla.mozilla.org/show_bug.cgi?id=1522409>`__ was +filed to automate codesigning local builds to enable Hardened Runtime by +default and eliminate this discrepancy. + +To obtain a Hardened Runtime build without using try infrastructure, a +developer can manually codesign builds using the macOS ``codesign(1)`` +command with the ``developer.entitlements.xml`` file from the tree. This +requires creating a codesigning identity. + +Disabling System Integrity Protection (SIP) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If debugging a production build is required, follow Apple's documented +steps for disabling System Integrity Protection (SIP). Note that +disabling SIP bypasses Hardened Runtime restrictions which can mask some +bugs that only occur with Hardened Runtime so it is recommended to test +fixes with SIP enabled. **Disabling SIP has system security implications +that should be understood before taking this step.** + +Creating an Xcode project +------------------------- + +If you try to create a new Xcode project in an existing directory +then Xcode will delete its existing contents (Xcode will warn you +beforehand). To work around that, the steps below have you initialize +the project outside the Mozilla source tree, close the project, copy +the .xcodeproj project "file" into the source tree, and then reopen +the project to finish setting it up. + +Note also that since Xcode 7.3.1 it doesn't seem to be possible to +have the Xcode project live outside the source tree. If you try to do +that then Xcode will simply **copy** the source files under the +project directory rather than link to them which breaks debugging and the +possibility to modify-rebuild-relaunch from inside Xcode. + +These steps were last updated for Xcode 10.3: + +#. Open Xcode, and create a new Project with File > New Project. Select + the "Cross-platform" tab then under the "Other" template group select + the "Empty" project type. the click Next. Name the project and click + Next. Create/select a temporary directory to contain the project and + then click Create. +#. Before going any further, close the project (File > Close Project) + and open Finder. Find the \*.xcodejproj directory in the temporary + directory, move it into your Mozilla source tree, and then + double-click on it to reopen it. +#. In the left-hand pane in Xcode you should see a tree item where the + root item has the project name. If the temporary directory that you + originally created the Xcode project in is under that, right click it + and delete it. Now, right click on the root item, select 'Add files + to "<project-name>"', select all the files and directories in your + source directory, untick "Copy items if needed", then click Add. + (These will then be progressively added under the root item + <project-name> in the left-hand pane. Note that subdirectories may + initially appear to be empty, but they too will progressively be + populated as Xcode processes the sourse files. Once done, you should + be able to open any file quickly by hitting Cmd-Shift-O and typing in + the name of a file.) +#. In the Product menu, select Scheme > New Scheme and name your scheme + (for example, "Debug"). After you click OK, Xcode should open the + settings window for the new scheme. (If not, then open its settings + from the Product > Edit Scheme menu.) +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on "None" and + selecting "Other...". A new dialog titled "Choose an executable to + launch" will pop up. Browse to the ``.app`` file that you want to + debug (``Firefox.app``, ``Nightly``\ ``Debug.app`` etc). The ``.app`` + file is typically found inside the ``dist`` folder in your build + directory. +#. If you are debugging Firefox, Thunderbird, or some other application + that supports multiple profiles, using a separate profile for + debugging purposes is recommended. See "Having a profile for + debugging purposes" below. Select the "Arguments" tab in the scheme + editor, and click the '+' below the "Arguments passed on launch" + field. Add "-P *profilename*", where *profilename* is the name of a + profile you created previously. Repeat that to also add the argument + "-no-remote". +#. Also in the "Arguments" panel, you may want to add an environment + variable MOZ_DEBUG_CHILD_PROCESS set to the value 1 to help with + debugging e10s. +#. Select "Build" from the left of the scheme editor window, and check + that there is nothing listed under Targets (otherwise it may cause + problems when you try to run the executable for debugging since you + will get build errors). +#. Click "Close" to close the scheme editor. + +At this point you can run the application from Xcode, and when you pause +or hit breakpoints it should show open the correct source file at the +correct line. + +Setting up lldb +--------------- + +``lldb`` is the debugger Xcode provides/uses. + +.. warning:: + + One important issue that the Mozilla .lldbinit file fixes is that by + default some breakpoints will be listed as "pending", and Xcode will + not stop at them. If you don't include the Mozilla's .lldbinit, you + must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Firefox with + lldb <Debugging Firefox with LLDB>`. + +The +`.lldbinit <http://searchfox.org/mozilla-central/source/.lldbinit>`__ +file in the source tree imports many useful `Mozilla specific lldb +settings, commands and +formatters <https://searchfox.org/mozilla-central/source/python/lldbutils/README.txt>`__ +into ``lldb``, but you may need to take one of the following steps to +make sure this file is used. + +If you are using ``lldb`` on the command line (independently of Xcode) +and you will always run it from either the top source directory, the +object directory or else the dist/bin subdirectory of the object +directory, then adding the following setting to your ``$HOME/.lldbinit`` +is sufficient: + +:: + + settings set target.load-cwd-lldbinit true + +*However*, if you will run lldb from a different directory, or if you +will be running it indirectly by debugging in Xcode (Xcode always runs +lldb from "/"), then this setting will not help you. Instead, add the +following to your ``$HOME/.lldbinit``: + +:: + + # This automatically sources the Mozilla project's .lldbinit as soon as lldb + # starts or attaches to a Mozilla app (that's in an object directory). + # + # This is mainly a workaround for Xcode not providing a way to specify that + # lldb should be run from a given directory. (Xcode always runs lldb from "/", + # regardless of what directory Xcode was started from, and regardless of the + # value of the "Custom working directory" field in the Scheme's Run options. + # Therefore setting `settings set target.load-cwd-lldbinit true` can't help us + # without Xcode providing that functionality.) + # + # The following works by setting a one-shot breakpoint to break on a function + # that we know will both run early (which we want when we start first start the + # app) and run frequently (which we want so that it will trigger ASAP if we + # attach to an already running app). The breakpoint runs some commands to + # figure out the object directory path from the attached target and then + # sources the .lldbinit from there. + # + # NOTE: This scripts actions take a few seconds to complete, so the custom + # formatters, commands etc. that are added may not be immediately available. + # + breakpoint set --name nsThread::ProcessNextEvent --thread-index 1 --auto-continue true --one-shot true + breakpoint command add -s python + # This script that we run does not work if we try to use the global 'lldb' + # object, since it is out of date at the time that the script runs (for + # example, `lldb.target.executable.fullpath` is empty). Therefore we must + # get the following objects from the 'frame' object. + target = frame.GetThread().GetProcess().GetTarget() + debugger = target.GetDebugger() + + # Delete our breakpoint (not actually necessary with `--one-shot true`): + target.BreakpointDelete(bp_loc.GetBreakpoint().GetID()) + + # For completeness, find and delete the dummy breakpoint (the breakpoint + # lldb creates when it can't initially find the method to set the + # breakpoint on): + # BUG WORKAROUND! GetID() on the *dummy* breakpoint appears to be returning + # the breakpoint index instead of its ID. We have to add 1 to correct for + # that! :-( + dummy_bp_list = lldb.SBBreakpointList(target) + debugger.GetDummyTarget().FindBreakpointsByName("nsThread::ProcessNextEvent", dummy_bp_list) + dummy_bp_id = dummy_bp_list.GetBreakpointAtIndex(0).GetID() + 1 + debugger.GetDummyTarget().BreakpointDelete(dummy_bp_id) + + # "source" the Mozilla project .lldbinit: + os.chdir(target.executable.fullpath.split("/dist/")[0]) + debugger.HandleCommand("command source -s true " + os.path.join(os.getcwd(), ".lldbinit")) + DONE + +see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. for more information. + +Having a profile for debugging purposes +--------------------------------------- + +It is recommended to create a separate profile to debug with, whatever +your task, so that you don't lose precious data like Bookmarks, saved +passwords, etc. So that you're not bothered with the profile manager +every time you start to debug, expand the "Executables" branch of the +"Groups & Files" list and double click on the Executable you added for +Mozilla. Click the plus icon under the "Arguments" list and type "-P +<profile name>" (e.g. "-P MozillaDebug"). Close the window when you're +done. + +Running a debug session +----------------------- + +Make sure breakpoints are active (which implies running under the +debugger) by opening the Product menu and selecting "Debug / Activate +Breakpoints" (also shown by the "Breakpoints" button in the top right +section of the main window). Then click the "Run" button or select "Run" +from the Product menu. + +Setting breakpoints +~~~~~~~~~~~~~~~~~~~ + +Setting a breakpoint is easy. Just open the source file you want to +debug in Xcode, and click in the margin to the left of the line of code +where you want to break. + +During the debugging session, each time that line is executed, the +debugger will break there, and you will be able to debug it. + +.. warning:: + + Note that with the default configuration, some breakpoints will be + listed as "pending", and Xcode will not stop at them. If you don't + include the Mozilla's .lldbinit, you must at least put + ``settings set target.inline-breakpoint-strategy always`` in your + ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with + lldb <Debugging Firefox with LLDB>`. + +Using Firefox-specific lldb commands +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you included the .lldbinit when `Setting up +lldb <#setting-up-lldb>`__, you can use Mozilla-specific lldb commands +in the console, located in the Debug area of Xcode. For example, type +``js`` to see the JavaScript stack. For more information, see :ref:`Debugging +Mozilla with lldb <Debugging Firefox with LLDB>`. + +Debugging e10s child processes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using Xcode to debug child processes created by an e10s-enabled browser +is a little trickier than debugging a single-process browser, but it can +be done. These directions were written using Xcode 6.3.1 + +#. Complete all the steps above under "Creating the Project" +#. From the "Product" menu, ensure the scheme you created is selected + under "Scheme", then choose "Scheme > Edit Scheme" +#. In the resulting popup, click "Duplicate Scheme" +#. Give the resulting scheme a more descriptive name than "Copy of + Scheme" +#. Select "Run" on the left-hand side of the settings window, then + select the "Info" tab. Set the Executable by clicking on the + "Executable" drop-down, and selecting the ``plugin-container.app`` + that is inside the app bundle of the copy of Firefox you want to + debug. +#. On the same tab, under "Launch" select "Wait for executable to be + launched" +#. On the "Arguments" tab, remove all arguments passed on launch. + +Now you're ready to start debugging: + +#. From the "Product" menu, ensure the scheme you created above is + selected under "Scheme" +#. Click the "Run" button. The information area at the top of the window + will show "Waiting for plugin-container to launch" +#. From a command line, run your build of Firefox. When that launches a + child process (for example, when you start to load a webpage), Xcode + will notice and attach to that child process. You can then debug the + child process like you would any other process. +#. When you are done debugging, click the "Stop" button and quit the + instance of Firefox that you were debugging in the normal way. + +For some help on using lldb see :ref:`Debugging Mozilla with +lldb <Debugging Firefox with LLDB>`. + +Other resources +--------------- + +Apple has an extensive list of `debugging tips and +techniques <https://developer.apple.com/library/mac/#technotes/tn2124/_index.html>`__. + +Questions? Problems? +~~~~~~~~~~~~~~~~~~~~ + +Try asking in our Element channels +`#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ or +`#macdev <https://chat.mozilla.org/#/room/#macdev:mozilla.org>`__. diff --git a/docs/contributing/debugging/debugging_on_windows.rst b/docs/contributing/debugging/debugging_on_windows.rst new file mode 100644 index 0000000000..34854c5694 --- /dev/null +++ b/docs/contributing/debugging/debugging_on_windows.rst @@ -0,0 +1,439 @@ +Debugging On Windows +==================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This document explains how to debug Gecko based applications such as +Firefox, Thunderbird, and SeaMonkey on Windows using the Visual C++ IDE. + +If VC++ and your Gecko application hang shortly after you launch the +application under the debugger, see `Problems Loading Debug +Symbols <#problems-loading-debug-symbols>`__. + +Ways to start the debugger +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First of all, it's necessary to install a Visual Studio extension to be +able to follow child processes as they are created. Firefox, in general, +and even in non-e10s mode, does not start the main process directly, it +starts it via a Launcher Process. This means that Visual Studio will +only attach to the first process it finds, and will not hit any +break-point (and even notifies you that it cannot find their location). +`Microsoft Child Process Debugging Power +Tool <https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool>`__ +allows automatically attaching to child processes, such as Web Content +process, GPU process, etc. Enable it by going its configuration menu in +"Debug > Other debugging targets > Child process debugging settings", +and ticking the box. + +If you have followed the steps in :ref:`Building Firefox for +Windows <Building Firefox On Windows>` +and have a local debug build, you can **execute ``./mach run --debug``** +from the same command line. It would open Visual Studio with Firefox's +run options configured. You can **click "Start" button** to run Firefox +then, already attached in the debugger. + +Alternatively, if you have generated the Visual Studio solution, via +``./mach build-backend -b VisualStudio``, opening this solution allows +you to run ``firefox.exe`` directly in the debugger. Making it the +startup project, by right clicking on it (it appears bold when it's the +case) can be useful. Breakpoints are kept across runs, this can be a +good way to debug startup issues. + +**Run the program until you hit an assertion.** You will get a dialog +box asking if you would like to debug. Hit "Cancel". The MSDEV IDE will +launch and load the file where the assertion happened. This will also +create a Visual C++ Mozilla project in the directory of the executable +by default. + +**Attach the debugger to an existing Mozilla process**. In the Visual +Studio, select Debug > Attach to Process. If you want to debug a content +process, you can **hover on the tab** of page you want to debug, which +would show the pid. You can then select the process from dialog opened +from "Attach to Process". For more information, see `Attach to Running +Processes with the Visual Studio +Debugger <http://msdn.microsoft.com/en-us/library/vstudio/3s68z0b3.aspx>`__. + +Debugging Release and Nightly Builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the steps to :ref:`use the Mozilla symbol +server <Using The Mozilla Symbol Server>` and :ref:`source +server <Using The Mozilla Source Server>` + +Creating a Visual C++ project for Firefox +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please refer to :ref:`this <Visual Studio Projects>`. + +Changing/setting the executable to debug +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VC++ 6.0: To change or set the executable to debug, go to Project > +Settings..., Debug tab and select General from the drop down list. +"Executable for debug session:" should show the executable you are +debugging. If it is empty or incorrect, use the arrow button and select +Browse... to locate the executable. + +Command line parameters and environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VC++ 6.0: To change or set the command line options, go to Project > +Settings..., Debug tab and select General from the drop down list. +"Program arguments:" should show the options. + +Some common options would be the URL of the file you want the browser to +open as soon as it starts, starting the Profile Manager, or selecting a +profile. You can also redirect the console output to a file (by adding +"``> filename.txt``" for example, without the quotes). + +In VC 7 and 8 this option is called Project > Properties > Debugging > +Command Arguments. VC 8 also allows you to set environment variables +there. + +Setting breakpoints in DLLs which are not yet loaded in memory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VC++ 6.0: Go to Project > Settings..., Debug tab and select "Additional +DLLs" from the drop down list. Check "Locate Additional DLLs" option. +For each DLL, click the "New" button which creates a new entry and then +hit the "..." buttons which lets you browse to the DLL. You will only be +able to add one DLL at a time. + +VC++ 7.0 automatically finds additional DLLs. + +Customizing the debugger's variable value view +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can customize how Visual C++ displays classes in the variable view. +By default VC++ displays "{...}" and you need to click the small + icon +to expand the members. You can change this behaviour, and make Visual +C++ display whatever data member you want in whatever order, formatter +however you like instead of just "{...}". + +You need to locate a file called "AUTOEXP.DAT" in your Visual C++ +installation. By default it will be: + +VC++ 6.0: + +.. code:: + + C:\Program Files\Microsoft Visual Studio\Common\MSDev98\Bin\AUTOEXP.DAT + +VC++ 7.0: + +.. code:: + + C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Packages\Debugger\AUTOEXP.DAT + +The file has information about the format in the beginning, and after a +little practice you should be well on your way. Here are some entries +that will make your life easier: + +:: + + ;; Mozilla (1.7beta and later) + nsAutoString=<mData,su> + nsString=<mData,su> + nsCString=<mData,s> + nsCAutoString=<mData,s> + nsRect=x=<x,d> y=<y,d> width=<width,d>; height=<height,d> + nsStaticAtomWrapper=<mStaticAtom->mString,s> + nsIAtom=<mString,su> + ; the following are not necessary in vc8 + nsCOMPtr<*>=<mRawPtr,x> + nsRefPtr=<mRawPtr,x> + nsAutoPtr=<mRawPtr,x> + +After you have made the changes and saved the file, you will need to +restart Visual C++ for the changes to take effect. + +For XPCOM Strings (the "external" string API) you can use the following +values: + +:: + + ;; Mozilla (1.9) + ; Internal Strings + nsAString_internal=<mData,su>, length=<mLength,u> + nsACString_internal=<mData,s>, length=<mLength,u> + ; XPCOM Strings + nsAString=<nsStringContainer.v,su>, length=<nsStringContainer.d1,u> + nsACString=<nsCStringContainer.v,s>, length=<nsCStringContainer.d1,u> + nsStringContainer=<v,su>, length=<d1,u> + nsCStringContainer=<v,s>, length=<d1,u> + +There is a more extensive version of this file in progress in +`AutoExpForVC8. <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/AutoExpForVC8>`__ + +Avoiding stepping into certain functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can avoid stepping into certain functions, such as nsCOMPtr methods, +using an undocumented feature of VC. See the blog post `How to Not Step +Into Functions using the Visual C++ +Debugger <http://blogs.msdn.com/andypennell/archive/2004/02/06/69004.aspx>`__ +for details. + +Here are some wildcards you can use (tested with VC 8): + +.. code:: + + nsCOMPtr.*\:\:.*=NoStepInto + (nsG|g)etter_*AddRefs.*=NoStepInto + NS_ConvertUTF.* + ; Might be too broad: + (ns|Promise)[^\:]*[sS]tring.* + ...add common functions to this list + +should probably make a .reg file for easy importing + +Obtaining ``stdout`` and other ``FILE`` handles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Running the following command in the Command Window in Visual Studio +returns the value of ``stdout``, which can be used with various +debugging methods (such as ``nsGenericElement::List``) that take a +``FILE*`` param: + +.. code:: + + Debug.EvaluateStatement {,,msvcr80d}(&__iob_func()[1]) + +(Alternatively you can evaluate ``{,,msvcr80d}(&__iob_func()[1])`` in +the QuickWatch window) + +Similarly, you can open a file on the disk using ``fopen``: + +.. code:: + + >Debug.EvaluateStatement {,,msvcr80d}fopen("c:\\123", "w") + 0x10311dc0 { ..snip.. } + >Debug.EvaluateStatement ((nsGenericElement*)0x03f0e710)->List((FILE*)0x10311dc0, 1) + <void> + >Debug.EvaluateStatement {,,msvcr80d}fclose((FILE*)0x10311dc0) + 0x00000000 + +Note that you may not see the debugging output until you flush or close +the file handle. + +Disabling ASSERTIONS +~~~~~~~~~~~~~~~~~~~~ + +There are basically two ways to disable assertions. One requires setting +an environment variable, while the other affects only the currently +running program instance in memory. + +Environment variable +^^^^^^^^^^^^^^^^^^^^ + +There is an environment variable that can disable breaking for +assertions. This is how you would normally set it: + +.. code:: + + set XPCOM_DEBUG_BREAK=warn + +The environment variable takes also other values besides ``warn``, see +``XPCOM_DEBUG_BREAK`` for more details. + +Note that unlike Unix, the default for Windows is not warn, it's to pop +up a dialog. To set the environment variable for Visual Studio, use +Project > Properties > Debugging > Environment and click the little box. +Then use + +.. code:: + + XPCOM_DEBUG_BREAK=warn + +Changing running code +^^^^^^^^^^^^^^^^^^^^^ + +You normally shouldn't need to do this (just quit the application, set +the environment variable described above, and run it again). And this +can be **dangerous** (like **trashing your hard disc and corrupting your +system**). So unless you feel comfortable with this, don't do it. **You +have been warned!** + +It is possible to change the interrupt code in memory (which causes you +to break into debugger) to be a NOP (no operation). + +You do this by running the program in the debugger until you hit an +assertion. You should see some assembly code. One assemly code +instruction reads "int 3". Check the memory address for that line. Now +open memory view. Type/copy/drag the memory address of "int 3" into the +memory view to get it to update on that part of the memory. Change the +value of the memory to "90", close the memory view and hit "F5" to +continue. + +| Confused? See the screenshot below: +| |Screenshot of disabling assertions| + +VC++ 7.0? + +Automatically handling ASSERTIONS without a debugger attached +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When an assertion happens and there is not a debugger attached, a small +helper application +(```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__) +is run. That application can automatically select a response to the "Do +you want to debug" dialog instead of prompting if you configure it, for +more info, see +```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__. + +Debugging optimized builds +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To effectively debug optimized builds, you should enable debugging +information which effectively leaves the debug symbols in optimized code +so you can still set breakpoints etc. Because the code is optimized, +stepping through the code may occasionally provide small surprises when +the debugger jumps over something. + +You need to make sure this configure parameter is set: + +.. code:: + + --enable-debugger-info-modules=yes + +You can also choose to include or exclude specific modules. This is +particularly useful to avoid linking layout with debugging information. + +Console debugging +~~~~~~~~~~~~~~~~~ + +When printing to STDOUT from a content process, the console message will +not appear on Windows. One way to view it is simply to disable e10s +(``./mach run --disable-e10s``) but in order to debug with e10s enabled +one can run + +:: + + ./mach run ... 2>&1 | tee + +It may also be necessary to disable the content sandbox +(``MOZ_DISABLE_CONTENT_SANDBOX=1 ./mach run ...``). + +Running two instances of Mozilla simultaneously +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can run two instances of Mozilla (e.g. debug and optimized) +simultaneously by setting the environment variable ``MOZ_NO_REMOTE``: + +.. code:: + + set MOZ_NO_REMOTE=1 + +Or, starting with Firefox 2 and other Gecko 1.8.1-based applications, +you can use the ``-no-remote`` command-line switch instead (implemented +in +`bug 325509 <https://bugzilla.mozilla.org/show_bug.cgi?id=325509>`__). + +You can also specify the profile to use with the ``-P profile_name`` +command-line argument. + +Debugging JavaScript +~~~~~~~~~~~~~~~~~~~~ + +Use `Venkman <https://developer.mozilla.org/en-US/docs/Archive/Mozilla/Venkman>`__, the JavaScript Debugger for Mozilla. + +You can use helper functions from +`nsXPConnect.cpp <https://searchfox.org/mozilla-central/source/js/xpconnect/src/nsXPConnect.cpp>`__ +to inspect and modify the state of JavaScript code from the MSVS +debugger. + +For example, to print current JavaScript stack to stdout, evaluate this +in QuickWatch window: + +.. code:: + + {,,xul}DumpJSStack() + +Visual C++ will show you something in the quick watch window, but +not the stack, you have to look in the OS console for the output. + +Also this magical command only works when the VC++ stack is in certain +states. It works when you have js_Interpret() in the newest stackframe + +Debugging minidumps +~~~~~~~~~~~~~~~~~~~ + +See :ref:`debugging a minidump <Debugging A Minidump>`. + +Debugging tinderbox builds +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See `Running Windows Debug Builds <https://developer.mozilla.org/en-US/docs/Archive/Mozilla/Running_Windows_Debug_Builds>`__ + +Problems Loading Debug Symbols +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If both your application and Visual C++ hang shortly after launching the +application under the debugger, you may be hitting a known deadlock in +the way Visual Studio downloads debug symbols for the system libraries; +see +https://connect.microsoft.com/VisualStudio/feedback/details/422970/hang-loading-rasapi32-pdb-when-using-symbol-server. + +There are two ways to work around this problem: + +#. Turn off automatic symbol downloading for system libraries: in Tools + > Options > Debugging > Symbols, uncheck the Microsoft symbol server. +#. Pre-load all the Windows debug symbols. These instructions apply to + Visual Studio 10 on Windows 7; other software versions likely need to + have file paths adjusted. + + #. Locate the Microsoft utility "SymChk.exe" on your system (it will + likely be in the installation directory of your Windows Debugging + Tools). + + #. Find the directory where Visual Studio caches downloaded symbols; + in VC++ 10 open the menu to Tools > Options > Debugging > Symbols + and copy the field "Cache symbols in this directory". + + #. In a command window, run + + :: + + symchk.exe /r C:\windows\SysWOW64\ /s "SRV*<your cache symbols directory>\MicrosoftPublicSymbols*http://msdl.microsoft.com/download/symbols" + + | + | Note the "``\MicrosoftPublicSymbols``" appended to the cache + directory configured in Visual Studio. + +Downloading all symbols can take a long time; you can replace +C:\windows\SysWOW64\\ with the name of a single .DLL to download symbols +only for the specific libraries you are trying to debug. Unfortunately, +it's hard to know which symbols to download without having VS hang and +seeing the "Downloading symbols for <library>" status at the bottom left +of the main window. + +Problems post-mortem debugging on Windows 7 SP1 x64? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you attempt to use ``NS_DebugBreak`` etc to perform post-mortem +debugging on a 64bit Windows 7, but as soon as you try and continue +debugging the program crashes with an Access Violation, you may be +hitting a Windows bug relating to AVX support. For more details, +including a work-around see `this blog +post <http://www.os2museum.com/wp/?p=960>`__ or `this social.msdn +thread <http://social.msdn.microsoft.com/Forums/vstudio/en-US/392ca62c-e502-42d9-adbc-b4e22d5da0c3/jit-debugging-32bit-app-crashing-with-access-violation>`__. +(And just in-case those links die, the work-around is to execute + +:: + + bcdedit /set xsavedisable 1 + +from an elevated command-prompt to disable AVX support.) + +Got a tip? +~~~~~~~~~~ + +If you think you know a cool Mozilla debugging trick, feel free to +discuss it with `#developers <https://chat.mozilla.org/#/room/#developers:mozilla.org>`__ and +then post it here. + +.. |Screenshot of disabling assertions| image:: https://developer.mozilla.org/@api/deki/files/420/=Win32-debug-nop.png + :class: internal diff --git a/docs/contributing/debugging/img/crashlist.jpg b/docs/contributing/debugging/img/crashlist.jpg Binary files differnew file mode 100644 index 0000000000..4cb376d0f2 --- /dev/null +++ b/docs/contributing/debugging/img/crashlist.jpg diff --git a/docs/contributing/debugging/img/reporter.jpg b/docs/contributing/debugging/img/reporter.jpg Binary files differnew file mode 100644 index 0000000000..0a62cda4e1 --- /dev/null +++ b/docs/contributing/debugging/img/reporter.jpg diff --git a/docs/contributing/debugging/process_dump_task_manager.rst b/docs/contributing/debugging/process_dump_task_manager.rst new file mode 100644 index 0000000000..d345171856 --- /dev/null +++ b/docs/contributing/debugging/process_dump_task_manager.rst @@ -0,0 +1,69 @@ +How to get a process dump with Windows Task Manager +=================================================== + +Introduction +------------ + +When tracking down the causes of process hangs, it is often helpful to +obtain a process dump while the process is experiencing a hang. This +article describes how to get a process dump with Task Manager on +Windows. (To get a process dump for Thunderbird or some other product, +substitute the product name where ever you see Firefox in these +instructions.) + + +Caution +------- + +The memory dump that will be created through this process is a complete +snapshot of the state of Firefox when you create the file, so it +contains URLs of active tabs, history information, and possibly even +passwords depending on what you are doing when the snapshot is taken. It +is advisable to create a new, blank profile to use when reproducing the +hang and capturing the memory dump. Please ask for help doing this! + + +Requirements +------------ + +Windows + To get a process dump, you need to be using Windows Vista or above. +A Firefox nightly or release + You need a Firefox version for which symbols are available from the + :ref:`symbol server <Using The Mozilla Symbol Server>`. You + can use any `official nightly + build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released + version of Firefox from Mozilla. You can find the latest trunk + nightly builds under + `http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/>`__. + + +Creating the Dump File +---------------------- + +Ensure that Firefox is not already running. + + +Run Firefox, reproduce the hang +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Start Firefox and perform whatever steps are necessary to cause Firefox +to hang. Once the browser hangs, continue with the steps below. + + +After the hang +~~~~~~~~~~~~~~ + +#. Open Windows Task Manager (CTRL+SHIFT+ESC). +#. Find Firefox.exe among the list of processes. +#. Right-click Firefox.exe and select "Create dump file". Task manager + should indicate where the dump file was written to. + + +See also +-------- + +- :ref:`How to get a stacktrace for a bug report <How to get a stacktrace for a bug report>` +- `How to create a user-mode process dump file in Windows Vista and in + Windows 7 + (MSDN) <https://docs.microsoft.com/en-us/windows/client-management/generate-kernel-or-complete-crash-dump#manually-generate-a-memory-dump-file>`__ diff --git a/docs/contributing/debugging/stacktrace_report.rst b/docs/contributing/debugging/stacktrace_report.rst new file mode 100644 index 0000000000..dfbfb714ca --- /dev/null +++ b/docs/contributing/debugging/stacktrace_report.rst @@ -0,0 +1,152 @@ +How to get a stacktrace for a bug report +======================================== + +If you file a bug report in Bugzilla about a crash you should include a +stacktrace (call stack) in your report. A stacktrace will tell Mozilla +developers what crashed and provide a starting point for investigating +its cause. This article describes how to use the Mozilla Crash Reporter +(Breakpad) to get a crash ID, which our engineers can use to get a +stacktrace, and alternative ways to get a stacktrace if you can't get a +crash ID. + +Requirements +------------ + +You need a binary build of Firefox from +`Mozilla.org <https://www.mozilla.org/firefox/>`__. SeaMonkey and +Thunderbird also support crash reporting. + +Mozilla's crash report server currently only has debug information for +Mozilla builds and thus the crash reporter cannot work if you use a +build from a Linux distribution or if you compile from source code. In +these cases you will need to use one of the :ref:`alternative +methods <Alternative ways to get a stacktrace>` listed below. + +.. note:: + + **Note:** When filing a crash report, it is important to know whether + the crash occurs with `Firefox safe + mode <http://support.mozilla.com/kb/Safe+Mode>`__. This helps + engineers determine whether a particular + `extension <http://support.mozilla.com/kb/Troubleshooting+extensions+and+themes>`__ + or + `plugin <http://support.mozilla.com/kb/Troubleshooting+plugins>`__ + is the cause of the crash. + + +How to get a crash ID with the Mozilla Crash Reporter +----------------------------------------------------- + +1 - Crash and submit a report to the system. + +.. image:: img/reporter.jpg + +The Mozilla Crash Reporter window should automatically come up after Firefox crashes. +If you have any additional information about the crash, such as additional detail on +what you were doing at the time that may have triggered the crash, please enter it +into the comments box. Be sure that you **check the "Tell Mozilla about this crash"** +checkbox and click the restart button. The crash reporter should now submit the +crash report and Firefox should open again. + +.. note:: + + The "Details" button gives additional data about the incident, + however this is not useful in a bug report. + + +2 - Tell us the ID of the report you submitted. + +.. image:: img/crashlist.jpg + +To access all of your submitted reports type "about:crashes" into the Firefox address bar +and press enter. Firefox should open a list of IDs for your submitted crash reports. +Copy two or three of the IDs for the appropriate crashes and paste them into your +Bugzilla report. Please check the listed times to avoid copying the ID of an unrelated +crash report. + +.. note:: + + You can prefix a "bp-" to the beginning of an ID to make Bugzilla turn it + into a link: bp-a70759c6-1295-4160-aa30-bc4772090918 + + +How to get the crash ID if Firefox crashes on startup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If Firefox crashes on startup you can still access your submitted crash +reports. Crash reports are accessible from all Firefox profiles, so if a +`new +profile <https://support.mozilla.org/kb/profile-manager-create-remove-switch-firefox-profiles>`__ +does not crash you can use it to access them through "about:crashes" as above. + + +Accessing crash report IDs outside of Firefox +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you cannot load Firefox at all you can find the crash report files at +this location depending on your operating system: + +* Windows : ``%APPDATA%\Mozilla\Firefox\Crash Reports\submitted\`` +* macOS : ``~/Library/Application Support/Firefox/Crash Reports/submitted/`` +* Linux : ``~/.mozilla/firefox/Crash Reports/submitted/`` + +Each file in this folder contains one submitted crash report ID. You can +check the modified or creation time for each file to discern which crash +reports are relevant to your bug report. + + +Alternative ways to get a stacktrace +------------------------------------ + +If the Mozilla crash reporter doesn't come up or isn't available you +will need to obtain a stacktrace manually: + + +Windows +~~~~~~~ + +See the article :ref:`Create a stacktrace with Windbg` for information +on how to do this. + +For a full process dump, see :ref:`How to get a process dump with Windows +Task Manager`. + + +macOS +~~~~~ + +Run /Applications/Utilities/Console.app. Expand "~/Library/Logs" and +"CrashReporter", then look for logs for "firefox-bin". + + +Linux +~~~~~ + +Note that for most distros, the package you need to get symbols for will +be something like "xulrunner", not "firefox". + + +Crash reports files on your computer +------------------------------------ + +When Breakpad initially catches a crash it first writes crash report +files (e.g. .dump and .extra files) into the 'pending' subdirectory of +its 'Crash Reports' directory. + +If Breakpad successfully sends the crash report to the reporting server +then, by default, the files added to the 'pending' subdirectory for the +crash are removed, and a .txt file is placed in the 'submitted' +directory containing the crash ID created by the reporting server. + +If you want Breakpad to leave the .dump and .extra files on your +computer so that you can examine them locally, then set the +MOZ_CRASHREPORTER_NO_DELETE_DUMP environment variable to 1. + +- Ubuntu: `Instructions from the Ubuntu + Team <https://wiki.ubuntu.com/MozillaTeam/Bugs#Obtain%20a%20backtrace%20from%20an%20apport%20crash%20report%20(using%20gdb)>`__ +- openSUSE: `General instructions from + openSUSE <https://en.opensuse.org/openSUSE:Bugreport_application_crashed>`__ +- Fedora: `Capturing Stack + Traces <https://fedoraproject.org/wiki/StackTraces>`__ +- Gentoo: `Debugging using + GDB <https://wiki.gentoo.org/wiki/Debugging_with_GDB>`__ diff --git a/docs/contributing/debugging/stacktrace_windbg.rst b/docs/contributing/debugging/stacktrace_windbg.rst new file mode 100644 index 0000000000..a0d2abfc25 --- /dev/null +++ b/docs/contributing/debugging/stacktrace_windbg.rst @@ -0,0 +1,232 @@ +How to get a stacktrace with WinDbg +=================================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Introduction +------------ + +Sometimes you need to get a stacktrace (call stack) for a crash or hang +but `Breakpad <http://kb.mozillazine.org/Breakpad>`__ fails because it's +a special crash or a hang. This article describes how to get a +stacktrace in those cases with WinDbg on Windows. (To get a stacktrace +for Thunderbird or some other product, substitute the product name where +ever you see Firefox in this instructions.) + +Requirements +------------ + +To get such a stacktrace you need to install the following software: + +Debugging Tools for Windows +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Microsoft distributes the Debugging Tools for Windows for free, those +include WinDbg which you will need here. Download it from `Install +Debugging Tools for +Windows <https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk>`__. +(*You'll want the 32-bit version*, even if you are using a 64-bit +version of Windows) Then install it, the standard settings in the +installation process are fine. + +A Firefox nightly or release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You need a Firefox version for which symbols are availables from the +:ref:`symbol server <Using The Mozilla Symbol Server>` to use +with WinDbg. You can use any `official nightly +build <https://ftp.mozilla.org/pub/firefox/nightly/>`__ or released +version of Firefox from Mozilla. You can find the latest trunk nightly +builds under +`http://ftp.mozilla.org/pub/mozilla.o.../latest-trunk/ <https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/>`__. + + +Debugging +--------- + +To begin debugging, ensure that Firefox is not already running and open +WinDbg from the Start menu. (Start->All Programs->Debugging Tools for +Windows->WinDbg) Next, open the **"File"** menu and choose **"Open +Executable..."**. In the file chooser window that appears, open the +firefox.exe executable in your Firefox program folder (C:\Program +Files\Mozilla Firefox). + +You should now see a "Command" text window with debug output at the top +and an input box at the bottom. Before debugging can start, several +commands must be entered into the one-line input box at the bottom of +the Command window. + +.. note:: + + Tip: All commands must be entered exactly as written, one line at a + time, into the bottom of the Command box. + + - Copying and pasting each line is the easiest method to avoid + mistakes + - Some commands start with a period (.) or a pipe character (|), + which is required. (The keystroke for a pipe character on US + keyboards is SHIFT+\) + - Submit the log file on a bug or via the support site, even if + nothing seems to happen during the debug process + + +Start debugging +~~~~~~~~~~~~~~~ + +Now that Firefox is opened in the debugger, you need to configure your +WinDbg to download symbols from the Mozilla symbol server. To load the +symbols, enter the three commands below, pressing enter after each one. +(More details are available at :ref:`symbol server <Using The Mozilla Symbol Server>`.) + +:: + + .sympath SRV*c:\symbols*http://symbols.mozilla.org/firefox;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols + .symfix+ c:\symbols + .reload /f + +Now wait for the symbols to download. This may take some time depending +on your connection speed; the total size of the Mozilla and Microsoft +symbols download is around 1.4GB. WinDbg will show "Busy" at the bottom +of the application window until the download is complete. + +Once the download is complete, you need to configure WinDbg to examine +child processes, ignore a specific event caused by Flash Player, and +record a log of loaded modules. You will also want to open a log file to +save data you collect. To do this, enter these four commands, pressing +enter after each one. + +:: + + .logopen /t c:\temp\firefox-debug.log + .childdbg 1 + .tlist + sxn gp + lm + +If you see firefox.exe listed in the output from .tlist more than once, +then you are already running the application and need to close the +running instance first before you start debugging, otherwise you won't +get useful results. + +Now run Firefox by opening the **Debug** menu and clicking **Go**. +**While Firefox is running, you will not be able to type any commands +into the debugger.** After it starts, try to reproduce the crash or +hanging issue that you are seeing. + +.. note:: + + If Firefox fails to start, and you see lines of text followed by a + command prompt in the debugger, a "breakpoint" may have been + triggered. If you are prompted for a command but don't see an error + about a crash, go back to the **Debug** menu and press **Go**. + +Once the browser crashes, you will see an error (such as "Access +violation") in the WinDbg Command window. If Firefox hangs and there is +no command prompt available in the debugger, open the **Debug** menu and +choose **Break.** Once the browser has crashed or been stopped, continue +with the steps below. + + +After the crash or hang +~~~~~~~~~~~~~~~~~~~~~~~ + +You need to capture the debug information to include in a bug comment or +support request. Enter these three commands, one at a time, to get the +stacktrace, crash/hang analysis and log of loaded modules. (Again, press +Enter after each command.) + +:: + + ~* kp + !analyze -v -f + lm + +After these steps are completed, find the file +**c:\temp\firefox-debug-(Today's Date).txt** on your hard drive. To +provide the information to the development community, submit this file +with a `support request <https://support.mozilla.com/>`__ or attach it +to a related bug on `Bugzilla <https://bugzilla.mozilla.org/>`__. + + +Producing a minidump +~~~~~~~~~~~~~~~~~~~~ + +Sometimes the stacktrace alone is not enough information for a developer +to figure out what went wrong. A developer may ask you for a "minidump" +or a "full memory dump", which are files containing more information +about the process. :ref:`You can easily produce minidumps from WinDBG and +provide them to developers <Capturing a minidump>`. + +FAQ + +Q: I am running Windows 7 (32-bit or 64-bit) and I see an exception in +the WinDbg command window that says 'ntdll32!LdrpDoDebuggerBreak+0x2c' +or 'ntdll32!LdrpDoDebuggerBreak+0x30'. What do I do now? + +A: If you see 'int 3' after either of those exceptions, you will need to +execute the following commands in WinDbg. + +:: + + bp ntdll!LdrpDoDebuggerBreak+0x30 + bp ntdll!LdrpDoDebuggerBreak+0x2c + eb ntdll!LdrpDoDebuggerBreak+0x30 0x90 + eb ntdll!LdrpDoDebuggerBreak+0x2c 0x90 + +| Make sure you enter them one at a time and press enter after each one. + If you use the 64-bit version of Windows, you need to replace "ntdll" + in these commands with "ntdll32". +| Q: The first four frames of my stack trace look like this: + +:: + + 0012fe20 7c90e89a ntdll!KiFastSystemCallRet + 0012fe24 7c81cd96 ntdll!ZwTerminateProcess+0xc + 0012ff20 7c81cdee kernel32!_ExitProcess+0x62 + + 0012ff34 6000179e kernel32!ExitProcess+0x14 + +This looks wrong to me?! + +A: You ran the application without the "Debug child processes also" +check box being checked. You need to detach the debugger and open the +application again, this time with the check box being checked. + +Q: WinDbg tells me that it is unable to verify checksum for firefox.exe. +Is this normal? + +A: Yes, this is normal and can be ignored. + +Q: Should I click yes or no when WinDbg asks me to "Save information for +workspace?" + +A: Click yes and WinDbg will save you from having to enter in the symbol +location for Firefox.exe in the future. Click no if you'd rather not +having WinDbg save this information. + +Q: I'm seeing "wow64" on top of each thread, is that ok ? + +A: No, you are running a 64 bit version of Windbg and trying to debug a +32 bit version of the mozilla software. Redownload and install the 32 +bit version of windbg. + + +Troubleshooting: Symbols will not download +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If symbols will not download no matter what you do, the problem may be +that Internet Explorer has been set to the **Work Offline** mode. You +will not receive any warnings of this in Windbg, Visual C++ or Visual +Studio. Even using the command line with symchk.exe to download symbols +will fail. This is because Microsoft uses Internet Explorer's internet & +proxy settings to download the symbol files. Check the File menu of +Internet Explorer to ensure "Work Offline" is unchecked. + + +See also +-------- + +- :ref:`symbol server <Using The Mozilla Symbol Server>` Maps addresses to human readable strings. +- :ref:`source server <Using The Mozilla Source Server>` Maps addresses to source code lines diff --git a/docs/contributing/debugging/understanding_crash_reports.rst b/docs/contributing/debugging/understanding_crash_reports.rst new file mode 100644 index 0000000000..667815a52e --- /dev/null +++ b/docs/contributing/debugging/understanding_crash_reports.rst @@ -0,0 +1,328 @@ +Understanding Crash Reports +=========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +If a user experiences a crash they will be prompted to submit a raw +crash report, which is generated by Breakpad. The raw crash report is +received by `Socorro <https://github.com/mozilla/socorro>`__ which +`creates <https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py>`__ +a processed crash report. The processed crash report is based on the raw +crash report but also has a signature, classifications, and a number of +improved fields (e.g. OS, product, version). Many of the fields in both +the raw crash report and the processed crash report are viewable and +searchable on `crash-stats <https://crash-stats.mozilla.org/>`__. +Although there are two distinct crash reports, the raw and the +processed, people typically talk about a single "crash report" because +crash-stats mostly presents them in a combined way. + +Each crash report contains a wealth of data about the crash +circumstances. Despite this, many crash reports lack sufficient data for +a developer to understand why the crash occurred. As well as providing a +general overview, this page aims to highlight parts of a crash report +that may provide non-obvious insights. + +Note that most crash report fields are visible, but a few +privacy-sensitive parts of it are only available to users who are logged +in and have "minidump access". A relatively small number of users have +minidump access, and they are required to follow certain rules. For +access, see the `Protected Data Access docs on Crash Stats +<https://crash-stats.mozilla.org/documentation/protected_data_access/>`__. + +Each crash report has the following tabs: Details, Metadata, Modules, +Raw Dump, Extensions, and (optional) Correlations. + +Details tab +----------- + +The Details tab is the first place to look because it contains the most +important pieces of information. + +Primary fields +~~~~~~~~~~~~~~ + +| The first part of the Details tab shows a table containing the most + important crash report fields. It includes such things as when the + crash occurred, in which product and version, the crash kind, and + various details about the OS and configuration of the machine on which + the crash occurred. The following screenshot shows some of these + fields. +| |Example fields in the "Details" tab of a crash report| + +All fields have a tool-tip. For many fields, the tool-tip describes its +meaning. For all fields, the tool-tip indicates the key to use when you +want to do searches involving this field. (The field name is usually but +not always similar to the search key. E.g. the field "Adapter Device ID" +has the search key "adapter_device_id".) These descriptions are shown in +the `SuperSearchFields +API <https://crash-stats.mozilla.org/api/SuperSearchFields/>`__ and can be +`modified in super_search_fields.py <https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py>`__ +or by writing up a `bug in Socorro <https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro>`__. + +The fields present in this tab vary depending on the crash kind. Not all +fields are always present. + +The "Signature" field is the main identifier or label for a crash report. +Rather than considering each crash report in isolation, we want to put +crash reports into clusters so we can deal with groups of them at once. +An ideal clustering algorithm would put all crash reports with the same +root cause into a single cluster, and all crash reports with different +root causes into different clusters. The crash signature is our +imperfect but still useful attempt at such an algorithm. Most crash +signatures are based on the crashing stack trace, but some +special-purpose annotations are used to indicate particular kinds of +crashes. + +- ``Abort``: A controlled abort, e.g. via ``NS_RUNTIMEABORT``. + (Controlled aborts that occur via ``MOZ_CRASH`` or + ``MOZ_RELEASE_ASSERT`` currently don't get an ``Abort`` annotation, + but they do get a "MOZ_CRASH Reason" field.) +- ``OOM | <size>``, where ``<size>`` is one of ``large``, ``small``, + ``unknown``: an out-of-memory (OOM) abort. The ``<size>`` annotation + is determined by the "OOM Allocation Size" field; if that field is + missing ``<size>`` will be ``unknown``. +- ``hang``: a hang prior to shutdown. +- ``shutdownhang``: a hang during shutdown. +- ``IPCError-browser``: a problem involving IPC. If the parent Firefox + process detects that the child process has sent broken or + unprocessable IPDL data, or is not shutting down in a timely manner, + it kills the child process with a crash report. These crashes will + now have a signature that indicates why the process was killed, + rather than the child stack at the moment. + +When no special-purpose annotation is present and the signature begins +with a stack frame, it's usually a vanilla uncontrolled crash. The crash +cause can be determined from the "Crash Reason" field. Most commonly +it's a bad memory access. In that case, on Windows you can tell from the +reason field if the crash occurred while reading, writing or executing +memory (e.g. ``EXCEPTION_VIOLATION_ACCESS_READ`` indicates a bad memory +read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you +cannot tell from this field what kind of memory access it was. + +See `this +file <https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst>`__ +for a detailed explanation of the crash report signature generation +procedure, and for information on how modify this procedure. + +There are no fields that uniquely identify the user that a crash report +came from, but if you want to know if multiple crashes come from a +single user the "Install Time" field is a good choice. Use it in +conjunction with other fields that don't change, such as those +describing the OS or graphics card, for additional confidence. + +For bad memory accesses, the "Crash Address" field can give additional +indications what went wrong. + +- 0x0 is probably a null pointer deference[*]. +- Small addresses like 0x8 can indicate an object access (e.g. + ``this->mFoo``) via a null ``this`` pointer. +- Addresses like 0xfffffffffd8 might be stack accesses, depending on + the platform[*]. +- Addresses like 0x80cdefd3 might be heap accesses, depending on the + platform. +- Addresses may be poisoned: 0xe4 indicates the address comes from + memory that has been allocated by jemalloc but not yet initialized; + 0xe5 indicates the address comes from memory freed by jemalloc. The + JS engine also has multiple poison values defined in + ``js/src/jsutil.h``. + +[*] Note that due to the way addressing works on x86-64, if the crash +address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for +a Windows crash report, it's highly likely that the value is incorrect. +(There is a `bug +report <https://bugzilla.mozilla.org/show_bug.cgi?id=1493342>`__ open +for this problem.) You can sanity-check these crashes by looking at the +raw dump or minidump in the Raw Dump tab (see below). + +Note that for non-release builds the "Version" field represents multiple +different builds since nightly and beta version numbers are reused for +builds created over a series of days until the version number is bumped. +(The "Build ID" field can disambiguate.) It's not currently possible to +`restrict searches to a given version or +later <https://bugzilla.mozilla.org/show_bug.cgi?id=1401517>`__ (using +>= with a build ID and a given release channel may work around this). + +Some fields, such as "URL" and "Email Address", are privacy-sensitive +and are only visible to users with minidump access. + +The Windows-only "Total Virtual Memory" field indicates if the Firefox +build and OS are 32-bit or 64-bit. + +- A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows. +- A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows + (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox. +- A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit + Firefox. (The "Build Architecture" field should be "amd64" in this + case.) + +Some crash reports have a field "ContainsMemoryReport", which indicates +that the crash report contains a memory report. This memory report will +have been made some time before the crash, at a time when available +memory was low. In this case, a number of key measurements from the +memory report are shown in the Details tab, each one having a field name +starting with "MR:", short for "memory report". The full memory report +can be obtained in the Raw Dump tab (see below). + +Bug-related information +~~~~~~~~~~~~~~~~~~~~~~~ + +The second part of the Details tab shows bug-related information, as the +following screenshot shows. + +|Information relating to bug reports in the "Details" tab of a crash +report| + +The "Report this bug in" links can be used to easily file bug reports. +Each one links to a Bugzilla bug report creation page that has various +fields pre-filled, such as the crash signature. + +The "Related Bugs" section shows related bug reports, as determined by +the crash signature. + +Stack traces +~~~~~~~~~~~~ + +The third part of the Details tab shows the stack trace and thread +number of the crashing thread, as the following screenshot shows. + +|Information relating to threads in the "Details" tab of a crash report| + +Each stack frame has a link to the source code, when possible. If a +crash is new, the regressing changeset can often be identified by +looking for recent changes in the blame annotations for one or more of +the top stack frames. Blame annotations are also good for identifying +who might know about the code in question. + +Sometimes the highlighted source code is puzzling, e.g. the identified +line may not touch memory even though the crash is memory-related. This +can be caused by compiler optimizations. It's often better to look at +the disassembly (e.g. in a minidump) to understand exactly what code is +being executed. + +Stack frame entries take on a variety of forms. + +- The simplest are functions names, such as ``NS_InitXPCOM2``. +- Name/address pairs such as ``nss3.dll@0x1eb720`` are within system + libraries. +- Names such as ``F1398665248_____________________________`` ('F' + followed by many numbers then many underscores) are in Flash. +- Addresses such as ``@0xe1a850ac`` may indicate an address that wasn't + part of any legitimate code. If an address such as this occurs in the + first stack frame, the crash may be + `exploitable <https://developer.mozilla.org/en-US/docs/Mozilla/Security/Exploitable_crashes>`__. + +Stack traces for other threads can be viewed by clicking on the small +"Show other threads" link. + +If the crash report is for a hang, the crashing thread will be the +"watchdog" thread, which exists purely to detect hangs; its top stack +frame will be something +like\ :literal:`mozilla::`anonymous namespace'::RunWatchdog`. In that +case you should look at the other threads' stack traces to determine the +problem; many of them will be waiting on some kind of response, as shown +by a top stack frame containing a function like +``NtWaitForSingleObject`` or ``ZwWaitForMultipleObjects``. + +Metadata tab +------------ + +The Metadata tab is similar to the first part of the Details tab, +containing a table with various fields. These are the fields from the +raw crash report, ordered alphabetically by field name, but with +privacy-sensitive fields shown only to users with minidump access. There +is some overlap with the fields shown in the Details tab. + +Modules tab +----------- + +The modules tab shows all the system libraries loaded at the time of the +crash, as the following screenshot shows. + +|Table of modules in the "Modules" tab of a crash report| + +On Windows these are mostly DLLs, on Mac they are mostly ``.dylib`` +files, and on Linux they are mostly ``.so`` files. + +This information is most useful for Windows crashes, because DLLs loaded +by antivirus software or malware often cause Firefox to crash. +Correlations between loaded modules and crash signatures can be seen in +the "Correlations" tab (see below). + +`This page <https://support.mozilla.org/en-US/kb/helping-crashes>`__ +says that files lacking version/debug identifier/debug filename are +likely to be malware. + +Raw Dump tab +------------ + +The first part of the Raw Dump tab shows the raw crash report, in JSON +format. Once again, privacy-sensitive fields are shown only to users +with minidump access. + +|JSON data in the "Raw Dump" tab of a crash report| + +For users with minidump access, the second part of the Raw Dump tab has +some links, as the following screenshot shows. + +|Links to downloadable files in the "Raw Dump" tab of a crash report| + +These links are to the following items. + +#. A minidump. Minidumps can be extremely useful in understanding a + crash report; see :ref:`this page <Debugging A Minidump>` for an + explanation how to use them. +#. The aforementioned JSON raw crash report. +#. The memory report contained within the crash report. Only crash + reports with the ``ContainsMemoryReport`` field set will have this + link. +#. The unredacted crash report, which has additional information. + +Extensions tab +-------------- + +The Extensions tab shows which extensions are installed and enabled. + +|Table of extensions in the "Extensions" tab of a crash report| + +Usually it just shows an ID rather than the proper extension name. + +Note that several extensions ship by default with Firefox and so will be +present in almost all crash reports. (The exact set of default +extensions depends on the release channel.) The least obvious of these +has an Id of ``{972ce4c6-7e08-4474-a285-3208198ce6fd}``, which is the +default Firefox theme. Some (but not all) of the other extensions +shipped by default have the following Ids: ``webcompat@mozilla.org``, +``e10srollout@mozilla.org``, ``firefox@getpocket.com``, +``flyweb@mozilla.org``, ``loop@mozilla.org``. + +If an extension only has a hexadecimal identifier, a Google search of +that identifier is usually enough to identify the extension's name. + +This information is useful because some crashes are caused by +extensions. Correlations between extensions and crash signatures can be +seen in the "Correlations" tab (see below). + +Correlations tab +---------------- + +This tab is only shown when crash-stats identifies correlations between +a crash and modules or extensions that are present, which happens +occasionally. + +See also +-------- + +- `A talk about understanding crash + reports <https://air.mozilla.org/a-talk-about-understanding-crash-reports/>`__, + by David Baron, from March 2016. +- :ref:`A guide to searching crash reports` + +.. |Example fields in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13579/Details1.png +.. |Information relating to bug reports in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13581/Details2.png +.. |Information relating to threads in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13583/Details3.png +.. |Table of modules in the "Modules" tab of a crash report| image:: https://mdn.mozillademos.org/files/13593/Modules1.png +.. |JSON data in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/13595/RawDump1.png +.. |Links to downloadable files in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/14047/raw-dump-links.png +.. |Table of extensions in the "Extensions" tab of a crash report| image:: https://mdn.mozillademos.org/files/13599/Extensions1.png diff --git a/docs/contributing/directory_structure.rst b/docs/contributing/directory_structure.rst new file mode 100644 index 0000000000..5eaa5cd07b --- /dev/null +++ b/docs/contributing/directory_structure.rst @@ -0,0 +1,575 @@ +Firefox Source Code Directory Structure +======================================= + +This article provides an overview of what the various directories contain. + +To simply take a look at the Firefox source code, you do not need to +download it. You can look at the source directly with your web browser +using Searchfox (start at https://searchfox.org/mozilla-central/source for +the complete firefox source code of branch HEAD). + +In order to modify the source, you have to acquire it either by +downloading a +`snapshot <https://developer.mozilla.org/docs/Mozilla/Developer_guide/Source_Code/Downloading_Source_Archives>`__ +of the sources or by checking out the current sources from :ref:`Mercurial <Firefox Contributors' Quick Reference>`. + +This document describes the directory structure -- i.e., directories that +are used by at least some of the +Mozilla project's client products. There are other directories in the +other Mozilla repository, such as those for Web tools and those for the +Classic codebase. + +See `source code directories +overview <https://developer.mozilla.org/docs/Archive/Misc_top_level/Source_code_directories_overview>`__ for a +somewhat different (older) version of the same information. Also see the +`more detailed overview of the pieces of +Gecko <https://wiki.mozilla.org/Gecko:Overview>`__. + +.cargo +------ + +Configuration files for the `Cargo package +manager <https://crates.io/>`__. + +.vscode +------- + +Configuration files used by the `Visual Studio Code +IDE <https://code.visualstudio.com/>`__ when working in the +mozilla-central tree. + +accessible +---------- + +Files for accessibility (i.e., MSAA (Microsoft Active Accessibility), +ATK (Accessibility Toolkit, used by GTK+ 2) support files). See +`Accessibility <https://developer.mozilla.org/docs/Web/Accessibility>`__. + + +browser +------- + +Contains the front end code (in XUL, Javascript, XBL, and C++) for the +Firefox browser. Many of these files started off as a copy of files in +`xpfe <https://developer.mozilla.org/docs/Mozilla/Developer_guide/Source_Code/Directory_structure#xpfe>`__/. + +browser/extensions +------------------ + +Contains `PDF.js <https://mozilla.github.io/pdf.js/>`__ and +`WebCompat <https://github.com/mozilla/webcompat-addon>`__ built-in extensions. + +browser/themes +-------------- + +Contains images and CSS files to skin the browser for each OS (Linux, +Mac and Windows) + +build +----- + +Miscellaneous files used by the build process. See also +`config <https://developer.mozilla.org/docs/Mozilla/Developer_guide/Source_Code/Directory_structure#config>`__/. + +caps +---- + +Capability-based web page security management. It contains C++ interfaces +and code for determining the capabilities of content based on the +security settings or certificates (e.g., VeriSign). See `Component +Security <https://www.mozilla.org/projects/security/components/>`__ . + +chrome +------ + +Chrome registry (See +`here <https://developer.mozilla.org/en/docs/Chrome_Registration>`__) +used with `toolkit <#toolkit>`__/. These files were originally copies of +files in `rdf/chrome/`. + +config +------ + +More files used by the build process, common includes for the makefiles, +etc. + + +devtools +-------- + +The `Firefox Developer Tools <https://developer.mozilla.org/docs/Tools>`__ server and client components. + + +docs +---- + +Contains the documentation configuration (`Sphinx <http://www.sphinx-doc.org/>`__ based), the index page +and the contribution pages. + + +docshell +-------- + +Implementation of the docshell, the main object managing things related +to a document window. Each frame has its own docshell. It contains +methods for loading URIs, managing URI content listeners, etc. It is the +outermost layer of the embedding API used to embed a Gecko browser into +an application. + +dom +--- + +- `IDL definitions <https://developer.mozilla.org/docs/Mozilla/Tech/XPIDL>`__ of the interfaces defined by + the DOM specifications and Mozilla extensions to those interfaces + (implementations of these interfaces are primarily, but not + completely, in `content <https://developer.mozilla.org/docs/Mozilla/Developer_guide/Source_Code/Directory_structure#content>`__). +- The parts of the connection between JavaScript and the + implementations of DOM objects that are specific both to JavaScript + and to the DOM. +- Implementations of a few of the core "DOM Level 0" objects, such as + `window <https://developer.mozilla.org/docs/Web/API/Window>`__ , `window.navigator <https://developer.mozilla.org/docs/Web/API/Window/navigator>`__, `window.location <https://developer.mozilla.org/docs/Web/API/Window/location>`__, etc. + +editor +------ + +The editor directory contains XUL/Javascript for the embeddable editor +component, which is used for the HTML Editor("Composer"), for plain and +HTML mail composition, and for text fields and text areas throughout the +product. The editor is designed like a +"browser window with editing features": it adds some special classes for +editing text and managing transaction undo/redo, but reuses browser code +for nearly everything else. + +extensions +---------- + +Contains several extensions to mozilla, which can be enabled at +compile-time using the ``--enable-extensions`` configure argument. + +Note that some of these are now built specially and not using the +``--enable-extensions`` option. For example, disabling xmlextras is done +using ``--disable-xmlextras``. + + +extensions/auth +--------------- + +Implementation of the negotiate auth method for HTTP and other +protocols. Has code for SSPI, GSSAPI, etc. See `Integrated +Authentication <https://www.mozilla.org/projects/netlib/integrated-auth.html>`__. + + +extensions/pref +--------------- + +Preference-related extensions. + +extensions/spellcheck +--------------------- + +Spellchecker for mailnews and composer. + +extensions/universalchardet +--------------------------- + +Detects the character encoding of text. + +gfx +--- + +Contains interfaces that abstract the capabilities of platform specific +graphics toolkits, along with implementations on various platforms. +These interfaces provide methods for things like drawing images, text, +and basic shapes. It also contains basic data structures such as points +and rectangles used here and in other parts of Mozilla. + +gradle +------ + +Containing files related to a Java build system. + +hal +--- + +Contains platform specified functions (e.g. obtaining battery status, +sensor information, memory information, Android +alarms/vibrate/notifications/orientation, etc) + +image +----- + +Image rendering library. Contains decoders for the image formats Firefox +supports. + +intl +---- + +Internationalization and localization support. See +`L10n:NewProjects <https://wiki.mozilla.org/L10n:NewProjects>`__. + +intl/locale +----------- + +Code related to determination of locale information from the operating +environment. + +intl/lwbrk +---------- + +Code related to line breaking and word breaking. + +intl/strres +----------- + +Code related to string resources used for localization. + +intl/uconv +---------- + +Code that converts (both ways: encoders and decoders) between UTF-16 and +many other character encodings. + +intl/unicharutil +---------------- + +Code related to implementation of various algorithms for Unicode text, +such as case conversion. + +ipc +--- + +Container for implementations of IPC (Inter-Process Communication). + +js/src +------ + +The JavaScript engine, also known as +`SpiderMonkey <https://developer.mozilla.org/docs/Mozilla/Projects/SpiderMonkey>`__. +See also `JavaScript <https://developer.mozilla.org/docs/JavaScript>`__. + +js/xpconnect +------------ + +Support code for calling JavaScript code from C++ code and C++ code from +JavaScript code, using XPCOM interfaces. See +`XPConnect <https://developer.mozilla.org/docs/XPConnect>`__. + +layout +------ + +Code that implements a tree of rendering objects that describe the types +and locations of the objects that are displayed on the screen (such as +CSS boxes, tables, form controls, XUL boxes, etc.), and code that +manages operations over that rendering tree (such as creating and +destroying it, doing layout, painting, and event handling). See +`documentation <https://www.mozilla.org/newlayout/doc/>`__ and `other +information <https://www.mozilla.org/newlayout/>`__. + +layout/base +----------- + +Code that deals with the rendering tree. + +layout/forms +------------ + +Rendering tree objects for HTML form controls. + +layout/generic +-------------- + +The basic rendering object interface and the rendering tree objects for +basic CSS boxes. + +layout/mathml +------------- + +Rendering tree objects for `MathML <https://developer.mozilla.org/docs/Web/MathML>`__. + +layout/svg +---------- + +Rendering tree objects for `SVG <https://developer.mozilla.org/docs/Web/SVG>`__. + +layout/tables +------------- + +Rendering tree objects for CSS/HTML tables. + +layout/xul +---------- + +Additional rendering object interfaces for `XUL <https://developer.mozilla.org/docs/XUL>`__ and +the rendering tree objects for XUL boxes. + +media +----- + +Contains sources of used media libraries for example *libpng*. + +memory +------ + +Cross-platform wrappers for *memallocs* functions etc. + +mfbt +---- + +Implementations of classes like *WeakPtr*. Multi-platform *assertions* +etc. `More on +MFBT <https://developer.mozilla.org/docs/Mozilla/MFBT>`__ + +mobile +------ + +mobile/android +-------------- + +Firefox for Android and Geckoview + +modules +------- + +Compression/Archiving, math library, font (and font compression), +Preferences Library + +modules/libjar +-------------- + +Code to read zip files, used for reading the .jar files that contain the +files for the mozilla frontend. + +modules/libpref +--------------- + +Library for reading and writing preferences. + +modules/zlib +------------ + +Source code of zlib, used at least in the networking library for +compressed transfers. + +mozglue +------- + +Glue library containing various low-level functionality, including a +dynamic linker for Android, a DLL block list for Windows, etc. + +netwerk +------- + +`Networking library <https://developer.mozilla.org/docs/Necko>`__, also known as Necko. +Responsible for doing actual transfers from and to servers, as well as +for URI handling and related stuff. + +netwerk/cookie +-------------- + +Permissions backend for cookies, images, etc., as well as the user +interface to these permissions and other cookie features. + +nsprpub +------- + +Netscape Portable Runtime. Used as an abstraction layer to things like +threads, file I/O, and socket I/O. See `Netscape Portable +Runtime <https://www.mozilla.org/projects/nspr/>`__. + +nsprpub/lib +----------- + +Mostly unused; might be used on Mac? + +other-licenses +-------------- + +Contains libraries that are not covered by the MPL but are used in some +Firefox code. + +parser +------ + +Group of structures and functions needed to parse files based on +XML/HTML. + +parser/expat +------------ + +Copy of the expat source code, which is the XML parser used by mozilla. + +parser/html +----------- + +The HTML parser (for everything except about:blank). + +parser/htmlparser +----------------- + +The legacy HTML parser that's still used for about:blank. Parts of it +are also used for managing the conversion of the network bytestream into +Unicode in the XML parsing case. + +parser/xml +---------- + +The code for integrating expat (from parser/expat) into Gecko. + +python +------ + +Cross module python code. + +python/mach +----------- + +The code for the `Mach <https://developer.mozilla.org/docs/Mozilla/Developer_guide/mach>`__ building +tool. + +security +-------- + +Contains NSS and PSM, to support cryptographic functions in mozilla +(like S/MIME, SSL, etc). See `Network Security Services +(NSS) <https://www.mozilla.org/projects/security/pki/nss/>`__ and +`Personal Security Manager +(PSM) <https://www.mozilla.org/projects/security/pki/psm/>`__. + +services +-------- + +Firefox accounts and sync (history, preferences, tabs, bookmarks, +telemetry, startup time, which addons are installed, etc). See +`here <https://docs.services.mozilla.com/>`__. + +servo +----- + +`Servo <https://servo.org/>`__, the parallel browser engine project. + +startupcache +------------ + +XXX this needs a description. + +storage +------- + +`Storage <https://developer.mozilla.org/docs/Mozilla/Tech/XPCOM/Storage>`__: XPCOM wrapper for sqlite. Wants to +unify storage of all profile-related data. Supersedes mork. See also +`Unified Storage <https://wiki.mozilla.org/Mozilla2:Unified_Storage>`__. + +taskcluster +----------- + +Scripts and code to automatically build and test Mozilla trees for the +continuous integration and release process. + +testing +------- + +Common testing tools for mozilla codebase projects, test suite +definitions for automated test runs, tests that don't fit anywhere else, +and other fun stuff. + +third_party +----------- + +Vendored dependencies maintained outside of Mozilla. + +toolkit +------- + +The "new toolkit" used by Thunderbird, Firefox, etc. This contains +numerous front-end components shared between applications as well as +most of the XBL-implemented parts of the XUL language (most of which was +originally forked from versions in `xpfe/`). + +toolkit/mozapps/extensions/test/xpinstall +----------------------------------------- + +The installer, which contains code for installing Mozilla and for +installing XPIs/extensions. This directory also contains code needed to +build installer packages. See `XPInstall <https://developer.mozilla.org/docs/XPInstall>`__ and +the `XPInstall project +page <https://www.mozilla.org/projects/xpinstall/>`__. + +tools +----- + +Some tools which are optionally built during the mozilla build process. + +tools/lint +---------- + +The linter declarations and configurations. +See `linting documentation </code-quality/lint/>`_ + +uriloader +--------- + +uriloader/base +-------------- + +Content dispatch in Mozilla. Used to load uris and find an appropriate +content listener for the data. Also manages web progress notifications. +See `Document Loading: From Load Start to Finding a +Handler <https://www.mozilla.org/docs/docshell/uri-load-start.html>`__ +and `The Life Of An HTML HTTP +Request <https://www.mozilla.org/docs/url_load.html>`__. + + +uriloader/exthandler +-------------------- + +Used to handle content that Mozilla can't handle itself. Responsible for +showing the helper app dialog, and generally for finding information +about helper applications. + +uriloader/prefetch +------------------ + +Service to prefetch documents in order to have them cached for faster +loading. + +view +---- + +View manager. Contains cross-platform code used for painting, scrolling, +event handling, z-ordering, and opacity. Soon to become obsolete, +gradually. + +widget +------ + +A cross-platform API, with implementations on each platform, for dealing +with operating system/environment widgets, i.e., code related to +creation and handling of windows, popups, and other native widgets and +to converting the system's messages related to painting and events into +the messages used by other parts of Mozilla (e.g., `view/` and +`content/`, the latter of which converts many of the +messages to yet another API, the DOM event API). + +xpcom +----- + +`Cross-Platform Component Object Model </en-US/docs/XPCOM>`__. Also +contains data structures used by the rest of the mozilla code. See also +`XPCOM Project <https://www.mozilla.org/projects/xpcom/>`__. + +xpfe +---- + +XPFE (Cross Platform Front End) is the SeaMonkey frontend. It contains +the XUL files for the browser interface, common files used by the other +parts of the mozilla suite, and the XBL files for the parts of the XUL +language that are implemented in XBL. Much of this code has been copied +to `browser/` and `toolkit/` for use in +Firefox, Thunderbird, etc. + + +xpfe/components +--------------- + +Components used by the Mozilla frontend, as well as implementations of +interfaces that other parts of mozilla expect. diff --git a/docs/contributing/editor.rst b/docs/contributing/editor.rst new file mode 100644 index 0000000000..7179b6ad41 --- /dev/null +++ b/docs/contributing/editor.rst @@ -0,0 +1,231 @@ +Editor / IDE integration +======================== + +You can use any editor or IDE to contribute to Firefox, as long as it can edit +text files. However, there are some steps specific to mozilla-central that may +be useful for a better development experience. This page attempts to document +them. + +.. note:: + + This page is a work in progress. Please enhance this page with instructions + for your favourite editor. + +Visual Studio Code +------------------ + +.. toctree:: + :hidden: + :maxdepth: 1 + + vscode + + +Go to :doc:`Visual Studio Code <vscode>` dedicated page. + +VIM +--- + +AutoCompletion +~~~~~~~~~~~~~~ + +There's C++ and Rust auto-completion support for VIM via +`YouCompleteMe <https://github.com/ycm-core/YouCompleteMe/>`__. + +As long as that is installed and you have run :code:`./mach build` or +:code:`./mach configure`, it should work out of the box. Configuration for this lives +in :code:`.ycm_extra_conf` at the root of the repo. + +Rust auto-completion should work both with the default completer (RLS, as of +this writing), or with `rust-analyzer <https://rust-analyzer.github.io/manual.html#youcompleteme>`__. + +ESLint +~~~~~~ + +The easiest way to integrate ESLint with VIM is using the `Syntastic plugin +<https://github.com/vim-syntastic/syntastic>`__. + +:code:`mach eslint --setup` installs a specific ESLint version and some ESLint +plugins into the repositories' :code:`node_modules`. + +You need something like this in your :code:`.vimrc` to run the checker +automatically on save: + +.. code:: + + autocmd FileType javascript,html,xhtml let b:syntastic_checkers = ['javascript/eslint'] + +You need to have :code:`eslint` in your :code:`PATH`, which you can get with +:code:`npm install -g eslint`. You need at least version 6.0.0. + +You can also use something like `eslint_d +<https://github.com/mantoni/eslint_d.js#editor-integration>`__ which should +also do that automatically: + +.. code:: + + let g:syntastic_javascript_eslint_exec = 'eslint_d' + +Emacs +----- + +Mozilla-specific packages +~~~~~~~~~~~~~~~~~~~~~~~~~ + +ESLint +~~~~~~ + +See `the devtools documentation <https://wiki.mozilla.org/DevTools/CodingStandards#Running_ESLint_in_Emacs>`__ +that describes how to integrate ESLint into Emacs. + +C/C++ Development Packages +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +General Guidelines to Emacs C++ Programming +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following guides give an overview of the C++ editing capabilities of emacs. + +It is worth reading through these guides to see what features are available. +The rest of this section is dedicated to Mozilla/Gecko specific setup for +packages. + + + * `C/C++ Development Environment for Emacs <https://tuhdo.github.io/c-ide.html>`__ + * `Emacs as C++ IDE <https://syamajala.github.io/c-ide.html>`__ + +rtags (LLVM/Clang-based Code Indexing) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Instructions for the installation of rtags are available at the +`rtags github repo <https://github.com/Andersbakken/rtags>`__. + +rtags requires a :ref:`compilation database <CompileDB back-end / compileflags>`. + +In order for rtags to index correctly, included files need to be copied and +unified compilation files need to be created. Either run a full build of the +tree, or if you only want indexes to be generated for the moment, run the +following commands (assuming you're in the gecko repo root): + +.. code:: + cd gecko_build_directory + make export + ./config.status + +To increase indexing speed, it's best to remove unified build files and test +files from database updates. This can be done by creating a :code:`~/.rdmrc` +file with the following contents, with :code:`[src_dir]` replaced with either +the repo or build directory for your checkout: + +.. code:: + + -X */[src_dir]/*Unified*;*/[src_dir]/*/test/*;*/[src_dir]/*/tests/* + +Once the rdm daemon is running, the compilation database can be added to rtags +like so: + +.. code:: + + rc -J [gecko_build_directory]/compile_commands.json + +Note that this process will take a while initially. However, once the database +is built, it will only require incremental updates. As long as the rdm daemon +is running in the background, the database will be updated based on changes to +files. + +irony (LLVM/Clang-based Code Completion) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Instructions on the installation of irony-mode are available at the +`irony-mode github repo <https://github.com/Sarcasm/irony-mode>`__. + +irony-mode requires a :ref:`compilation database <CompileDB back-end / compileflags>`. + +Note that irony-mode, by default, uses elisp to parse the +:code:`compile_commands.json` file. As gecko is a very large codebase, this +file can easily be multiple megabytes, which can make irony-mode take multiple +seconds to load on a gecko file. + +It is recommended to use `this fork of irony-mode <https://github.com/Hylen/irony-mode/tree/compilation-database-guessing-4-pull-request>`__, +which requires the boost System and Filesystem libraries. + +`Checking the bug to get this patch into the mainline of irony-mode <https://github.com/Sarcasm/irony-mode/issues/176>`__ +is recommended, to see if the fork can be used or if the mainline repo can be +used. Using the Boost version of the irony-mode server brings file load times +to under 1s. + +Projectile (Project Management) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Instructions on the installation of projectile are available at the +`projectile github repo <https://github.com/bbatsov/projectile>`__. + +Projectile comes preconfigured for many project types. Since, gecko uses its +own special build system (mach), a new project type needs to be added. This can +be done via adding the following elisp configuration command to your emacs +configuration file. + +.. code:: + + (projectile-register-project-type 'gecko + '("mach" "moz.build") + "python mach --log-no-times build" + "python mach mochitest" + "python mach run") + +Assuming projectile-global-mode is on, this will allow projectile to run the +correct commands whenever it is working in a gecko repo. + +gdb +^^^ + +Emacs comes with great integration with gdb, especially when using +`gdb-many-windows <https://www.gnu.org/software/emacs/manual/html_node/emacs/GDB-User-Interface-Layout.html>`__. + +However, when gdb is invoked via mach, some special arguments +need to be passed in order to make sure the correct display mode is used. To +use M-x gdb with mach on firefox, use the following command: + +.. code:: + + gecko_repo_directory/mach run --debug --debugparams=-i=mi + +Eclipse +------- + +You can generate an Eclipse project by running: + +.. code:: + + ./mach ide eclipse + +See also the `Eclipse CDT <https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Eclipse/Eclipse_CDT>`__ docs on MDN. + +Visual Studio +------------- + +You can run a Visual Studio project by running: + +.. code:: + + ./mach ide visualstudio + +CompileDB back-end / compileflags +--------------------------------- + +You can generate a :code:`compile_commands.json` in your object directory by +running: + +.. code:: + + ./mach build-backend --backend=CompileDB + +This file, the compilation database, is understood by a variety of C++ editors / IDEs +to provide auto-completion capabilities. You can also get an individual compile command by +running: + +.. code:: + + ./mach compileflags path/to/file + +This is how the :ref:`VIM <VIM>` integration works, for example. diff --git a/docs/contributing/how_to_submit_a_patch.rst b/docs/contributing/how_to_submit_a_patch.rst new file mode 100644 index 0000000000..54cf3f14d3 --- /dev/null +++ b/docs/contributing/how_to_submit_a_patch.rst @@ -0,0 +1,276 @@ +How to submit a patch +===================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +Submitting a patch, getting it reviewed, and committed to the Firefox +source tree involves several steps. This article explains how. + +.. note:: + + We are also providing a :ref:`Firefox Contributors Quick Reference <Firefox Contributors' Quick Reference>` for contributors. + +The process of submission is illustrated by the following diagram, and +each step is detailed below: + +.. mermaid:: + + graph TD; + Preparation --> c[Working on a patch]; + c[Working on a patch] --> Testing; + Testing --> c[Working on a patch]; + Testing --> e[Submit the patch]; + e[Submit the patch] --> d[Getting Reviews] + d[Getting Reviews] -- Addressing Review comment --> c[Working on a patch]; + d[Getting Reviews] --> h[Push the change]; + + + + +Preparation +----------- + +Every change to the code is tracked by a bug report +in `bugzilla.mozilla.org <https://bugzilla.mozilla.org/>`__. Without a +bug, code will not be reviewed, and without review, code will not be +accepted. To avoid duplication, `search for an existing +bug <https://bugzilla.mozilla.org/query.cgi?format=specific>`__ about +your change, and only if none exists, file a new one. Most communication +about code changes take place in the associated code +review, so be sure the bug describes the exact problem being solved. + +Please verify the bug is for the correct product and component. For more +information, ask questions on the newsgroups, or on the #developers room +on `chat.mozilla.org <https://chat.mozilla.org>`__. + +The person working on a bug should be the 'assignee' of that bug in +Bugzilla. If somebody else is currently the assignee of a bug, email +this person to coordinate changes. If the bug is unassigned, leave a +message in the bug's comments, stating that you intend working on it, +and suggest that someone with bug-editing privileges assign it to you. + +Some teams wait for new contributors to attach their first patch before +assigning a bug. This makes it available for other contributors, in case +the new contributor is unable to level up to patch creation. By +expressing interest in a bug comment, someone from that team should +guide you through their process. + + +Module ownership +---------------- + +All code is supervised by a `module +owner <https://www.mozilla.org/en-US/about/governance/policies/module-ownership/>`__. +This person will be responsible for reviewing and accepting the change. +Before writing your code, determine the module owner, verifying your +proposed change is considered acceptable. They may want to look over any +new user interface (UI review), functions (API review), or testcases for +the proposed change. + +If module ownership is not clear, ask on the newsgroups or `on +Matrix <https://chat.mozilla.org>`__. The revision log for the relevant +file might also be helpful. For example, see the change log for {{ +Source("browser/base/content/browser.js") }}, by clicking the "Hg Log" +link at the top of `Searchfox <https://searchfox.org/mozilla-central/source/>`__, or +by running ``hg log browser/base/content/browser.js``. The corresponding +checkin message will contain something like "r=nickname", identifying +active code submissions, and potential code reviewers. + + +Working on a patch +------------------ + +Changes to the Firefox source code are presented in the form of a patch. +A patch is a commit to version control. Firefox and related code is +stored in our `Mercurial +server <https://hg.mozilla.org/mozilla-central>`__. We have extensive +documentation on using Mercurial in our guide, :ref:`Mercurial Overview`. + +Each patch should represent a single complete change, separating +distinct changes into multiple individual patches. If your change +results in a large, complex patch, seek if it can be broken into +`smaller, easy to understand patches representing complete +steps <https://secure.phabricator.com/book/phabflavor/article/writing_reviewable_code/#many-small-commits>`__, +applied on top of each other. This makes it easier to review your +changes, `leading to quicker +reviews, <https://groups.google.com/group/mozilla.dev.planning/msg/2f99460f57f776ef?hl=en>`__ +and improved confidence in this review outcome. + +Also ensure that your commit message is formatted appropriately. A +simple commit message should look like this: + +:: + + Bug 123456 - Change this thing to work better by doing something. r=reviewers + +The ``r=reviewers`` part is optional; if you are using Phabricator, +Lando will add it automatically based on who actually granted review, +and in any case the person who does the final check-in of the patch will +make sure it's added. + +The text of the message should be what you did to fix the bug, not a +description of what the bug was. If it is not obvious why this change is +appropriate, then `explain why in the commit +message <https://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview/commits.html#write-detailed-commit-messages>`__. +If this does not fit on one line, then leave a blank line and add +further lines for more detail and/or reasoning. + +You can edit the message of the current commit at any time using +``hg commit --amend`` or ``hg histedit``. + +Also look at our :ref:`Reviewer Checklist` for a list +of best practices for patch content that reviewers will check for or +require. + + +Testing +------- + +All changes must be tested. In most cases, an `automated +test <https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing>`__ is required for every +change to the code. + +While we desire to have automated tests for all code, we also have a +linter tool which runs static analysis on our JavaScript, for best +practices and common mistakes. See :ref:`ESLint` for more information. + +Ensure that your change has not caused regressions, by running the +automated test suite locally, or using the `Mozilla try +server <https://wiki.mozilla.org/Build:TryServer>`__. Module owners, or +developers `on Matrix <https://chat.mozilla.org>`__ may be willing to +submit jobs for those currently without try server privileges. + + +Submit the patch +---------------- + +.. note:: + + Make sure you rebase your patch on top of the latest build before you + submit to prevent any merge conflicts. + +Mozilla uses Phabricator for code review. See the `Mozilla Phabricator +User +Guide <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`__ +for instructions. + +Don't be shy in posting partial patches, demonstrating potential +approaches, and asking for preliminary feedback. It is easier for others +to comment, and offer suggestions, when a question is accompanied by +some code. + + +Getting reviews +--------------- + +Thorough code reviews are one of Mozilla's ways of ensuring code +quality. Every patch must be reviewed by the module owner of the code, +or one of their designated peers. + +For more information about the review process, see the :ref:`Code Review FAQ`. + +To request a review, you will need to specify one or more usernames +either when you submit the patch, or afterward in the UI. See the +`Mozilla Phabricator User +Guide <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`__ +for details. + +*Getting attention:* If a reviewer doesn't respond within a week, or so, +of the review request: + +- Join #developers on Mozilla's `Matrix + server <https://chat.mozilla.org>`__, and ask if anyone knows why a + review may be delayed. Please link to the bug too. +- If the review is still not addressed, mail the reviewer directly, + asking if/when they'll have time to review the patch, or might + otherwise be able to review it. +- As a last resort, ask in the appropriate newsgroup on + ``news.mozilla.org``. + + +Addressing review comments +-------------------------- + +It is unusual for patches to be perfect the first time around. The +reviewer may use the ‘Request Changes’ +`action <http://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#reviewing-patches>`__ +and list problems that must be addressed before the patch can be +accepted. Please remember that requesting revisions is not meant to +discourage participation, but rather to encourage the best possible +resolution of a bug. Carefully work through the changes that the +reviewer recommends, attach a new patch, and request review again. + +Sometimes a reviewer will grant conditional review with the ‘Accept +Revision’ action but will also indicate minor necessary changes, such as +spelling, or indentation fixes. All recommended corrections should be +made, but a re-review is unnecessary. Make the changes and submit a new +patch. If there is any confusion about the revisions, another review +should be requested. + +Sometimes, after a patch is reviewed, but before it can be committed, +someone else makes a conflicting change. If the merge is simple, and +non-invasive, post an updated version of the patch. For all non-trivial +changes, another review is necessary. + +If at any point the review process stalls for more than two weeks, see +the previous 'Getting attention' section. + +In many open source projects, developers will accept patches in an +unfinished state, finish them, and apply the completed code. In +Mozilla's culture, **the reviewer will only review and comment on a +patch**. If a submitter declines to make the revisions, the patch will +sit idle, until someone chooses to take it on. + + +Pushing the change +------------------ + +A patch can be pushed (aka. 'landed') after it has been properly +reviewed. + +.. note:: + + Note: Be sure to build the application with the patch applied. This + ensures it runs as expected, passing automated tests, and/or runs + through the `try + server <https://wiki.mozilla.org/Build:TryServerAsBranch>`__. In the + bug, please also mention you have completed this step. + + Submitting untested patches wastes the committer's time, and may burn + the release tree. Please save everyone's time and effort by + completing all necessary verifications. + + +Add the tag "check-in needed" on the revision(s) in phabricator. To do +so, click on the "Edit" button on a phabricator revision, and start +typing "check-in needed" in the Tags field. It should auto-complete. If +Phabricator doesn't allow you to add the keyword, ask someone else to +add it. Community members with commit access, regularly search for +revisions with the checkin-needed keyword, often committing in a day or +so. If the patch does not get checked, within a reasonable timeframe, +join #developers on `chat.mozilla.org <https://chat.mozilla.org>`__, asking +someone to check on your behalf. In most circumstances, a link to a +passing Try run will be required, in order to verify the patch will not +cause any new failures after landing. + +If pushing the patch yourself, please follow :ref:`Committing rules and responsibilities`. +`Lando <https://moz-conduit.readthedocs.io/en/latest/lando-user.html>`__ is used +to automatically land your code. + + +Regressions +----------- + +It is possible your code causes functional or performance regressions. +There is a tight +`policy <https://www.mozilla.org/about/governance/policies/regressions/>`__ on +performance regressions, in particular. This means your code may be +dropped, leaving you to fix and resubmit it. Regressions, ultimately +mean the tests you ran before checking in are not comprehensive enough. +A resubmitted patch, or a patch to fix the regression, should be +accompanied by appropriate tests. + +After authoring a few patches, consider `getting commit access to +Mozilla source code <https://www.mozilla.org/about/governance/policies/commit/>`__. diff --git a/docs/contributing/img/auto_completion.gif b/docs/contributing/img/auto_completion.gif Binary files differnew file mode 100644 index 0000000000..4d545c29fd --- /dev/null +++ b/docs/contributing/img/auto_completion.gif diff --git a/docs/contributing/img/diagnostic_error.gif b/docs/contributing/img/diagnostic_error.gif Binary files differnew file mode 100644 index 0000000000..25d50a7d7c --- /dev/null +++ b/docs/contributing/img/diagnostic_error.gif diff --git a/docs/contributing/img/find_references.gif b/docs/contributing/img/find_references.gif Binary files differnew file mode 100644 index 0000000000..e986894566 --- /dev/null +++ b/docs/contributing/img/find_references.gif diff --git a/docs/contributing/img/format_selection.gif b/docs/contributing/img/format_selection.gif Binary files differnew file mode 100644 index 0000000000..28c88b3fa3 --- /dev/null +++ b/docs/contributing/img/format_selection.gif diff --git a/docs/contributing/img/goto_definition.gif b/docs/contributing/img/goto_definition.gif Binary files differnew file mode 100644 index 0000000000..9b3dcbf90f --- /dev/null +++ b/docs/contributing/img/goto_definition.gif diff --git a/docs/contributing/img/rename_symbol.gif b/docs/contributing/img/rename_symbol.gif Binary files differnew file mode 100644 index 0000000000..d41fcd7ed6 --- /dev/null +++ b/docs/contributing/img/rename_symbol.gif diff --git a/docs/contributing/img/type_hierarchy.gif b/docs/contributing/img/type_hierarchy.gif Binary files differnew file mode 100644 index 0000000000..139bdf88cb --- /dev/null +++ b/docs/contributing/img/type_hierarchy.gif diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst new file mode 100644 index 0000000000..ea4860ace3 --- /dev/null +++ b/docs/contributing/index.rst @@ -0,0 +1,39 @@ +Working on Firefox +================== + +Welcome to the Firefox codebase. This is the home of the Firefox +development process and source code documentation. + +.. toctree:: + :caption: Making Changes To Firefox + :maxdepth: 1 + + contribution_quickref + pocket-guide-shipping-firefox + editor + reviews + +.. toctree:: + :caption: The Mercurial Version Control System + :maxdepth: 1 + :glob: + + vcs/* + + +.. toctree:: + :caption: Debugging + :maxdepth: 1 + :glob: + + debugging/* + + +.. toctree:: + :caption: Additional Information + :maxdepth: 1 + + directory_structure + build/artifact_builds + build/building_mobile_firefox + build/supported_configurations diff --git a/docs/contributing/pocket-guide-shipping-firefox.rst b/docs/contributing/pocket-guide-shipping-firefox.rst new file mode 100644 index 0000000000..4ff84927e3 --- /dev/null +++ b/docs/contributing/pocket-guide-shipping-firefox.rst @@ -0,0 +1,434 @@ +Pocket Guide: Shipping Firefox +============================== + +*Estimated read time:* 15min + + +Introduction +------------ + +The purpose of this document is to provide a high level understanding of +how Mozilla ships Firefox. With the intention of helping new Mozillians +(and those who would like a refresher) understand the basics of our +release process, tools, common terms, and mechanisms employed in +shipping Firefox to our users. Often this document will introduce a +concept, explain how it fits into the process, and then provide a link +to learn more if interested. + +.. note:: + + This does not contain an overview of how we + ship :ref:`Fenix <fenix>` (Our next gen Android browser) as + that product is largely uncoupled from how we ship to desktop and the + process we've historically followed. + +Repositories & Channels +----------------------- + +Shipping Firefox follows a software release :ref:`train model <train model>` along 3 primary code +:ref:`repositories <repositories>`; mozilla-central (aka “m-c”), +mozilla-beta, and mozilla-release. Each of these repositories are +updated within a defined cadence and built into one of our Firefox +products which are released through what is commonly referred to as +:ref:`Channels <channels>`: Firefox Nightly, Firefox Beta, and Firefox +Release. + +**Firefox Nightly** offers access to the latest cutting edge features +still under active development. Released every 12 hours with all the +changes that have :ref:`landed <landing>` on mozilla-central. + +Every `4 weeks <https://wiki.mozilla.org/RapidRelease/Calendar>`__, we +:ref:`merge <merge>` the code from mozilla-central to our +mozilla-beta branch. New code or Features can be added to mozilla-beta +outside of this 4 week cadence but will be required to land in +mozilla-central and then be :ref:`uplift <uplift>`\ed into +mozilla-beta. + +**Firefox Beta** is for developers and early adopters who want to see +and test what’s coming next in Firefox. We release a new Beta version +three times a week. + +.. note:: + + The first and second beta builds of a new cycle are shipped to a + subset of our Beta population. The full Beta population gets updated + starting with Beta 3 only.* + +Each Beta cycle lasts a total of 4 weeks where a final build is +validated by our QA and tagged for release into the mozilla-release +branch. + +.. note:: + + **Firefox Developer Edition** *is a separate product based on + the mozilla-beta repo and is specifically tailored for Web Developers.* + +**Firefox Release** is released every 4 weeks and is the final product +of our Beta cycle. As our primary product shipping to hundreds of +millions of users, interim updates and +:ref:`ride-alongs <ride alongs>` are only shipped to Release if +they contain :ref:`dot release drivers <dot release drivers>`. + +.. note:: + **Firefox ESR (Extended Support Release)** *is a separate + product intended for Enterprise use. Major updates are rolled out 1-2 + times per year to maintain stability and predictability. ESR also + contains a number of policy options not available in the standard + Firefox Release. Minor updates are shipped in sync with the Firefox + Release schedule for security and stability fixes only.* + +Further Reading/Useful links: + +- `Firefox Release + Process <https://wiki.mozilla.org/Release_Management/Release_Process>`__ +- `Release + Calendar <https://wiki.mozilla.org/Release_Management/Calendar>`__ +- `Firefox Delivery + dashboard <https://mozilla.github.io/delivery-dashboard/>`__ + +Landing Code and Shipping Features +---------------------------------- + +Mozillians (those employed by MoCo and the broader community) land lots +of code in the Mozilla repositories: fixes, enhancements, compatibility, +new features, etc. and is managed by :ref:`Mercurial <Mercurial Overview>` (aka +hg). All new code is tracked in :ref:`Bugzilla <bugzilla>`, reviewed +in :ref:`Phabricator <Phabricator>`, and then checked into the +mozilla-central repository using :ref:`Lando <Lando>`. + +.. note:: + + Some teams use `GitHub <github>` during development + but will still be required to use Phabricator (tracked in Bugzilla) to + check their code into the mozilla-central hg repository. + +The standard process for code to be delivered to our users is by ‘riding +the trains’, meaning that it’s landed in mozilla-central where it waits +for the next Beta cycle to begin. After merging to Beta the code will +stabilize over a 4 week period (along with everything else that merged +from mozilla-central). At the end of the beta cycle a release candidate +(:ref:`RC <rc>`) build will be generated, tested thoroughly, and +eventually become the next version of Firefox. + +Further Reading/Useful links: + +- `Phabricator and why we use it <https://wiki.mozilla.org/Phabricator>`__ +- `Firefox Trello <https://trello.com/b/8k1hT2vh/firefox>`__ (Distilled + list of critical features riding the trains) + +An exception to this process... +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Not all code can simply wait for the normal train model to be included +in a Firefox build. There are a variety of reasons for this; critical +fixes, security concerns, stabilizing a feature that’s already in Beta, +shipping high priority features faster, and so on. + +In these situations an uplift can be requested to take a recent landing +in mozilla-central and merge specific bits to another repository outside +the standard train model. After the request is made within Bugzilla, +`Release Management <release management>` will assess the potential risk +and will make a decision on whether it’s accepted. + +Further Reading/Useful links: + +- `Patch uplifting + rules <https://wiki.mozilla.org/Release_Management/Uplift_rules>`__ + +Ensuring build stability +~~~~~~~~~~~~~~~~~~~~~~~~ + +Throughout the process of landing code in mozilla-central to riding the +trains to Firefox Release, there are many milestones and quality +checkpoints from a variety of teams. This process is designed to ensure +a quality and compelling product will be consistently delivered to our +users with each new version. See below for a distilled list of those +milestones. + +========================================= ========== =============== =============================================================================== +Milestone Week Day of Week +----------------------------------------- ---------- --------------- ------------------------------------------------------------------------------- +Merge Day Nightly W1 Monday Day 1 of the new Nightly Cycle +`PI Request <pi request>` deadline Nightly W1 Friday Manual QA request deadline for high risk features +Feature technical documentation due Nightly W2 Friday Deadline for features requiring manual QA +Beta release notes draft Nightly W4 Wednesday +Nightly features Go/No-Go decisions Nightly W4 Wednesday +Feature Complete Milestone Nightly W4 Wednesday Last day to land risky patches and/or enable new features +Nightly soft code freeze start Nightly W4 Thursday Stabilization period in preparation to merge to Beta +String freeze Nightly W4 Thursday Modification or deletion of strings exposed to the end-users is not allowed +QA pre-merge regression testing completed Nightly W4 Friday +Merge Day Beta W1 Monday Day 1 of the new Beta cycle +Pre-release sign off Beta W3 Friday Final round of QA testing prior to Release +Firefox RC week Beta W4 Monday Validating Release Candidate builds in preparation for the next Firefox Release +Release Notes ready Beta W4 Tuesday +What’s new page ready Beta W4 Wednesday +Firefox go-live @ 6am PT Release W1 Tuesday Day 1 of the new Firefox Release to 25% of Release users +Firefox Release bump to 100% Release W1 Thursday Increase deployment of new Firefox Release to 100% of Release users +========================================= ========== =============== =============================================================================== + + +The Release Management team (aka “Relman”) monitors and enforces this +process to protect the stability of Firefox. Each member of Relman +rotates through end-to-end ownership of a given :ref:`release +cycle <release cycle>`. The Relman owner of a cycle will focus on the +overall release, blocker bugs, risks, backout rates, stability/crash +reports, etc. Go here for a complete overview of the `Relman Release +Process +Checklist <https://wiki.mozilla.org/Release_Management/Release_Process_Checklist_Documentation>`__. + +.. note:: + + While Relman will continually monitor the overall health of each + Release it is the responsibility of the engineering organization to + ensure the code they are landing is of high quality and the potential + risks are understood. Every Release has an assigned :ref:`Regression + Engineering Owner <reo>` (REO) to ensure a decision is made + about each regression reported in the release.* + +Further Reading/Useful links: + +- `Release Tracking + Rules <https://wiki.mozilla.org/Release_Management/Tracking_rules>`__ +- `Release + Owners <https://wiki.mozilla.org/Release_Management/Release_owners>`__ +- `Regression Engineering + Owners <https://wiki.mozilla.org/Platform#Regression_Engineering_Owner_.28REO.29>`__ +- `Commonly used Bugzilla queries for all + Channels <https://pascalc.net/rm_queries/>`__ + +Enabling/Disabling code (Prefs) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Within Firefox we allow the ability to Enable/Disable bits of code or +entire features using `Preferences <preferences>`. There are many +reasons why this is useful. Here are some examples: + +- Continual development over multiple release cycles without exposing + partially completed features to our users +- Provide the ability to quickly disable a feature if there is a + problem found during the release process +- Control features which are experimental or not ready to be shown to a + specific channel population (e.g. enabled for Beta but disabled for + Release) +- A/B testing via :ref:`telemetry <telemetry>` experiments + +.. note:: + + :ref:`Normandy <normandy>` Pref Rollout is a feature that + allows Mozilla to change the state of a preference for a targeted set of + users, without deploying an update to Firefox. This is especially useful + when conducting experiments or a gradual rollout of high risk features + to our Release population. + +Further Reading/Useful links: + +- `Brief guide to Mozilla + preferences <https://developer.mozilla.org/en-US/docs/Mozilla/Preferences/A_brief_guide_to_Mozilla_preferences>`__ +- `Normandy Pref + rollout <https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout>`__ + +Release & Feature QA +~~~~~~~~~~~~~~~~~~~~ + +Release QA is performed regularly and throughout the Release Cycle. +Organized in two-week sprints its primary goals are: + +- Qualifying builds for release +- Feature testing +- Product Integrity requests +- Bug work +- Community engagement + +Features that can have significant impact and/or pose risk to the code +base should be nominated for QA support by the :ref:`feature +owner <feature owner>` in its intended release. This process is kicked +off by filing a :ref:`Product Integrity <product integrity>` team request +:ref:`PI request <pi request>`. These are due by the end of week 2 +of the Nightly cycle. + +.. note:: + + Manual QA testing is only required for features as they go + through the Beta cycle. Nightly Feature testing is always optional. + +Further Reading/Useful links: + +- `QA Feature + Testing <https://wiki.mozilla.org/QA/Feature_Testing_v2>`__ +- `Release QA + overview <https://docs.google.com/document/d/1ic_3TO9-kNmZr11h1ZpyQbSlgiXzVewr3kSAP5ML4mQ/edit#heading=h.pvvuwlkkvtc4>`__ +- `PI Request template and + overview <https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=PI&title=PI+Request>`__ + +Experiments +~~~~~~~~~~~ + +As we deliver new features to our users we continually ask ourselves +about the potential impacts, both positive and negative. In many new +features we will run an experiment to gather data around these impacts. +A simple definition of an experiment is a way to measure how a change to +our product affects how people use it. + +An experiment has three parts: + +1. A new feature that can be selectively enabled +2. A group of users to test the new feature +3. Telemetry to measure how people interact with the new feature + +Experiments are managed by an in-house tool called +`Experimenter <https://experimenter.services.mozilla.com/>`__. + +Further Reading/Useful links: + +- `More about experiments and + Experimenter <https://github.com/mozilla/experimenter>`__ +- `Requesting a new + Experiment <https://experimenter.services.mozilla.com/experiments/new/>`__ + (Follow the ‘help’ links to learn more) +- `Telemetry <https://wiki.mozilla.org/Telemetry>`__ + +Definitions +----------- + +.. _bugzilla: + +**Bugzilla** - Web-based general purpose bug tracking system and testing +tool + +.. _channel: + +**Channel** - Development channels producing concurrent releases of +Firefox for Windows, Mac, Linux, and Android + +.. _dot release drivers: + +**Dot Release Drivers** - Issues/Fixes that are significant enough to +warrant a minor dot release to the Firefox Release Channel. Usually to +fix a stability (top-crash) or Security (Chemspill) issue. + +.. _feature owner: + +**Feature Owner** - The person who is ultimately responsible for +developing a high quality feature. This is typically an Engineering +Manager or Product Manager. + +.. _fenix: + +**Fenix** - Also known as Firefox Preview is an all-new browser for +Android based on GeckoView and Android Components + +.. _github: + +**Github** - Web-based version control and collaboration platform for +software developers + +.. _landing: + +**Landing** - A general term used for when code is merged into a +particular source code repository + +.. _lando: + +**Lando** - Automated code lander for Mozilla. It is integrated with +our `Phabricator instance <https://phabricator.services.mozilla.com>`__ +and can be used to land revisions to various repositories. + +.. _mercurial: + +**Mercurial** - A source-code management tool which allows users to keep +track of changes to the source code locally and share their changes with +others + +.. _merge: + +**Merge** - General term used to describe the process of integrating and +reconciling file changes within the mozilla repositories + +.. _normandy: + +**Normandy** - Normandy is a collection of servers, workflows, and +Firefox components that enables Mozilla to remotely control Firefox +clients in the wild based on precise criteria + +.. _phabricator: + +**Phabricator** - Mozilla’s instance of the web-based software +development collaboration tool suite. Read more about `Phabricator as a +product <https://phacility.com/phabricator/>`__. + +.. _pi request: + +**PI Request** - Short for Product Integrity Request is a form +submission request that’s used to engage the PI team for a variety of +services. Most commonly used to request Feature QA it can also be used +for Security, Fuzzing, Performance, and many other services. + +.. _preferences: + +**Preferences** - A preference is any value or defined behavior that can +be set (e.g. enabled or disabled). Preference changes via user interface +usually take effect immediately. The values are saved to the user’s +Firefox profile on disk (in prefs.js). + +.. _product integrity: + +**Product Integrity** - The Product Integrity team is responsible for +ensuring product quality and release consistency by testing features, +validating builds, and managing the overall release process. In +addition, PI provides various engineering support functions such as +sheriffing, bug triage and investigation. + +.. _rc: + +**Release Candidate** - Beta version with potential to be a final +product, which is ready to release unless significant bugs emerge. + +.. _release cycle: + +**Release Cycle** - The sum of stages of development and maturity for +the Firefox Release Product. + +.. _reo: + +**Regression Engineering Owner** - A partner for release management +assigned to each release. They both keep a mental state of how we are +doing and ensure a decision is made about each regression reported in +the release + +.. _release management: + +**Release Management** - Team primarily responsible for the process of +managing, planning, scheduling and controlling a software build through +different stages and environments + +.. _Repository: + +**Repository** - a collection of stored data from existing databases +merged into one so that it may be shared, analyzed or updated throughout +an organization + +.. _ride alongs: + +**Ride Alongs** - Bug fixes that are impacting release users but not +considered severe enough to ship without an identified dot release +driver. + +.. _telemetry: + +**Telemetry** - Firefox measures and collects non-personal information, +such as performance, hardware, usage and customizations. This +information is used by Mozilla to improve Firefox. + +.. _train model: + +**Train model** - a form of software release schedule in which a number +of distinct series of versioned software releases are released as a +number of different "trains" on a regular schedule. + +.. _uplift: + +**Uplift** - the action of taking parts from a newer version of a +software system (mozilla-central or mozilla-beta) and porting them to an +older version of the same software (mozilla-beta or mozilla-release) + + diff --git a/docs/contributing/reviewer_checklist.rst b/docs/contributing/reviewer_checklist.rst new file mode 100644 index 0000000000..cfe772dba9 --- /dev/null +++ b/docs/contributing/reviewer_checklist.rst @@ -0,0 +1,181 @@ +Reviewer Checklist +================== + + Submitting patches to Mozilla source code needn't be complex. This + article provides a list of best practices for your patch content that + reviewers will check for or require. Following these best practices + will lead to a smoother, more rapid process of review and acceptance. + + +Good web citizenship +-------------------- + +- Make sure new web-exposed APIs actually make sense and are either + standards track or preffed off by default. +- In C++, wrapper-cache as needed. If your object can be gotten from + somewhere without creating it in the process, it needs to be + wrapper-cached. + + +Correctness +----------- + +- The bug being fixed is a valid bug and should be fixed. +- The patch fixes the issue. +- The patch is not unnecessarily complicated. +- The patch does not add duplicates of existing code ('almost + duplicates' could mean a refactor is needed). Commonly this results + in "part 0" of a bug, which is "tidy things up to make the fix easier + to write and review". +- If QA needs to verify the fix, you should provide steps to reproduce + (STR). + + +Quality +------- + +- If you can unit-test it, you should unit-test it. +- If it's JS, try to design and build so that xpcshell can exercise + most functionality. It's quicker. +- Make sure the patch doesn't create any unused code (e.g., remove + strings when removing a feature) +- All caught exceptions should be logged at the appropriate level, + bearing in mind personally identifiable information, but also + considering the expense of computing and recording log output. + [Fennec: Checking for log levels is expensive unless you're using + Logger.] + + +Style +----- + +- Follow the `style + guide <https://firefox-source-docs.mozilla.org/code-quality/coding-style/index.html>`__ + for the language and module in question. +- Follow local style for the surrounding code, even if that local style + isn't formally documented. +- New files have license declarations and modelines. +- New JS files should use strict mode. +- Trailing whitespace (git diff and splinter view both highlight this, + as does hg with the color extension enabled). Whitespace can be fixed + easily in Mercurial using the `CheckFiles + extension <https://www.mercurial-scm.org/wiki/CheckFilesExtension>`__. + In git, you can use git rebase --whitespace=fix. + + +Security issues +--------------- + +- There should be no writing to arbitrary files outside the profile + folder. +- Be careful when reading user input, network input, or files on disk. + Assume that inputs will be too big, too short, empty, malformed, or + malicious. +- Tag for sec review if unsure. +- If you're writing code that uses JSAPI, chances are you got it wrong. + Try hard to avoid doing that. + + +Privacy issues +-------------- + +- There should be no logging of URLs or content from which URLs may be + inferred. +- [Fennec: Android Services has Logger.pii() for this purpose (e.g., + logging profile dir)]. +- Tag for privacy review if needed. + + +Resource leaks +-------------- + +- In Java, memory leaks are largely due to singletons holding on to + caches and collections, or observers sticking around, or runnables + sitting in a queue. +- In C++, cycle-collect as needed. If JavaScript can see your object, + it probably needs to be cycle-collected. +- [Fennec: If your custom view does animations, it's better to clean up + runnables in onDetachFromWindow().] +- Ensure all file handles and other closeable resources are closed + appropriately. +- [Fennec: When writing tests that use PaintedSurface, ensure the + PaintedSurface is closed when you're done with it.] + + +Performance impact +------------------ + +- Check for main-thread IO [Fennec: Android may warn about this with + strictmode]. +- Remove debug logging that is not needed in production. + + +Threading issues +---------------- + +- Enormous: correct use of locking and volatility; livelock and + deadlock; ownership. +- [Fennec: All view methods should be touched only on UI thread.] +- [Fennec: Activity lifecycle awareness (works with "never keep + activities"). Also test with oom-fennec + (`https://hg.mozilla.org/users/blassey_mozilla.com/oom-fennec/) <https://hg.mozilla.org/users/blassey_mozilla.com/oom-fennec/%29>`__]. + + +Compatibility +------------- + +- Version files, databases, messages +- Tag messages with ids to disambiguate callers. +- IDL UUIDs are updated when the interface is updated. +- Android permissions should be 'grouped' into a common release to + avoid breaking auto-updates. +- Android APIs added since Froyo should be guarded by a version check. + + +Preffability +------------ + +- If the feature being worked on is covered by prefs, make sure they + are hooked up. +- If working on a new feature, consider adding prefs to control the + behavior. +- Consider adding prefs to disable the feature entirely in case bugs + are found later in the release cycle. +- [Fennec: "Prefs" can be Gecko prefs, SharedPreferences values, or + build-time flags. Which one you choose depends on how the feature is + implemented: a pure Java service can't easily check Gecko prefs, for + example.] + + +Strings +------- + +- There should be no string changes in patches that will be uplifted + (including string removals). +- Rev entity names for string changes. +- When making UI changes, be aware of the fact that strings will be + different lengths in different locales. + + +Documentation +------------- + +- The commit message should describe what the patch is changing (not be + a copy of the bug summary). The first line should be a short + description (since only the first line is shown in the log), and + additional description, if needed, should be present, properly + wrapped, in later lines. +- Adequately document any potentially confusing pieces of code. +- Flag a bug with dev-doc-needed if any addon or web APIs are affected. +- Use Javadocs extensively, especially on any new non-private methods. +- When moving files, ensure blame/annotate is preserved. + + +Accessibility +------------- + +- For HTML pages, images should have the alt attribute set when + appropriate. Similarly, a button that is not a native HTML button + should have role="button" and the aria-label attribute set. +- [Fennec: Make sure contentDescription is set for parts of the UI that + should be accessible] diff --git a/docs/contributing/reviews.rst b/docs/contributing/reviews.rst new file mode 100644 index 0000000000..d5991f0d14 --- /dev/null +++ b/docs/contributing/reviews.rst @@ -0,0 +1,104 @@ +Getting reviews +=============== + + +Thorough code reviews are one of Mozilla's ways of ensuring code quality. +Every patch must be reviewed by the module owner of the code, or one of their designated peers. + +To request a review, you will need to specify a review group (starts with #). If there is not, you should select one or more usernames either when you submit the patch, or afterward in the UI. +If you have a mentor, the mentor can usually either also review or find a suitable reviewer on your behalf. + +Getting attention: If a reviewer doesn't respond within a week, or so of the review request: + + * Contact the reviewer directly (either via e-mail or on Matrix). + * Join developers on `Mozilla's Matrix server <https://chat.mozilla.org>`_, and ask if anyone knows why a review may be delayed. Please link to the bug too. + * If the review is still not addressed, mail the reviewer directly, asking if/when they'll have time to review the patch, or might otherwise be able to review it. + +Review groups +------------- + + +.. list-table:: + :header-rows: 1 + + * - Name + - Owns + - Members + * - #build or #firefox-build-system-reviewers + - The configure & build system + - `Member list <https://phabricator.services.mozilla.com/project/members/20/>`__ + * - #dom-workers-and-storage-reviewers + - DOM Workers & Storage + - `Member list <https://phabricator.services.mozilla.com/project/members/115/>`__ + * - #devtools-inspector-reviewers + - The devtools inspector tool + - `Member list <https://phabricator.services.mozilla.com/project/members/109/>`__ + * - #fluent-reviewers + - Changes to Fluent (FTL) files (translation). + - `Member list <https://phabricator.services.mozilla.com/project/members/105/>`__ + * - #firefox-source-docs-reviewers + - Documentation files and its build + - `Member list <https://phabricator.services.mozilla.com/project/members/118/>`__ + * - #firefox-ux-team + - User experience (UX) + - `Member list <https://phabricator.services.mozilla.com/project/members/91/>`__ + * - #firefox-svg-reviewers + - SVG-related changes + - `Member list <https://phabricator.services.mozilla.com/project/members/97/>`__ + * - #geckoview-reviewers + - Changes to GeckoView + - `Member list <https://phabricator.services.mozilla.com/project/members/92/>`__ + * - #gfx-reviewers + - Changes to Graphics code + - `Member list <https://phabricator.services.mozilla.com/project/members/122/>`__ + * - #intermittent-reviewers + - Test manifest changes + - `Member list <https://phabricator.services.mozilla.com/project/members/110/>`__ + * - #layout-reviewers + - Layout changes. + - `Member list <https://phabricator.services.mozilla.com/project/members/126/>`__ + * - #linter-reviewers + - tools/lint/* + - `Member list <https://phabricator.services.mozilla.com/project/members/119/>`__ + * - #marionette-reviewers + - Changes to Marionette + - `Member list <https://phabricator.services.mozilla.com/project/members/117/>`__ + * - #mozbase + - Changes to Mozbase + - `Member list <https://phabricator.services.mozilla.com/project/members/113/>`__ + * - #mozbase-rust + - Changes to Mozbase in Rust + - `Member list <https://phabricator.services.mozilla.com/project/members/114/>`__ + * - #perftest-reviewers + - Perf Tests + - `Member list <https://phabricator.services.mozilla.com/project/members/102/>`__ + * - #preferences-reviewers + - Firefox for Desktop Preferences (Options) user interface + - `Member list <https://phabricator.services.mozilla.com/project/members/132/>`__ + * - #remote-protocol-reviewers + - Remote protocol + - `Member list <https://phabricator.services.mozilla.com/project/members/101/>`__ + * - #remote-debugging-reviewers + - Remote Debugging UI & tools + - `Member list <https://phabricator.services.mozilla.com/project/members/108/>`__ + * - #static-analysis-reviewers + - Changes related to Static Analysis + - `Member list <https://phabricator.services.mozilla.com/project/members/120/>`__ + * - #style or #firefox-style-system-reviewers + - Firefox style system (servo, layout/style). + - `Member list <https://phabricator.services.mozilla.com/project/members/90/>`__ + * - #webcompat-reviewers + - System addons maintained by the Web Compatibility team + - `Member list <https://phabricator.services.mozilla.com/project/members/124/>`__ + * - #webdriver-reviewers + - Marionette and Geckodriver in Firefox + - `Member list <https://phabricator.services.mozilla.com/project/members/103/>`__ + * - #webidl + - Changes related to WebIDL + - `Member list <https://phabricator.services.mozilla.com/project/members/112/>`__ + * - #xpcom-reviewers + - Changes related to XPCOM + - `Member list <https://phabricator.services.mozilla.com/project/members/125/>`__ + +To create a new group, fill a `new bug in Conduit::Administration <https://bugzilla.mozilla.org/enter_bug.cgi?product=Conduit&component=Administration>`__. +See `bug 1613306 <https://bugzilla.mozilla.org/show_bug.cgi?id=1613306>`__ as example. diff --git a/docs/contributing/stack_quickref.rst b/docs/contributing/stack_quickref.rst new file mode 100644 index 0000000000..bb85859576 --- /dev/null +++ b/docs/contributing/stack_quickref.rst @@ -0,0 +1,103 @@ +Working with stack of patches Quick Reference +============================================= + +Working on Firefox, we strongly recommend working with stack of patches. +Patches should be small and could be landed in the order used to push them. +This also helps to breakdown the work for different reviewers. + +As it can be complex for new comers, this documentation explains the +various commands. + +For the overall quick reference guide, see the :ref:`Firefox Contributors Quick Reference <Firefox Contributors' Quick Reference>` + +Visualize the stack +------------------- + +.. code-block:: shell + + # Mercurial + $ hg wip + + # Git + $ git log + +Reorder the stack +----------------- + +Sometimes, we want to change the order the patches in the stack. +Fortunately, VCS support this easily. + +.. code-block:: shell + + # Mercurial + # Just change the order the patch. The tool should highlight + # potential risks of conflicts. + # Note that ctrl+c works well if used + $ hg histedit + + # Git + # In the editor, just move the line below/above + # Remove everything if you want to cancel the operation + $ git rebase -i + + +Make a change on a patch at the beginning of the stack +------------------------------------------------------ + +In some cases, the reviewer is asking for a change at the bottom of the stack (ie not at the top). +So, a simple `hg/git commit --amend` would not work. + +In such case, the following approach can be used: + +.. code-block:: shell + + # Mercurial + # hg will try to guess in which an unambiguous prior commit + $ hg absorb + + # if this doesn't work, create a temporary commit + # and merge it using "fold" or "roll" + $ hg histedit + + # Git + $ git commit --fixup <hash of the commit> + + +Removing patches in the stack +----------------------------- + +To remove a patch in the stack: + +.. code-block:: shell + + # Mercurial + # select "drop" (letter "d") + $ hg histedit + + # Git + # Replace "pick" by "drop" + # Or simply remove the line for this commit + $ git rebase -i + + +Rebasing the stack +------------------ + +As the codebase moves fast, it can be necessary to pull changes from +mozilla-central before landing the changes. + +.. code-block:: shell + + # Mercurial + # First, see where your patches are in the stack + $ hg wip + # Then, rebase it: + # If you are a beginner, don't hesitate to add "--dry-run" + $ hg pull + $ hg rebase -b . -d central + + + # Git + $ git remote update + $ git rebase mozilla/central + diff --git a/docs/contributing/vcs/mercurial.rst b/docs/contributing/vcs/mercurial.rst new file mode 100644 index 0000000000..3969c41e9f --- /dev/null +++ b/docs/contributing/vcs/mercurial.rst @@ -0,0 +1,208 @@ +Mercurial Overview +================== + +Mercurial is a source-code management tool which allows users to keep track of changes to the source code locally and share their changes with others. +We use it for the development of Firefox. + +Installation +------------ + +See `Mercurial Page <https://www.mercurial-scm.org/downloads>`__ for installation. + + +Using `hg clone` +---------------- + +If you are not worried about network interruptions, then you can simply +use Mercurial to directly clone the repository you're interested in +using its URL, as given below. For example, to use the command line to +clone ``mozilla-central`` into a directory called ``firefox-source``, +you would use the following: + +.. code-block:: shell + + hg clone https://hg.mozilla.org/mozilla-central/ firefox-source + cd firefox-source + +Using Mercurial bundles +----------------------- + +If you are worried that your Internet connection is not fast or robust +enough to download such a large amount of data all in one go without +being interrupted and cannot clone using the command given above, then you are recommended to try :ref:`Mercurial bundles <Mercurial bundles>`. If interrupted, they can be resumed (continued without downloading +from the beginning) if the app you're using to download supports it. For +example, in Firefox you would right click on the download and select +`Resume` once your connection to the Internet was reestablished. + +Basic configuration +------------------- + +You should configure Mercurial before submitting patches to Mozilla. + +If you will be pulling the Firefox source code or one of the derived repositories, the easiest way to configure Mercurial is to run the vcs-setup mach command: + +.. code-block:: shell + + $ ./mach vcs-setup + +This command starts an interactive wizard that will help ensure your Mercurial is configured with the latest recommended settings. This command will not change any files on your machine without your consent. + + +Other configuration tips +------------------------ + +If you don't have the Firefox source code available, you should edit your Mercurial configuration file to look like the following: + +.. code-block:: shell + + [ui] + username = Your Real Name <user@example.com> + merge = your-merge-program (or internal:merge) + + [diff] + git = 1 + showfunc = 1 + unified = 8 + + [defaults] + commit = -v + +On Windows, these settings can be added to `$HOME\.hgrc` or `$HOME\Mercurial.ini`, or, if you'd like global settings, `C:\mozilla-build\hg\Mercurial.ini` +or `C:\Program Files\Mercurial\Mercurial.ini.` On UNIX-like systems, they should be in your `$HOME/.hgrc` file. + +You can configure the editor to use for commit messages using the `editor` option in the `[ui]` section or by setting the `EDITOR` environment variable. + +If you are trying to access the repository through a proxy server, see `these +instructions <http://www.selenic.com/mercurial/hgrc.5.html#http-proxy>`__ + + +Selecting a repository (tree) +----------------------------- + +There are multiple hg repositories hosted at mozilla.org to choose from. +A summary of the main trees is given below, but see +https://hg.mozilla.org/ for the full list. + +mozilla-central +--------------- + +This is the main development tree for Firefox. Most developers write +patches against mozilla-central. + +URL: https://hg.mozilla.org/mozilla-central/ + + +mozilla-beta +------------ + +The source for the current beta version of Firefox (and the next and all +previous betas). This code represents the expected next release of the +Firefox browser, and should be pretty stable. + +URL: https://hg.mozilla.org/releases/mozilla-beta/ + +mozilla-release +--------------- + +The source for the current release of Firefox (and the next and all +previous releases). + +URL: https://hg.mozilla.org/releases/mozilla-release/ + +autoland +-------- + +This is the integration tree for Firefox. Patches land in this repository first, +and then are merged by the sheriffs in mozilla-central. + +URL: https://hg.mozilla.org/integration/autoland/ + +L10n repos +---------- + +Mainly useful for localizers working on localizing Firefox. Code for all +l10n projects lives here and is organized into separate repos that (in +most cases) have the locale's two character ISO code. To get the repo +that you need look for the repo you're interested in on the following +page. + +URL: https://hg.mozilla.org/l10n-central/ + +Unified Repositories +-------------------- + +It is common for advanced users to want to interact with more than one +firefox repository. If you get to the point where having individual +copies of repositories is annoying you, then see +https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html +for instructions on doing this efficiently. + +Selecting a revision to build +----------------------------- + +Most of the time the `tip` revision of most repositories will build +without issue. If you are worried about it not, then you may want to +`get the latest revision that has passed the automatic +tests <https://developer.mozilla.org/docs/Mozilla/Developer_guide/Source_Code/LatestPassingSource>`__. + +Building +-------- + +By default with no configuration a similar-to-release build is done. If +you wish you can :ref:`configure <Configuring Build Options>` the build using a ``.mozconfig`` file +and ``mach build``. +Different OSs have different prerequisites for a successful build, +please refer to the :ref:`build documentation <Getting Set Up To Work On The Firefox Codebase>` +to verify they are available on your build machine. + +Extensions +---------- + +There's a number of extensions you can enable. See http://mercurial.selenic.com/wiki/UsingExtensions. Almost everyone should probably enable the following, most of them are enabled by ``mach boostrap``: + +#. color - Colorize terminal output +#. histedit - Provides git rebase --interactive behavior. +#. progress - Draw progress bars on long-running operations. +#. rebase - Ability to easily rebase patches on top of other heads. +#. evolve - Enable and enhance the inprogress ChangesetEvolution work. +#. firefoxtree - Enhances the interaction with Firefox repositories. +#. transplant - Easily move patches between repositories, branches, etc. + +These can all be turned on by just adding this to your `.hgrc` file: + +.. code-block:: shell + + [extensions] + color = + rebase = + histedit = + progress = + firefoxtree = + evolve = + transplant = + +In addition, there are some 3rd party extensions that are incredibly +useful for basic development: + +`mozext <https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/mozext>`__ + Mozilla-specific functionality to aid in developing Firefox/Gecko. + +`trychooser <https://github.com/pbiggar/trychooser>`__ + Automatically creates a try commit message and then pushes changes to + Mozilla's Try infrastructure. Just run: + +.. code-block:: shell + + hg trychooser + +Configuring the try repository +------------------------------ + +About `Try Server <Try Server>`__. + +Learning to use Mercurial +------------------------- + +If you are new to Mercurial, you should start with the `official guide <https://www.mercurial-scm.org/guide>`__. + +Then, move on to the `version control tool docs <https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/>`__ for Mozilla-centric Mercurial information. diff --git a/docs/contributing/vcs/mercurial_bundles.rst b/docs/contributing/vcs/mercurial_bundles.rst new file mode 100644 index 0000000000..bf953b1f56 --- /dev/null +++ b/docs/contributing/vcs/mercurial_bundles.rst @@ -0,0 +1,61 @@ +Mercurial Bundles +================= + +If you have a poor network connection that is preventing ``hg clone`` from completing, you may want to try downloading a bundle of the repository you're interested in. This is useful since a file download, unlike ``hg clone``, can be resumed if the connection is interrupted. Once you have the bundle, staying up-to-date shouldn't take much time at all, if you keep up with it regularly. + + +Clone the sources +----------------- + +Up-to-date bundles of some of the repositories listed at https://hg.mozilla.org/ are available on a CDN at https://hg.cdn.mozilla.net/. + +If you have Mercurial 4.1 (released March 2017) or later, download the zstd bundle for the repository you're interested in. The zstd bundles are faster to download (smaller) and faster to decompress. + +Setting up the repository +------------------------- + +Once you have downloaded the repository bundle, follow the steps below to recreate the repository locally based upon that bundle. Be sure to replace "``mozilla-central``" with the project you're working with as appropriate. + +1. Initialize a new repository (in a directory called ``mozilla-central`` here): + +.. code-block:: shell + + mkdir mozilla-central + hg init mozilla-central + +2. Un-bundle the bundle file to that repository: + +To use the below command in Windows, export the ``\path\to\hg`` and invoke the command from command prompt. + +On Linux/Mac click on the properties of the file and you can find the path. In case the name of the file is not bundle.hg rename it. + +.. code-block:: shell + + cd mozilla-central + hg unbundle /path/to/your/bundle.hg + +Get comfortable. Grab a coffee (or your favorite tasty beverage). Maybe a nap. This unbundling process is going to take quite a lot of time. + +3. Add the following lines to the repository's config file (``.hg/hgrc``) so that Mercurial will automatically know where to pull changes from future updates. You can open the template config file in your editor by running ``hg config --edit`` or ``EDITOR=<editor-of-your-choice> hg config --edit`` + +.. code-block:: shell + + [paths] + default = https://hg.mozilla.org/mozilla-central/ + +4. Update the repository to get all the changes since the bundle was created (this step also doubles as a check of the bundle integrity since if its contents are not exactly the same as what's in the official repository then the ``hg pull`` will fail): + +.. code-block:: shell + + hg pull + +5. Check out a working copy from your new up to date repository: + +.. code-block:: shell + + hg update + +You now have a clone of ``mozilla-central`` that is identical to one made via ``hg clone``. You can adjust your build settings, or you can go straight ahead and build Firefox! + +If at any point you are stuck, feel free to ask on Riot/Matrix at `https://chat.mozilla.org <https://chat.mozilla.org>`__ +in `#introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`__ channel. diff --git a/docs/contributing/vscode.rst b/docs/contributing/vscode.rst new file mode 100644 index 0000000000..a361dc91bc --- /dev/null +++ b/docs/contributing/vscode.rst @@ -0,0 +1,108 @@ +Visual Studio Code +================== + +General Knowledge +~~~~~~~~~~~~~~~~~ + +`VSCode <https://code.visualstudio.com/>`__ is a multi-platform open-source programming editor developed by Microsoft and volunteers. +It has support for many programming languages using extensions. + +For more general information on the VSCode project see `repo <https://github.com/Microsoft/vscode/>`__. + +C/C++ Features and Support +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For C++ support we offer an out of the box configuration based on +`clangd <https://clangd.llvm.org>`__. + +Leveraging the `clang` toolchain compiler we now have support in the IDE for the following features: + +**1.** Syntax highlighting + +**2.** IntelliSense with comprehensive code completion and suggestion + +.. image:: img/auto_completion.gif + +**3.** Go-to definition and Go-to declaration + +.. image:: img/goto_definition.gif + +**4.** Find all references + +.. image:: img/find_references.gif + +**5.** Open type hierarchy + +.. image:: img/type_hierarchy.gif + +**6.** Rename symbol, all usages of the symbol will be renamed, including declaration, definition and references + +.. image:: img/rename_symbol.gif + +**7.** Code formatting, based on `clang-format` that respects our coding standard using the `.clang-format` and `.clang-format-ignore` files. Format can be performed on an entire file or on a code selection + +.. image:: img/format_selection.gif + +**8.** Inline parsing errors with limited auto-fix hints + +.. image:: img/diagnostic_error.gif + +**9.** Basic static-code analysis using `clang-tidy` and our list of enabled checkers. (This is still in progress not all checkers are supported by `clangd`) + +Clangd-specific Commands +~~~~~~~~~~~~~~~~~~~~~~~~ + +Clangd supports some commands that are specific to C/C++: + +.. code:: + + "clangd.switchheadersource" + +This command navigates from the currently open header file to its corresponding source file (if there is one), or vice versa. + +This command can be invoked from the command menu (activated via ``F1``), or using its keybinding of ``Alt+o`` (``Alt+cmd+o`` on Mac). The keybinding can also be customized in ``Keyboard Shortcuts``. + +Generating Configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to build the configuration for `VS Code` simply run from +the terminal: + +.. code:: + + ./mach ide vscode + +If `VS Code` is already open with a previous configuration generated, please make sure to +restart `VS Code` otherwise the new configuration will not be used, and the `compile_commands.json` +needed by `clangd` server will not be refreshed. This is a known `bug <https://github.com/clangd/vscode-clangd/issues/42>`__ +in `clangd-vscode` extension + +Useful preferences +~~~~~~~~~~~~~~~~~~ + +When setting the preference + +.. code:: + + "editor.formatOnSave": true + +you might find that this isn't working on large source code files, but triggering formatting manually works. This is due to the default timeout for formatOnSave, which is quite short (750ms). You might want to increase this timeout, e.g. + +.. code:: + + "editor.formatOnSaveTimeout": 5000 + + +Recommended extensions +~~~~~~~~~~~~~~~~~~~~~~ + +VS Code provides number of extensions for JavaScript, Rust, etc. +By default, Firefox source tree comes with its own set of recommendations of Visual Studio Code extensions. They are listed in `.vscode/extensions.json <https://searchfox.org/mozilla-central/source/.vscode/extensions.json>`__. + +For Rust development, the `rust-analyzer <https://marketplace.visualstudio.com/items?itemName=matklad.rust-analyzer>`__ extension is recommended. +`See the manual <https://rust-analyzer.github.io/manual.html>`__ for more information. + +Filing Bugs +~~~~~~~~~~~ + +Bugs should be filed in the `Firefox Build System` product under `Developer Environment Integration`, preferably blocking `Bug 1662709 <https://bugzilla.mozilla.org/show_bug.cgi?id=1662709>`__. diff --git a/docs/crash-reporting/img/default-search-results.png b/docs/crash-reporting/img/default-search-results.png Binary files differnew file mode 100644 index 0000000000..394f997554 --- /dev/null +++ b/docs/crash-reporting/img/default-search-results.png diff --git a/docs/crash-reporting/img/default-search-results2.png b/docs/crash-reporting/img/default-search-results2.png Binary files differnew file mode 100644 index 0000000000..03fb33d8c3 --- /dev/null +++ b/docs/crash-reporting/img/default-search-results2.png diff --git a/docs/crash-reporting/img/facet-search-results.png b/docs/crash-reporting/img/facet-search-results.png Binary files differnew file mode 100644 index 0000000000..a53db96b65 --- /dev/null +++ b/docs/crash-reporting/img/facet-search-results.png diff --git a/docs/crash-reporting/img/facet-search-results2.png b/docs/crash-reporting/img/facet-search-results2.png Binary files differnew file mode 100644 index 0000000000..7166302974 --- /dev/null +++ b/docs/crash-reporting/img/facet-search-results2.png diff --git a/docs/crash-reporting/img/facet-search-results3.png b/docs/crash-reporting/img/facet-search-results3.png Binary files differnew file mode 100644 index 0000000000..bc96d30ee9 --- /dev/null +++ b/docs/crash-reporting/img/facet-search-results3.png diff --git a/docs/crash-reporting/img/narrower-search-results.png b/docs/crash-reporting/img/narrower-search-results.png Binary files differnew file mode 100644 index 0000000000..38b410b362 --- /dev/null +++ b/docs/crash-reporting/img/narrower-search-results.png diff --git a/docs/crash-reporting/img/super-search-form.png b/docs/crash-reporting/img/super-search-form.png Binary files differnew file mode 100644 index 0000000000..63b35a23ad --- /dev/null +++ b/docs/crash-reporting/img/super-search-form.png diff --git a/docs/crash-reporting/img/super-search-form2.png b/docs/crash-reporting/img/super-search-form2.png Binary files differnew file mode 100644 index 0000000000..02dd3a541e --- /dev/null +++ b/docs/crash-reporting/img/super-search-form2.png diff --git a/docs/crash-reporting/img/super-search-form3.png b/docs/crash-reporting/img/super-search-form3.png Binary files differnew file mode 100644 index 0000000000..473706e548 --- /dev/null +++ b/docs/crash-reporting/img/super-search-form3.png diff --git a/docs/crash-reporting/index.rst b/docs/crash-reporting/index.rst new file mode 100644 index 0000000000..c1273c6927 --- /dev/null +++ b/docs/crash-reporting/index.rst @@ -0,0 +1,52 @@ +Crash reporting +=============== + +Firefox ships with an open-source crash reporting system. This system is +combination of projects: + +- `Google + Breakpad <https://chromium.googlesource.com/breakpad/breakpad>`__ + client and server libraries +- Mozilla-specific crash reporting user interface and bootstrap code +- `Socorro <https://github.com/mozilla-services/socorro>`__ Collection + and reporting server + + +Where did my crash get submitted? +--------------------------------- + +Crash data submitted using the Mozilla Crash Reporter is located on +`crash-stats <https://crash-stats.mozilla.org/>`__. If you want to find +a specific crash that you submitted, you first need to find the Crash ID +that the server has assigned your crash. Type ``about:crashes`` into +your location bar to get a page listing both submitted and unsubmitted +crash reports. For more information, see :ref:`How to get a stacktrace for a bug report`. + + +Reports and queries +------------------- + +crash-stats has built-in reports of "topcrashes" for each release +grouped by signature. There is also a `custom query tool <https://crash-stats.mozilla.org/search/>`__ +which allows users to limit searches on more precise information. + +Finally, a set of Mozilla employees have access to directly query the +underlying data in either SQL summary or using mapreduce on the storage +cluster. If you are interested in obtaining this advanced access, read +`Crash Stats Documentation: Protected Data Access <https://crash-stats.mozilla.org/documentation/protected_data_access/>`__ + + +See also +-------- + +- :ref:`Understanding crash reports` +- :ref:`A guide to searching crash reports` +- `crash-stats <https://crash-stats.mozilla.org/>`__ +- `Crash pings (Telemetry) and crash reports (Socorro/Crash + Stats) <https://bluesock.org/~willkg/blog/mozilla/crash_pings_crash_reports.html>`__ +- :ref:`Building with Debug Symbols` +- :ref:`Environment variables affecting crash reporting <Crash Reporter#Environment variables affecting crash reporting>` +- :ref:`Uploading symbols to Mozilla's symbol server` +- :ref:`Crash reporter` +- :ref:`Crash manager` +- :ref:`Crash ping` diff --git a/docs/crash-reporting/searching_crash_reports.rst b/docs/crash-reporting/searching_crash_reports.rst new file mode 100644 index 0000000000..0668a03654 --- /dev/null +++ b/docs/crash-reporting/searching_crash_reports.rst @@ -0,0 +1,257 @@ +A guide to searching crash reports +================================== + +.. note:: + + Please read the :ref:`documentation about individual crash + reports <Understanding crash reports>` before reading + this page. + +The Mozilla `crash-stats <https://crash-stats.mozilla.org/>`__ site +provides facilities for investigating large numbers of Firefox `crash +reports <Understanding crash reports>`__. This guide to +searching through crash reports may help you locate the crash reports +that will help you find and fix the Firefox bug you're working on. + +Specifically, crash-stats offers two basic functions: + +Searching + You can search the crash reports database by over 100 criteria: crash + signature, date, platform, product, version, etc. +Grouping + You can cluster the results of each search into groups using the same + criteria. + +To achieve full power and flexibility requires a good understanding of +both of these functions. Search is easy to understand, but the grouping +capabilities are easy to overlook. + +Searching +--------- + +The search form +~~~~~~~~~~~~~~~ + +You can get to the `search +page <https://crash-stats.mozilla.org/search/?product=Firefox&_dont_run=1>`__ +by clicking on the "Super Search" link near the toolbar at the top right +of any page in crash-stats. This brings up a search form like the one in +the following screenshot. + +|Search in crash-stats| + +Fields are provided for four common search criteria: product, version, +platform, and process type. The product field is pre-populated with +"Firefox" because that is a common case. As the fine print says, the +default date range is the past week. + +The default search: Signature facet +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you click on the "Search" button, you will get +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature>`__ +like the ones in the following screenshot. + +|Results of a default search in crash-stats| + +By default, the "Signature facet" tab is selected. ("Facet" is a term +that means "group".) In these results, the found crash reports are +grouped according to crash signature and ranked by group size. The +columns show each group's rank, signature, size (both a count and a +proportion of matching crash reports), and finally a list of bugs that +have been marked as relating to this signature. + +The numbers are large because this search matched all Firefox crash +reports from the past seven days. The first group has over 100,000 crash +reports, which accounts for 7.77% of all matching crashes. This +indicates there are over 1.3 million crash reports matching this search. + +You can reorder the groups in various ways by clicking on the column +headers. The links within the results do the following things. + +- The first link in each "Signature" column cell links to a signature + report, which contains additional details about crash reports with + that signature. +- The "Add term" link in each "Signature" column cell lets you perform + a narrower subsequent search among crash reports with that signature. +- The links in each "Bugs" column cell link to bug reports in Bugzilla. + +The default search: Crash reports +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you switch to the "Crash Reports" tab you will see +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports>`__ +like the ones in the following screenshot. + +|Results of a default search in crash-stats (crash reports tab)| + +This is a list of all the individual crash reports that match the search +criteria. If the number of matches is large -- in this case it exceeds +1.3 million, just as we saw in the "Signature facet" tab -- the results +will be spread across multiple pages, which you can visit by clicking +the links at the top right of the tab. + +The links within the results do the following things. + +- The link in each "Crash ID" column cell links to an individual crash + report. +- The links in each "Signature" column cell have the same effect that + they did in the "Signature facet" tab. +- The links in the remaining column cells also let you perform a + narrower subsequent search with that link's value added to the search + criteria. + +A narrower search +~~~~~~~~~~~~~~~~~ + +You can add criteria to perform a narrower search. For example, to +perform a search for all Mac crash reports that occurred while +JavaScript garbage collection was running, do the following. + +- Add "Mac OS X" to the "Platform" field. +- Select "New line", and then choose a field ("is garbage collecting") + and an operator ("is true"). The operators available for each field + depends on its type. + +With these criteria added the search form looks like the following +screenshot. + +|crash-stats Super Search form with additional criteria| + +After clicking on "Search" we get +`results <https://crash-stats.mozilla.org/search/?is_garbage_collecting=__true__&product=Firefox&platform=Mac%20OS%20X&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform>`__ +like those in the following screenshot. + +|Results of a narrower search in crash-stats| + +The number of crash reports matching this search is in the thousands, +i.e. much smaller than the previous search. + +Proto signature +~~~~~~~~~~~~~~~ + +The "proto signature" field is just the raw unprocessed crash stack +concatenated together. + +You can do things like: + +- Search for crashes where the signature is Foo, and the proto + signature contains Bar. This is helpful if you have a fairly generic + signature and you want to see how many of them are a particular case + of it that you've come across. Or instead of a signature Foo, a moz + crash reason or something else. +- Use it as a facet. This lets you skim the full signatures of crashes + at a glance, bucketed together a bit. Note that because the proto + signature includes the entire signature, things aren't grouped all + that well. + +Grouping +-------- + +In the previous section we saw one example of grouping, in the +"Signature facet" tab that is shown by default. But there are many other +interesting ways to group searches. + +Facets in the search form +~~~~~~~~~~~~~~~~~~~~~~~~~ + +To do a search with non-signature grouping first click on the "More +options..." text, which reveals the additional fields shown in the +following screenshot. + +|crash-stats Super Search form with different facets| + +(The "Show columns" and "Sort by" fields are straightforward. They apply +to the "Crash reports" tab of any search results, and are not related to +grouping.) + +The "Facet on" field is the one that controls grouping. By default, it +contains the value "signature", which explains why we saw a "Signature +facet" tab in the earlier search results. But we can change the values +in this field and get different facet tabs in the search results. + +Grouping by platform +~~~~~~~~~~~~~~~~~~~~ + +For example, if we start with a default search for all Firefox crashes +in the past week, but then replace the "signature" facet with "platform" +and "moz crash reason", we get search results with two facet tabs. The +first of these is a "Platform facet" tab, with +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=platform&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-platform>`__ +like those shown in the following screenshot. + +|Results of a faceted search in crash-stats| + +This has the same columns as the "Signature facet" tab we saw earlier, +except for the "Bugs" column, because that is a special column that only +applies to the signature facet. This tab shows the distribution of crash +reports across the various platforms. Crash reports always include a +platform field (though it may be empty if something has gone wrong) and +so the percentages add up to 100. + +Grouping by "moz crash reason" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The second facet tab is a "Moz crash reason facet" tab, with +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=platform&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-moz_crash_reason>`__ +like those shown in the following screenshot. + +|Results of a faceted search in crash-stats (moz crash reason tab)| + +This immediately shows which ``MOZ_CRASH`` calls are being hit +frequently by users. Only a subset of crash reports have the "moz crash +reason" field -- those that crashed due to hitting a ``MOZ_CRASH`` call +-- so all crashes that lack that field are omitted from this tab. For +that reason, the percentages do not add up to 100. + +An example of less useful grouping +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The usefulness of grouping varies from field to field. In particular, +fields that can have many possible values (such as numeric fields) often +don't group well. For example, if we do a default search grouped by +uptime we get +`results <https://crash-stats.mozilla.org/search/?product=Firefox&_sort=-date&_facets=uptime&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-uptime>`__ +like those in the following screenshot. + +|Results of a faceted search in crash-stats (uptime)| + +In this example the top 10 groups account for less than 12% of all +crashes, and there is an extremely long tail. These results would be +improved by using numeric ranges instead of individual values, but +unfortunately that isn't supported. + +Advanced Usage +-------------- + +The combination of searching and grouping is powerful. Searches find +crash reports that match particular criteria, and grouping organizes +those crash reports into interesting groups. + +When a search is performed, the page's URL is updated to include the +search parameters. This means that the results of any search can be +easily shared by copying and pasting the page's URL. + +To become an expert at searching and grouping requires understanding the +full range of the 100+ fields available for searching and grouping. One +way to learn about them is to read lots of individual crash reports; +note that all fields shown in the Details tab of an individual crash +report have a tool-tip that indicates its key for search. Alternatively, +you can browse the `complete +list <https://crash-stats.mozilla.org/documentation/supersearch/api/#section-filters>`__. + +There is also an API through which searches can be performed +programmatically. See the `API +documentation <https://crash-stats.mozilla.org/documentation/supersearch/>`__ +for full details; note that it uses the term "aggregation" for +grouping/faceting. + +.. |Search in crash-stats| image:: img/super-search-form.png +.. |Results of a default search in crash-stats| image:: img/default-search-results.png +.. |Results of a default search in crash-stats (crash reports tab)| image:: img/default-search-results2.png +.. |crash-stats Super Search form with additional criteria| image:: img/super-search-form2.png +.. |Results of a narrower search in crash-stats| image:: img/narrower-search-results.png +.. |crash-stats Super Search form with different facets| image:: img/super-search-form3.png +.. |Results of a faceted search in crash-stats| image:: img/facet-search-results.png +.. |Results of a faceted search in crash-stats (moz crash reason tab)| image:: img/facet-search-results2.png +.. |Results of a faceted search in crash-stats (uptime)| image:: img/facet-search-results3.png diff --git a/docs/crash-reporting/uploading_symbol.rst b/docs/crash-reporting/uploading_symbol.rst new file mode 100644 index 0000000000..1a7624ba07 --- /dev/null +++ b/docs/crash-reporting/uploading_symbol.rst @@ -0,0 +1,49 @@ +Uploading symbols to Mozilla's symbol server +============================================ + +As a third-party releasing your own builds of Firefox or B2G, you should +consider uploading debug symbols from the builds to Mozilla's symbol +server. If you have not disabled crash reporting in your builds, crash +reports will be submitted to `Mozilla's crash reporting +server <https://crash-stats.mozilla.org/>`__. Without the debug symbols +that match your build the crash reports will not contain actionable +information. + +Symbols can be uploaded either via a web browser or a web API. + + +Building a Symbol Package +------------------------- + +To upload symbols, you need to build a symbol package. This is a +.zip file which contains the symbol files in a specific directory structure. + +If you are building Firefox,or a similar application using the Mozilla +build system, you can build the symbol package using a make target: + +:: + + ./mach buildsymbols + +This will create a symbol package in ``dist/`` named something like +``firefox-77.0a1.en-US.linux-x86_64.crashreporter-symbols.zip`` . + +This step requires the ``dump_syms`` tool which should have been automatically +installed when you setup the Firefox build with ``./mach bootstrap``. If for +some reason it's missing or outdated running the bootstrap step again will +retrieve and install an up-to-date version of the tool. + +Uploading symbols +----------------- + +Symbols are uploaded via your account on symbols.mozilla.org. Visit +https://symbols.mozilla.org and log in. Then request upload +permission by filing a bug in the Socorro component using `this +template <https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=--&bug_status=NEW&bug_type=task&cc=gsvelto%40mozilla.com&cc=willkg%40mozilla.com&cf_fx_iteration=---&cf_fx_points=---&comment=What%20e-mail%20account%20are%20you%20requesting%20access%20for%3F%0D%0A...%0D%0A%0D%0AWhat%20symbols%20will%20you%20be%20uploading%20using%20this%20account%3F%0D%0A...%0D%0A%0D%0AIs%20there%20somebody%20at%20Mozilla%20who%20can%20vouch%20for%20you%3F%0D%0A...%0D%0A&component=Upload&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&filed_via=standard_form&flag_type-4=X&flag_type-607=X&flag_type-800=X&flag_type-803=X&flag_type-936=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Tecken&rep_platform=Unspecified&short_desc=Symbol-upload%20permission%20for%20%3CPerson%3E&target_milestone=---&version=unspecified>`__. +If you don't have an account yet use the template to request one. + +After symbol upload is turned on, you can upload the symbol archive +directly using the web form at https://symbols.mozilla.org/uploads. +It is also possible to upload via automated scripts: see the `symbol upload API +docs <https://tecken.readthedocs.io/en/latest/>`__ for more +details. diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000000..460a2c9e2b --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,59 @@ +================================= +Firefox Source Tree Documentation +================================= + +.. toctree:: + :caption: Getting Started + :maxdepth: 1 + + {setup_doc} + +.. toctree:: + :caption: Working On Firefox + :maxdepth: 2 + + {contributing_doc} + +.. toctree:: + :caption: Source Code Documentation + :maxdepth: 2 + + {source_doc} + +.. toctree:: + :caption: The Firefox Build System + :maxdepth: 1 + + {build_doc} + +.. toctree:: + :caption: Testing & Test Infrastructure + :maxdepth: 1 + + {testing_doc} + +.. toctree:: + :caption: Localization & Internationalization + :maxdepth: 2 + + {l10n_doc} + +.. toctree:: + :caption: Firefox and Python + :maxdepth: 1 + + {python_doc} + +.. toctree:: + :caption: Metrics Collected in Firefox + :maxdepth: 1 + + {metrics_doc} + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/jsdoc.json b/docs/jsdoc.json new file mode 100644 index 0000000000..534334b7dd --- /dev/null +++ b/docs/jsdoc.json @@ -0,0 +1,5 @@ +{ + "source": { + "includePattern": ".+\\.jsm?$" + } +} diff --git a/docs/setup/building_with_debug_symbols.rst b/docs/setup/building_with_debug_symbols.rst new file mode 100644 index 0000000000..316e04af31 --- /dev/null +++ b/docs/setup/building_with_debug_symbols.rst @@ -0,0 +1,61 @@ +Building with Debug Symbols +=========================== + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +By default, a release build of Firefox will not generate debug symbols +suitable for debugging or post-processing into the +:ref:`breakpad <Crash reporting>` symbol format. Use the +following :ref:`mozconfig <Configuring Build Options>` settings +to do a build with symbols: + + + +Building Firefox with symbols +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There is a single configure option to enable building with symbols on +all platforms. This is enabled by default so unless you have explcitly +disabled it your build you should include symbols. + +:: + + ac_add_options --enable-debug-symbols + +This can optionally take an argument for the type of symbols that need +to be produced (like "-g3"). By default it uses "-g" on Linux and MacOS. +This value takes precedence over the flags set in ``MOZ_DEBUG_FLAGS`` + +Note that this will override the values provided for ``CFLAGS`` and +``CXXFLAGS``. + + +Breakpad symbol files +~~~~~~~~~~~~~~~~~~~~~ + +After the build is complete, run the following command to generate an +archive of :ref:`Breakpad <Crash reporting>` symbol files: + +.. code:: bash + + mach buildsymbols + +Treeherder uses an additional ``uploadsymbols`` target to upload +symbols to a socorro server. See +https://searchfox.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py +for more information about the environment variables used by this +target. + + +``make package`` +~~~~~~~~~~~~~~~~ + +If you use ``make package`` to package your build, symbols will be +stripped. If you want to keep the symbols in the patches, you need to +add this to your mozconfig: + +.. code:: + + ac_add_options --disable-install-strip diff --git a/docs/setup/configuring_build_options.rst b/docs/setup/configuring_build_options.rst new file mode 100644 index 0000000000..b870632ad4 --- /dev/null +++ b/docs/setup/configuring_build_options.rst @@ -0,0 +1,440 @@ +Configuring Build Options +========================= + ++--------------------------------------------------------------------+ +| This page is an import from MDN and the contents might be outdated | ++--------------------------------------------------------------------+ + +This document details how to configure Firefox builds. +Most of the time a ``mozconfig`` file is not required. The default +options are the most well-supported, so it is preferable to add as few +options as possible. Please read the following directions carefully +before building, and follow them in order. Skipping any step may cause +the build to fail, or the built software to be unusable. Build options, +including options not usable from the command-line, may appear in +"``confvars.sh``" files in the source tree. + + +Using a ``mozconfig`` configuration file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The choice of which Mozilla application to build and other configuration +options can be configured in a ``mozconfig`` file. (It is possible to +manually call ``configure`` with command-line options, but this is not +recommended). The ``mozconfig`` file should be in your source directory +(that is, ``/mozilla-central/mozconfig``). + +Create a blank ``mozconfig`` file: + +.. code:: bash + + echo "# My first mozilla config" > mozconfig + +If your mozconfig isn't in your source directory, you can also use the +``MOZCONFIG`` environment variable to specify the path to your +``mozconfig``. The path you specify **must** be an **absolute** path or +else ``client.mk`` will not find it. This is useful if you choose to +have multiple ``mozconfig`` files for different applications or +configurations (see below for a full example). Note that in the +``export`` example below the filename was not ``mozconfig``. Regardless +of the name of the actual file you use, we refer to this file as the +``mozconfig`` file in the examples below. + +Setting the ``mozconfig`` path: + +.. code:: bash + + export MOZCONFIG=$HOME/mozilla/mozconfig-firefox + +.. note:: + + Calling the file ``.mozconfig`` (with a leading dot) is also + supported, but this is not recommended because it may make the file + harder to find. This will also help when troubleshooting because + people will want to know which build options you have selected and + will assume that you have put them in your ``mozconfig`` file. + + +``mozconfig`` contains two types of options: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Options prefixed with ``mk_add_options`` are passed to + ``client.mk``. The most important of these is ``MOZ_OBJDIR``, which + controls where your application gets built (also known as the object + directory). +- Options prefixed with ``ac_add_options`` are passed to ``configure``, + and affect the build process. + + +Building with an objdir +~~~~~~~~~~~~~~~~~~~~~~~ + +This means that the source code and object files are not intermingled in +your directory system and you can build multiple applications (e.g., +Firefox and Thunderbird) from the same source tree. If you do not +specify a ``MOZ_OBJDIR``, it will be automatically set to +``@TOPSRCDIR@/obj-@CONFIG_GUESS@``. + +If you need to re-run ``configure``, the easiest way to do it is using +``./mach configure``; running ``configure`` manually is strongly +discouraged. + +Adding the following line to your ``mozconfig`` allows you to change the +objdir: + +.. code:: bash + + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-@CONFIG_GUESS@ + +It is a good idea to have your objdir name start with ``obj`` so that +Mercurial ignores it. + +Sometimes it can be useful to build multiple versions of the source +(such as with and without diagnostic asserts). To avoid the time it +takes to do a full rebuild, you can create multiple ``mozconfig`` files +which specify different objdirs. For example, a ``mozconfig-dbg``: + +.. code:: bash + + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-ff-dbg + ac_add_options --enable-debug + +and a ``mozconfig-rel-opt``: + +.. code:: bash + + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-ff-rel-opt + ac_add_options --disable-debug + ac_add_options --enable-optimize + +allow for building both versions by specifying the configuration via +the ``MOZCONFIG`` environment variable: + +.. code:: bash + + $ env MOZCONFIG=/path/to/mozconfig-dbg ./mach build + $ env MOZCONFIG=/path/to/mozconfig-rel-opt ./mach build + +Don't forget to set the ``MOZCONFIG`` environment variable for the +``mach run`` command as well. + +Be aware that changing your ``mozconfig`` will require the configure +process to be rerun and therefore the build will take considerably +longer, so if you find yourself changing the same options regularly, it +may be worth having a separate ``mozconfig`` for each. The main downside +of this is that each objdir will take up a significant amount of space +on disk. + + +Parallel compilation +~~~~~~~~~~~~~~~~~~~~ + +.. note:: + + **Note**: The build system automatically makes an intelligent guess + for how many CPU cores to use when building. The option below is + typically not needed. + +Most modern systems have multiple cores or CPUs, and they can be +optionally used concurrently to make the build faster. The ``-j`` flag +controls how many parallel builds will run concurrently. You will see +(diminishing) returns up to a value approximately 1.5× to 2.0× the +number of cores on your system. + +.. code:: bash + + mk_add_options MOZ_MAKE_FLAGS="-j4" + +If your machine is overheating, you might want to try a lower value, +e.g. ``-j1``. + + +Choose an application +~~~~~~~~~~~~~~~~~~~~~ + +The ``--enable-application=application`` flag is used to select an +application to build. Firefox is the default. + +Choose one of the following options to add to your ``mozconfig`` file: + +Browser (Firefox) + .. code:: + + ac_add_options --enable-application=browser + + .. note:: + + **Note**: This is the default + +Mail (Thunderbird) + .. code:: + + ac_add_options --enable-application=comm/mail + +Mozilla Suite (SeaMonkey) + .. code:: + + ac_add_options --enable-application=suite + +Calendar (Lightning Extension, uses Thunderbird) + .. code:: + + ac_add_options --enable-application=comm/mail + ac_add_options --enable-calendar + + +Selecting build options +~~~~~~~~~~~~~~~~~~~~~~~ + +The build options you choose depends on what application you are +building and what you will be using the build for. If you want to use +the build regularly, you will want a release build without extra +debugging information; if you are a developer who wants to hack the +source code, you probably want a non-optimized build with extra +debugging macros. + +There are many options recognized by the configure script which are +special-purpose options intended for embedders or other special +situations, and should not be used to build the full suite/XUL +applications. The full list of options can be obtained by running +``./configure --help``. + +.. warning:: + + Do not use a configure option unless you know what it does. + The default values are usually the right ones. Each additional option + you add to your ``mozconfig`` file reduces the chance that your build + will compile and run correctly. + +The following build options are very common: + +sccache +^^^^^^^ + +`SCCache <https://github.com/mozilla/sccache>`__ allows speeding up subsequent +C / C++ builds by caching compilation results. Unlike +`ccache <https://ccache.dev>`__, it also allows caching Rust artifacts, and +supports `distributed compilation +<https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md>`__. + +In order to enable ``sccache`` for Firefox builds, you can use +``ac_add_options --with-ccache=sccache``. + +Optimization +^^^^^^^^^^^^ + +``ac_add_options --enable-optimize`` + Enables the default compiler optimization options + + .. note:: + + **Note**: This is enabled by default + +``ac_add_options --enable-optimize=-O2`` + Chooses particular compiler optimization options. In most cases, this + will not give the desired results, unless you know the Mozilla + codebase very well; note, however, that if you are building with the + Microsoft compilers, you probably **do** want this as ``-O1`` will + optimize for size, unlike GCC. +``ac_add_options --enable-debug`` + Enables assertions in C++ and JavaScript, plus other debug-only code. + This can significantly slow a build, but it is invaluable when + writing patches. **People developing patches (especially in C++) + should generally use this option.** +``ac_add_options --disable-optimize`` + Disables compiler optimization. This makes it much easier to step + through code in a debugger. +``ac_add_options --enable-release`` + Enables more conservative, release engineering-oriented options. This may + slow down builds. This also turns on full optimizations for Rust. Note this + is the default when building release/beta/esr. +``ac_add_options --enable-debug-js-modules`` + Enable only JavaScript assertions. This is useful when working + locally on JavaScript-powered components like the DevTools. This will + help catch any errors introduced into the JS code, with less of a + performance impact compared to the ``--enable-debug`` option. +``export RUSTC_OPT_LEVEL=2`` + Enable full optimizations for Rust code. + +You can make an optimized build with debugging symbols. See :ref:`Building +with Debug Symbols <Building with Debug Symbols>`. + +Extensions +^^^^^^^^^^ + +``ac_add_options --enable-extensions=default|all|ext1,ext2,-skipext3`` + There are many optional pieces of code that live in {{ + Source("extensions/") }}. Many of these extensions are now considered + an integral part of the browsing experience. There is a default list + of extensions for the suite, and each app-specific ``mozconfig`` + specifies a different default set. Some extensions are not compatible + with all apps, for example: + + - ``cookie`` is not compatible with thunderbird + - ``typeaheadfind`` is not compatible with any toolkit app (Firefox, + Thunderbird) + + Unless you know which extensions are compatible with which apps, do + not use the ``--enable-extensions`` option; the build system will + automatically select the proper default set of extensions. + +Tests +^^^^^ + +``ac_add_options --disable-tests`` + By default, many auxiliary test applications are built, which can + help debug and patch the mozilla source. Disabling these tests can + speed build time and reduce disk space considerably. Developers + should generally not use this option. + +Localization +^^^^^^^^^^^^ + +``mk_add_options MOZ_CO_LOCALES=ISOcode`` + TBD. +``ac_add_options --enable-ui-locale=ISOcode`` + TBD. +``ac_add_options --with-l10n-base=/path/to/base/dir`` + TBD. + +Other Options +^^^^^^^^^^^^^ + +``mk_add_options AUTOCLOBBER=1`` + If a clobber would be required before a build, this will cause mach + to clobber and continue with the build instead of asking the user to + manually clobber and exiting. + +``ac_add_options --enable-crashreporter`` + This enables the machinery that allows Firefox to write out a + `minidump <https://docs.microsoft.com/en-us/windows/desktop/Debug/minidump-files>`__ + files when crashing as well as the tools to process and submit crash + reports to Mozilla. After enabling the crash reporter in your local + build, you will need to run mach with the --enable-crash-reporter + (note the extra dash) to enable it at runtime, like so: + ``./mach run --enable-crash-reporter`` +``ac_add_options --enable-warnings-as-errors`` + This makes compiler warnings into errors which fail the build. This + can be useful since certain warnings coincide with reviewbot lints + which must be fixed before merging. + +.. _Example_.mozconfig_Files: + +Example ``mozconfig`` Files +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Mozilla's official builds use mozconfig files from the appropriate +directory within each repository. + +.. warning:: + + These ``mozconfig`` files are taken from production builds + and are provided as examples only. It is recommended to use the default + build options, and only change the properties from the list above as + needed. The production builds aren't really appropriate for local + builds." + +- .. rubric:: Firefox, `Debugging Build (macOS + 64bits) <http://hg.mozilla.org/mozilla-central/file/tip/browser/config/mozconfigs/macosx64/debug>`__ + :name: Firefox.2C_Default_Release_Configuration + +Building multiple applications from the same source tree +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is possible to build multiple applications from the same source tree, +as long as you `use a different objdir <#Building_with_an_Objdir>`__ for +each application. + +You can either create multiple ``mozconfig`` files, or alternatively, +use the ``MOZ_BUILD_PROJECTS`` make option. + +Using ``MOZ_BUILD_PROJECTS`` in a single ``mozconfig`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To use ``MOZ_BUILD_PROJECTS``, you must specify a ``MOZ_OBJDIR`` and a +``MOZ_BUILD_PROJECTS`` make option, containing space separated names. +Each name can be an arbitrary directory name. For each name, a +subdirectory is created under the toplevel objdir. You then need to use +the ``ac_add_app_options`` with the specified names to enable different +applications in each object directory. + +For example: + +.. code:: + + ac_add_options --disable-optimize --enable-debug + mk_add_options MOZ_OBJDIR=/mozilla/src/obj-@CONFIG_GUESS@ + mk_add_options MOZ_BUILD_PROJECTS="browser mail" + ac_add_app_options browser --enable-application=browser + ac_add_app_options mail --enable-application=comm/mail + +If you want to build only one project using this ``mozconfig``, use the +following command line: + +.. code:: + + MOZ_CURRENT_PROJECT=browser ./mach build + +This will build only the browser. + +Using multiple mozconfig files +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Alternatively, you may want to create separate ``mozconfig`` files. + +As an example, the following steps can be used to build Firefox and +Thunderbird. You should first create three ``mozconfig`` files. + +``mozconfig-common``: + +.. code:: + + # add common options here, such as making an optimized release build + mk_add_options MOZ_MAKE_FLAGS="-j4" + ac_add_options --enable-optimize --disable-debug + +``mozconfig-firefox``: + +.. code:: + + # include the common mozconfig + . ./mozconfig-common + + # Build Firefox + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-firefox + ac_add_options --enable-application=browser + +``mozconfig-thunderbird``: + +.. code:: + + # include the common mozconfig + . ./mozconfig-common + + # Build Thunderbird + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-thunderbird + ac_add_options --enable-application=comm/mail + +To build Firefox, run the following commands: + +.. code:: + + export MOZCONFIG=/path/to/mozilla/mozconfig-firefox + ./mach build + +To build Thunderbird, run the following commands: + +.. code:: + + export MOZCONFIG=/path/to/mozilla/mozconfig-thunderbird + ./mach build + +Using mozconfigwrapper +^^^^^^^^^^^^^^^^^^^^^^ + +Mozconfigwrapper is similar to using multiple mozconfig files except +that it abstracts and hides them so you don't have to worry about where +they live or which ones you've created. It also saves you from having to +export the MOZCONFIG variable each time. For information on installing +and configuring mozconfigwrapper, see +https://github.com/ahal/mozconfigwrapper. diff --git a/docs/setup/contributing_code.rst b/docs/setup/contributing_code.rst new file mode 100644 index 0000000000..91a1264e0d --- /dev/null +++ b/docs/setup/contributing_code.rst @@ -0,0 +1,199 @@ +How To Contribute Code To Firefox +================================= + +The whole process can be a bit long, and it might take time to get things right. +If at any point you are stuck, please don't hesitate to ask at `https://chat.mozilla.org <https://chat.mozilla.org>`_ +in the `#introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`_ channel. + +We make changes to Firefox by writing patches, testing them and pushing them into "the tree", the +term we use for all the code in Mozilla-Central. Let's get started. + +Please see the :ref:`Firefox Contributors Quick Reference <Firefox Contributors' Quick Reference>` for simple check list. + +Finding something to work on +---------------------------- + +| Bugs listed as 'Assigned' are not usually a good place to start, + unless you're sure you have something worthy to contribute. Someone + else is already working on it! +| Even with no assignee, it is polite to check if someone has recently + commented that they're looking at fixing the issue. +| Once you have found something to work on, go ahead and comment! Let + the bug submitter, reviewer, and component owner know that you'd like + to work on the bug. You might receive some extra information, perhaps + also made the assignee. + +Find a bug we've identified as a good fit for new contributors. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +With more than a million bugs filed in Bugzilla, it can be hard to know +where to start, so we've created these bug categories to make getting +involved a little easier: + +- `Codetribute <https://codetribute.mozilla.org/>`_ - our site for + finding bugs that are mentored, some are good first bugs, some are + slightly harder. Your mentor will help guide you with the bug fix and + through the submission and landing process. +- `Good First Bugs <https://mzl.la/2yBg3zB>`_ + - are the best way to take your first steps into the Mozilla + ecosystem. They're all about small changes, sometimes as little as a + few lines, but they're a great way to learn about setting up your + development environment, navigating Bugzilla, and making + contributions to the Mozilla codebase. +- Visit `firefox-dev.tools <http://firefox-dev.tools>`_ - we list + Firefox Developer Tools bugs for new contributors. +- `Student Projects <https://bugzil.la/kw:student-project>`_ - are + larger projects, such as might be suitable for a university student + for credit. Of course, if you are not a student, feel free to fix one + of these bugs. We maintain two lists: one for projects `based on the + existing codebase <https://bugzil.la/kw:student-project>`_. + +Fix that one bug +~~~~~~~~~~~~~~~~ + +If there's one particular bug you'd like to fix about Firefox, Thunderbird, or +your other favorite Mozilla application, this can be a great place to +start. There are a number of ways to do this: + +- `Search bugzilla <https://bugzilla.mozilla.org/query.cgi>`_ for + relevant keywords. See pages on + `Bugzilla <https://developer.mozilla.org/docs/Mozilla/Bugzilla>`_ and `Searching + Bugzilla <https://developer.mozilla.org/docs/Mozilla/QA/Searching_Bugzilla>`_ for further + help +- Learn the `bugzilla + component <https://bugzilla.mozilla.org/describecomponents.cgi>`_, + with which your pet bug is implemented, using the components list. + Browse this component on bugzilla for related bugs + +Fixing your bug +--------------- + +We leave this in your hands. Here are some further resources to help: + +- Check out + `https://developer.mozilla.org/docs/Developer_Guide <https://developer.mozilla.org/docs/Developer_Guide>`_ + and its parent document, + https://developer.mozilla.org/docs/Mozilla +- Our :ref:`reviewer checklist <Reviewer_Checklist>` is very + useful, if you have a patch near completion, and seek a favorable + review +- Utilize our build tool :ref:`mach`, its linting, + static analysis, and other code checking features + +Getting your code reviewed +-------------------------- + +Once you fix the bug, you can advance to having your code reviewed. + +Mozilla uses +`Phabricator <https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html>`_ +for code review. + +Who is the right person to ask for a review? + +- If you have a mentored bug: ask your mentor. They will help, or can + easily find out. It might be them! +- Run ``hg blame`` on the file and look for the people who have touched + the functions you're working on. They too are good candidates. + Running ``hg log`` and looking for regular reviewers might be a + solution too. +- The bug itself may contain a clear indication of the best person to + ask for a review +- Are there related bugs on similar topics? The reviewer in those bugs + might be another good choice +- We have an out of date `list of + modules <https://wiki.mozilla.org/Modules>`_, which lists peers and + owners for the module. Some of these will be good reviewers. In a + worst case scenario, set the module owner as the reviewer, asking + them in the comments to pick someone more suitable + +Please select only one reviewer. + +Following up and responding +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you've asked for a review, a reviewer will often respond within a +day or two, reviewing the patch, or saying when they will be able to +review it, perhaps due to a backlog. If you don't hear back within this +time, naturally reach out to them: add a comment to the bug saying +'review ping?', check the "Need more information from" box, and add the +reviewer's name. If they don't respond within a day or two, you can ask +for help on Matrix in the +`#introduction:mozilla.org <https://riot.im/app/#/room/#introduction:mozilla.org>`_ +or +`#developers:mozilla.org <https://chat.mozilla.org/#/room/#developers:mozilla.org>`_ +channels, or contact `Mike +Hoye <mailto:mhoye@mozilla.com?subject=Code%20Review%20Request%20&body=URL%3A%20%20%5Bplease%20paste%20a%20link%20to%20your%20patch%20here.%5D>`_ +directly. + +Don't hesitate to contact your mentor as well if this isn't moving. + +For most new contributors, and even for long-time Mozillians, the first +review of your patch will be "Requested Changes" (or an "r-" in +Bugzilla). This does not mean you've done bad work. There is more work +to do before the code can be merged into the tree. Your patch may need +some changes - perhaps minor, perhaps major - and your reviewer will +give you some guidance on what needs to be done next. + +This is an important process, so don't be discouraged! With our +long-lived codebase, and hundreds of millions of users, the care and +attention helping contributors bring good patches is the cornerstone of +the Mozilla project. Make any changes your reviewer seeks; if you're +unsure how, be sure to ask! Push your new patch up to Phabricator again and +ask for a further review from the same reviewer. If they accept your +changes, this means your patch can be landed into the tree! + +Getting code into Firefox +------------------------- + +Once your patch has been accepted, it is ready to go. Before it can be +merged into the tree, your patch will need to complete a successful run +through our `try +server <https://wiki.mozilla.org/ReleaseEngineering/TryServer>`_, +making sure there are no unexpected regressions. If you don't have try +server access already, your mentor, or the person who reviewed your +patch, will be able to help. + +Once you have a **green** try server run, mark that your patch is ready +to commit by + +#. opening the Phabricator page for your patch +#. clicking the 'Edit Revision' link in the sidebar on the right +#. then into the 'Tags' field and +#. typing 'Check-In Needed' to get the tag added. + +A friendly Mozillian, with commit access, will be along shortly to push +your patch to the repository, and update the bug as required. If your +patch passes all Mozilla's automated testing, it will soon be merged +into the main branch, and become a part of the Nightly build. + +Do it all again! +---------------- + +Thank you. You've fixed your very first bug, and the Open Web is +stronger for it. But don't stop now. + +Go back to step 3, as there is plenty more to do. Your mentor might +suggest a new bug for you to work on, or `find one that interests +you <http://www.whatcanidoformozilla.org/>`_. Now that you've got your +first bug fixed you should request level 1 access to the repository to +push to the try server and get automated feedback about your changes on +multiple platforms. After fixing a nontrivial number of bugs you should +request level 3 access so you can land your own code after it has been +reviewed. + +More information +---------------- + +We're in the process of improving information on this page for newcomers +to the project. We'll be integrating some information from these pages +soon, but until then you may find them interesting in their current +form: + +- `A guide to learning the Firefox + codebase <http://www.joshmatthews.net/blog/2010/03/getting-involve-with-mozilla/>`_ +- `A beginner's guide to SpiderMonkey, Mozilla's Javascript + engine <https://wiki.mozilla.org/JavaScript:New_to_SpiderMonkey>`_ +- `Mozilla platform development + cheatsheet <https://web.archive.org/web/20160813112326/http://www.codefirefox.com:80/cheatsheet>`_ + (archive.org) diff --git a/docs/setup/getting_set_up.rst b/docs/setup/getting_set_up.rst new file mode 100644 index 0000000000..63482148aa --- /dev/null +++ b/docs/setup/getting_set_up.rst @@ -0,0 +1,65 @@ +Welcome to the Firefox codebase! +-------------------------------- + +This page is here to help you get from "I want to build Firefox" +to "I'm building my own Firefox" to "I can contribute to Firefox". +So if you'd like to help Mozilla build the best web browser in the +world, you're in the right place. + +.. rubric:: Need help? + :name: Need_help + +The Mozilla community prides itself on being an open, accessible, and +friendly community for new participants. If you have any difficulties +getting involved or finding answers to your questions, please `come and +ask your questions in our +chatroom <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`_, +where we can help you get started. + +We know even before you start contributing that getting set up to work +on Firefox and finding a bug that's a good fit for your skills can be a +challenge, and we're always looking for ways to improve this process: making +Mozilla more open, accessible, and easier to participate with. If you're +having any trouble following this documentation, or hit a barrier you +can't get around, please join us in the the Introduction room on Matrix +or contact Mike Hoye directly at mhoye@mozilla.com. + +What skills do I need? +---------------------- + +Mozilla is a large project and we are thrilled to have contributors with +very diverse skills: + +- If you know **C++,** **Rust,** **JavaScript,** **HTML** or **CSS**, + you can contribute to the core layers of Firefox and many other Mozilla + projects. +- If you know **Rust**, you can also contribute to the Rust programming + language itself, and `Servo <https://servo.org/>`_, the web browser engine + designed for parallelism and safety. +- If you know **Java**, you can contribute to Firefox on Android, + `Firefox Focus for + Android <https://github.com/mozilla-mobile/focus-android>`_ . +- If you know **Kotlin**, you can contribute to `Firefox + Preview <https://github.com/mozilla-mobile/fenix>`_ (code name: + "Fenix"). +- If you know **Swift**, you can contribute to `Firefox for + iOS <https://github.com/mozilla-mobile/firefox-ios>`_ and `Firefox + Focus for iOS <https://github.com/mozilla-mobile/focus-ios>`_ +- If you know **Python**, you can contribute to our web services, + including Firefox Sync and Firefox Accounts +- If you know **Make**, **shell**, **Perl**, or **Python**, you can + contribute to our build systems, release engineering, and automation +- If you know **C**, you can contribute to NSS, Opus, and Daala +- There are even many ways to contribute to the Mozilla mission without + programming. If getting involved in design, support, translation, + testing, or other types of contributions sparks your interest please + see the `Volunteer Opportunities + wiki <https://contribute.mozilla.org>`_ or the `Mozilla + community <https://mozilla.community/>`_ site. + +Perhaps you do not know programming yet, but you want to start learning? +There are `plenty of +resources <https://developer.mozilla.org/learn>`_ available on +the MDN Web Docs! + +Read on for information about how to set up your machine to build Firefox. diff --git a/docs/setup/index.rst b/docs/setup/index.rst new file mode 100644 index 0000000000..332ac8d001 --- /dev/null +++ b/docs/setup/index.rst @@ -0,0 +1,21 @@ +Getting Set Up To Work On The Firefox Codebase +============================================== + +This page will help you get set up to build Firefox on your own machine. + +.. toctree:: + :caption: Thank you for contributing to Firefox + + getting_set_up + +.. toctree:: + :caption: Setting Up Your Machine + + windows_build + macos_build + linux_build + +.. toctree:: + :caption: Getting Ready To Contribute + + contributing_code diff --git a/docs/setup/linux_32bit_build_on_64bit_OS.rst b/docs/setup/linux_32bit_build_on_64bit_OS.rst new file mode 100644 index 0000000000..fb6682672e --- /dev/null +++ b/docs/setup/linux_32bit_build_on_64bit_OS.rst @@ -0,0 +1,78 @@ +Building Firefox 32-bit On Linux 64-bit +======================================= + +Instructions for Fedora 20 and 19 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First ensure that your compiler toolchain and Gecko build dependencies +are installed. + +.. code:: + + sudo yum install \ + ccache cmake gcc gcc-c++ glibc-devel.i686 \ + libstdc++-devel libstdc++-devel.i686 \ + gtk2-devel.i686 gtk+-devel.i686 gtk+extra-devel.i686 \ + glib-devel.i686 glib2-devel.i686 \ + dbus-devel.i686 dbus-glib-devel.i686 \ + alsa-lib-devel.i686 yasm-devel.i686 \ + libxml2-devel.i686 zlib-devel.i686 \ + freetype-devel.i686 \ + atk-devel.i686 pango-devel.i686 fontconfig-devel.i686 \ + cairo-devel.i686 gdk-pixbuf2-devel.i686 \ + libX11-devel.i686 libXt-devel.i686 libXext-devel.i686 \ + libXrender-devel.i686 libXau-devel.i686 libxcb-devel.i686 \ + pulseaudio-libs-devel.i686 harfbuzz-devel.i686 \ + mesa-libGL-devel.i686 libXxf86vm-devel.i686 \ + libXfixes-devel.i686 libdrm-devel-2.4.49-2.fc19.i686 \ + mesa-libEGL-devel libXdamage-devel.i686 libXcomposite-devel.i686 + +Then you need to use a .mozconfig that looks like the following example. + +.. code:: + + # Flags set for targeting x86. + export CROSS_COMPILE=1 + export PKG_CONFIG_PATH=/usr/lib/pkgconfig + + CC="ccache gcc -m32" + CXX="ccache g++ -m32" + AR=ar + ac_add_options --target=i686-pc-linux + + # Normal build flags. These make a debug browser build. + ac_add_options --enable-application=browser + mk_add_options MOZ_MAKE_FLAGS="-s -j6" + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../ff-dbg + + ac_add_options --enable-debug + ac_add_options --disable-optimize + +Instructions for Ubuntu 19.10 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These steps were verified to work as of June 2020 + +#. Install mercurial or git and :ref:`fetch a copy <Mercurial Bundles>` of ``mozilla-central``. +#. Run ``./mach bootstrap`` to install some dependencies. Note that this + will install some amd64 packages that are not needed to build 32-bit + Firefox. +#. Run ``rustup target install i686-unknown-linux-gnu`` to install the + 32-bit Rust target. +#. Install 32-bit dependencies with the following command (this shouldn't try to + remove packages. If this is the case, those instructions won't work as-is): + + .. code:: + + sudo apt install gcc-multilib g++-multilib libdbus-glib-1-dev:i386 \ + libgtk2.0-dev:i386 libgtk-3-dev:i386 libpango1.0-dev:i386 libxt-dev:i386 \ + libx11-xcb-dev:i386 libpulse-dev:i386 libdrm-dev:i386 + +5. Create a file called ``mozconfig`` in the top-level directory of you + ``mozilla-central`` checkout, containing at least the following: + + .. code:: + + ac_add_options --target=i686 + +6. Run ``./mach build``. diff --git a/docs/setup/linux_build.rst b/docs/setup/linux_build.rst new file mode 100644 index 0000000000..321db0a96a --- /dev/null +++ b/docs/setup/linux_build.rst @@ -0,0 +1,282 @@ +Building Firefox On Linux +========================= + +They aren’t complicated, but there are a few prerequisites to building Firefox on Linux. You need: + +#. A 64-bit installation of Linux. You can check by opening a terminal window; if ``uname -m`` returns ``x86_64`` you can proceed. +#. Next, you’ll need Python 3.6 or later installed. You can check with ``python3 --version`` to see if you have it already. If not, see `Installing Python <#installingpython>`_. You'll also need to install Mercurial and can do so with ``pip3 install Mercurial``. +#. Finally, a reasonably fast internet connection and 30GB of free disk space. + +Getting Started +--------------- + +Getting set up on Linux is fast and easy. + +If you don’t have one yet, create a "``src``" directory for +yourself under your home directory: + +.. code-block:: shell + + mkdir src && cd src + +Next `download the bootstrap.py +script <https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py>`_ +and save it in the ``src/`` directory created above. + +.. warning:: + + Building Firefox in Linux on top of a non-native file system - + for example, on a mounted NTFS partition - is explicitly not + supported. While a build environment like this may succeed it + may also fail while claiming to have succeeded, which can be + quite difficult to diagnose and fix. + +And finally, in your terminal from above start the bootstrapper +like this: + +.. code-block:: shell + + python3 bootstrap.py + +... and follow the prompts. This will use mercurial to checkout +the source code. If you prefer to work with ``git``, use this command +instead (make sure you have ``git`` installed): + +.. code-block:: shell + + python3 bootstrap.py --vcs=git + +Let’s Build Firefox +------------------- + +You’re ready; now we can tie it all together. In your terminal: + +.. code-block:: shell + + cd mozilla-unified # ... or the name of the repo you chose in the above step + +If you are not working on the C/C++ files you can also opt for +:ref:`Artifact Builds <Understanding Artifact Builds>` +which are much faster. To enable artifact build set up a +:ref:`mozconfig <Configuring Build Options>` +file with the following options: + +.. code-block:: shell + + # Automatically download and use compiled C++ components: + # This option will disable C/C++ compilation + ac_add_options --enable-artifact-builds + + # Write build artifacts to (not mandatory): + mk_add_options MOZ_OBJDIR=./objdir-frontend + +If you plan to walk through code with a debugger, set up a +:ref:`.mozconfig <Configuring Build Options>` +file with the following options: + +.. code-block:: shell + + ac_add_options --disable-optimize + ac_add_options --enable-debug + + +Older clang versions (especially clang 6) `from LTS linux +distributions sometimes miscompile +Firefox <https://bugzilla.mozilla.org/show_bug.cgi?id=1594686>`_, +resulting in startup crashes when starting the resulting build. +If this happens, you can force the use of the ``clang`` version +that ``./mach bootstrap`` downloaded by adding the following to +your ``.mozconfig``: + +.. code-block:: shell + + export CC=path/to/home/.mozbuild/clang/bin/clang + export CXX=path/to/home/.mozbuild/clang/bin/clang++ + +And finally, run the build command: + +.. code-block:: shell + + ./mach build + +If you encounter any error related to LLVM/Clang on Ubuntu or +Debian, download the latest version of LLVM and Clang and then +re-run ``./mach build``. + +And you’re on your way, building your own copy of Firefox from +source. Don’t be discouraged if this takes a while; this takes +some time on even the fastest modern machines, and as much as two +hours or more on older hardware. When the +``--enable-artifact-builds`` option is used, builds usually finish +within a few minutes. + +Now the fun starts +------------------ + +You have the code, you’ve compiled Firefox. Fire it up with +``./mach run`` and you’re ready to start hacking. The next steps +are up to you: join us on `Matrix <https://chat.mozilla.org/>`_ +in the `Introduction <https://chat.mozilla.org/#/room/#introduction:mozilla.org>`_ +channel, and find `a bug to start working on. <https://codetribute.mozilla.org/>`_ + + +General considerations +---------------------- + +#. 4GB RAM with an additional 4GB of available swap space is the bare minimum, and more RAM is always better - having 8GB or more will dramatically improve build time. +#. A 64-bit x86 CPU and a 64-bit OS. As of early 2015 it is no longer possible to do a full build of Firefox from source on most 32-bit systems; a 64-bit OS is required. ":ref:`Artifact Builds <Understanding Artifact Builds>`" may be possible, but are not a supported configuration. On Linux you can determine this by typing "``uname -a``" in a terminal. It is possible to build a 32-bit Firefox on a 64-bit system, see :ref:`Building Firefox 32-bit on Linux 64-bit <Building Firefox 32-bit On Linux 64-bit>`. +#. A recent version of Clang is required to build Firefox. You can learn more about the features we use and their :ref:`compiler support <Using C++ in Mozilla code>`. +#. If you are on a Fedora machine then simply install the following prerequisites from the terminal window: + +.. code-block:: shell + + sudo dnf install @development-tools @c-development gtk2-devel gtk3-devel libXt-devel GConf2-devel dbus-glib-devel yasm-devel alsa-lib-devel pulseaudio-libs-devel + + +.. _installingpython: + +Installing Python +----------------- + +To build Firefox, it's necessary to have a Python of version 3.6 or later +installed. Python 2 is no longer required to build Firefox, although it is still +required for some development tasks, like testing and pushing to ``try``. + +Often, you can install both Python 2 and 3 with your system package manager. +Make sure your system is up to date! However, users on older Linux distributions +might find they are unable to install a recent enough Python 3, while users on +newer Linux distributions may find that they can no longer install Python 2.7. +`pyenv <https://github.com/pyenv/pyenv>`_ is an easy way to install arbitrary +Python versions if you fall into either of these categories. Your system package +manager may or may not provide ``pyenv``, but the ``pyenv`` GitHub repository +provides detailed `manual installation instructions +<https://github.com/pyenv/pyenv#installation>`_ in any case. + +Once you have ``pyenv`` configured properly and ``pyenv``'s ``shims`` directory +at the front of your ``$PATH``, you can easily install any version of Python +and configure your project to use them. For example, at the root of your +checkout, do the following: + +.. code-block:: shell + + pyenv install 2.7.17 + pyenv install 3.7.8 + pyenv local 3.7.8 2.7.17 + + +Requirements for Debian / Ubuntu users +-------------------------------------- + +You need a number of different packages: + +.. code-block:: shell + + # the rust compiler + aptitude install rustc + + # the rust package manager + aptitude install cargo + + # the headers of important libs + aptitude install libgtk-2-dev + aptitude install libgtk-3-dev + aptitude install libgconf2-dev + aptitude install libdbus-glib-1-dev + aptitude install libpulse-dev + + # rust dependencies + cargo install cbindgen + + # an assembler for compiling webm + aptitude install yasm + + # Python 3 dependencies. This will work on Ubuntu 18.04LTS and later or + # Debian buster and later. For earlier releases of Ubuntu or Debian, you + # may prefer to use pyenv. + aptitude install python3 python3-dev python3-pip python3-setuptools + + # Python 2 dependencies. This will work on Ubuntu versions prior to 20.04 LTS + # and Debian versions prior to bullseye. For later releases of Ubuntu or + # Debian, you may prefer to use pyenv. + aptitude install python python-dev python-pip python-setuptools + + +One-Line Bootstrapping +---------------------- + +Our system bootstrapping script can automatically install the required +dependencies. You can download and run it by copying this line and +pasting it into a terminal window: + +.. code-block:: shell + + wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O bootstrap.py && python3 bootstrap.py + +.. note:: + + Note: piping bootstrap.py to stdin of a python process will cause + interactive prompts in the bootstrap script to fail, causing the + bootstrap process to fail. You must run Python against a local file. + +If the above command fails, the reason is often because some Linux +distributions ship with an outdated list of root certificates. In this +case, you should upgrade your Linux distribution or use your browser to +download the file. That ensures that you will get it from the right +source. +If you get an error from this process, consider `filing a +bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Build%20Config>`_ +saying that the bootstrapper didn't work and `contact Mike +Hoye <mailto:mhoye@mozilla.com>` directly for help. Please include the +error message and some details about your operating system. + +If you have already checked out the source code via Mercurial or Git you +can also use :ref:`mach` with the bootstrap command: + +.. code-block:: shell + + ./mach bootstrap + + + +Common Bootstrapper Failures +---------------------------- + +.. code-block:: shell + + wget: command not found + +You may not have wget (or curl) installed. In that case, you can either +install it via your package manager: + +On Debian-based distros like Ubuntu: + +.. code-block:: shell + + sudo apt install wget + +On Fedora-based distros: + +.. code-block:: shell + + sudo dnf install wget + +or you can just `download +bootstrap.py <https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py>`_ +using your browser and then run it with this command: + +.. code-block:: shell + + python3 bootstrap.py + +In some cases people who've customized their command prompt to include +emoji or other non-text symbols have found that bootstrap.py fails with +a ``UnicodeDecodeError``. We have a bug filed for that but in the +meantime if you run into this problem you'll need to change your prompt +back to something boring. + + +More info +--------- + +The above bootstrap script supports popular Linux distributions. If it +doesn't work for you, see :ref:`Linux build prerequisites <Building Firefox On Linux>` for more. diff --git a/docs/setup/macos_build.rst b/docs/setup/macos_build.rst new file mode 100644 index 0000000000..098f2b9f5b --- /dev/null +++ b/docs/setup/macos_build.rst @@ -0,0 +1,477 @@ +Building Firefox On macOS +========================= + +This document will help you get set up to build Firefox on your own +computer. Getting set up won't be difficult, but it can take a while - +we need to download a lot of bytes! Even on a fast connection, this can +take ten to fifteen minutes of work, spread out over an hour or two. + +The details are further down this page, but this quick-start guide +should get you up and running: + +.. rubric:: Quick start (Try this first!) + :name: Quick_start_Try_this_first! + +.. rubric:: Prerequisites + :name: Prerequisites + +You will need: + +- an Apple ID to download and install Apple-distributed developer tools + mentioned below +- from 5 minutes to 1 hour to download and install Xcode, which is + large +- download and install a local copy of specific macOS SDK version + +You will need administrator permissions on your machine to install these +prerequisites. (You can verify that you have these permissions in System +Preferences -> Users & Groups.) + +See :ref:`1.1 Install Xcode and Xcode command line tools <xcode>` and :ref:`1.2 +Get the local macOS SDK <macossdk>` for more information on how to +install these prerequisites. + +.. rubric:: Getting the source + :name: Getting_the_source + :class: heading-tertiary + +Firstly you need to prepare a directory and get the bootstrap script +that will do the rest: + +.. code-block:: shell + + # the bootstrap script needs this directory, but you can choose a different target directory for the Mozilla code later + cd ~ && mkdir -p src && cd src + + # download the bootstrap script + curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -o bootstrap.py + +If you don't have Python 3.6 or later or Mercurial installed, see :ref:`2.1a Install +dependencies via Homebrew <#install-via-homebrew>` for more information on how +to do so. Then in your terminal from above start the bootstrapper like this: + +.. code-block:: shell + + python3 bootstrap.py + +... and follow the prompts. This will use mercurial to checkout the +source code. If you prefer to work with git, use this command instead: + +.. code-block:: shell + + python3 bootstrap.py --vcs=git + +If you don't have `Homebrew <https://brew.sh/>`_ or +`Ports <https://www.macports.org/>`__ installed - software package +managers that will let us install some programs we'll need - you'll be +asked to pick one. Either will work, but most Mozilla developers use +Homebrew. + +If you don't let the ``bootstrap.py`` script clone the source for you +make sure you do it manually afterward before moving onto the next step. + +.. rubric:: Build Firefox! + :name: Build_Firefox! + :class: heading-tertiary highlight-spanned + +Now we tie it all together. + +In your terminal window, ``cd`` to your Mozilla source directory chosen +before and type: + +.. code-block:: shell + + # create a minimal build options file + echo "ac_add_options --with-macos-sdk=$HOME/SDK-archive/MacOSX10.12.sdk" >> mozconfig + + ./mach bootstrap + + ./mach build + +The ``./mach bootstrap`` step is a catch-all for any dependencies not +covered in this documentation. If you are working on Firefox frontends +or building Firefox without any changes, select :ref:`Artifact Builds +<Understanding Artifact Builds>` in +the first question in ``./mach bootstrap``. Artifact builds will +complete more quickly! Artifact builds are unsuitable for those working +on C++ code. + +You’re on your way. Don’t be discouraged if this takes a while; it takes +some time even on the fastest modern machines and as much as two hours +or more on older hardware. Firefox is pretty big, because the Web is +big. + +.. rubric:: Now the Fun Starts + :name: Now_the_Fun_Starts + :class: heading-tertiary + +You have the code, you’ve compiled Firefox. Fire it up with +``./mach run`` and you’re ready to start hacking. + +Build steps (details) +--------------------- + +Building on macOS is divided into the following steps: + +#. Install Apple-distributed developer tools - Xcode, Xcode cli tools + and macOS SDK locally +#. Install supplementary build tools +#. Obtain a copy of the Mozilla source code +#. Configure the Mozilla source tree to suit your needs +#. Build Firefox + + +.. _xcode: + +1.1 Install Xcode and Xcode command line tools +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You first need to install Xcode, for which you have two options but both +require you to sign in with an Apple ID: + +- From Apple Developer Download page - `direct + link <https://developer.apple.com/download/release/>`__. Install the + latest **release** (non-beta) version of Xcode, open ``Xcode.xip``, + and then **before** **running the extracted Xcode.app, move it from + the download folder to /Applications**. (Running it from another + location may screw up various build paths, homebrew builds, etc. Fix + by running ``sudo xcode-select -switch /Applications/Xcode.app`` ) +- From the Mac App Store - `direct + link <https://apps.apple.com/us/app/xcode>`__. + +Open /Applications/Xcode.app and let it do its initial first run and +setup stuff. + +Install the Xcode command line tools by +running ``xcode-select --install`` in your terminal. + +.. _macossdk: + +1.2 Get the local macOS SDK +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Firefox currently requires a local copy of macOS 10.12 SDK to build (all +your other apps will still use your more recent version of this SDK, +most probably matching your macOS version). + +There are various issues when building the Mozilla source code with +other SDKs and that's why we recommend this specific version. + +To get the 10.12 SDK, first download Xcode 8.2 from the `More +Downloads for Apple +Developers <https://developer.apple.com/download/more/>`__ page. Once +downloaded, mount the .dmg file. Then in the Terminal run the following: + +.. code-block:: shell + + mkdir -p $HOME/SDK-archive + cp -a /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk $HOME/SDK-archive/MacOSX10.12.sdk + +2. Install supplementary build tools +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Mozilla's source tree requires a number of third-party tools and +applications to build it. You will need to install these before you can +build anything. + +You have the choice of how to install all these components. You can use +a package manager like Homebrew or Ports. Or, you can obtain, compile, +and install them individually. For simplicity and to save your time, +using a package manager is recommended. The following sections describe +how to install the packages using existing package managers. Choose +whatever package manager you prefer. + +.. _install-via-homebrew: + +2.1a Install dependencies via Homebrew +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +`Homebrew <http://brew.sh/>`__ is "the missing package manager for +macOS." It provides a simple command-line interface to install packages, +typically by compiling them from source. + +The first step is to install Homebrew. See https://brew.sh/ + +Once you have Homebrew installed, you'll need to run the following: + +.. code-block:: shell + + brew install yasm mercurial gawk ccache python + +Python 2 is never necessary solely to build Firefox, but it is still required +for some development tasks (including testing and pushing to ``try``). If your +system does not already have a Python 2 installed, you can use ``brew`` to +install one: + +.. code-block:: shell + + brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/86a44a0a552c673a05f11018459c9f5faae3becc/Formula/python@2.rb + +2.1b Install Dependencies via MacPorts +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +MacPorts is a package manager for macOS. If you are running Homebrew, +you can ignore this section. + +To install MacPorts, go to their `install +page <http://www.macports.org/install.php>`_, download the .dmg for +your platform, and install it. If you already have MacPorts installed, +ensure it is up to date by running: + +.. code:: + + sudo port selfupdate + sudo port sync + +The first of these commands will ask for your root password. + +Common errors include: + +- ``sudo`` doesn't accept a blank password: create a password for your + account in System Preferences. +- ``port`` command not found: add it to your path (see the + troubleshooting section below). + +Use MacPorts to install the packages needed for building Firefox: + +.. code:: + + sudo port install libidl yasm python27 py27-gnureadline + +You'll then see lots of output as MacPorts builds and installs these +packages and their dependencies -- it takes a while, so go grab a cup of +coffee. + +**Note:** By default, this will install Python 2.7, which in turn will +pull in all of the X11 libraries, which may take a while to build. You +don't need any of those to build Firefox; you may want to consider +adding +no\_tkinter to the install line to build a python without +support for the X11 UI packages. This should result in a much faster +install. + +**Note:** With older versions of Xcode (eg 6.4) you may need to use +MacPorts to get the proper version of clang, such as clang-3.6 or later. +See bugs in Core, Build Config referring to clang. + +2.2 Install Mercurial +~~~~~~~~~~~~~~~~~~~~~ + +Mozilla's source code is hosted in Mercurial repositories. You use +Mercurial to interact with these repositories. There are many ways to +install Mercurial on macOS: + +1. Install `official builds from + Selenic <http://mercurial.selenic.com/>`_ + +2. Install via Homebrew: + +.. code-block:: shell + + brew install mercurial + +3. Install via MacPorts: + +.. code-block:: shell + + sudo port install mercurial + +4. Install via Pip: + +.. code-block:: shell + + easy_install pip && pip install mercurial + +Once you have installed Mercurial, test it by running: + +.. code-block:: shell + + hg version + +If this works, congratulations! You'll want to configure your Mercurial +settings to match other developers. See :ref:`Getting Mozilla Source Code +Using Mercurial <Mercurial Overview>`. + +If this fails with the error "``ValueError: unknown locale: UTF-8``", +then see the +`workarounds <http://www.selenic.com/mercurial/wiki/index.cgi/UnixInstall#head-1c10f216d5b9ccdcb2613ea37d407eb45f22a394>`_ +on the Mercurial wiki's Unix Install page. + +When trying to clone a repository you may get an HTTP 500 error +(internal server error). This seems to be due to something that Mac +Mercurial sends to the server (it's been observed both with MacPort and +selenic.com Mercurial binaries). Try restarting your shell, your +computer, or reinstall Mercurial (in that order), then report back here +what worked, please. + +3. Obtain a copy of the Mozilla source code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You may want to read :ref:`Getting Mozilla Source Code +Using Mercurial <Mercurial Overview>` for the +complete instructions. + +If you are interested in Firefox development only then run the following +command, which will create a new directory, ``mozilla-central``, in the +current one with the contents of the remote repository. + +Below command will take many minutes to run, as it will be copying a +couple hundred megabytes of data over the internet. + +.. code:: + + hg clone https://hg.mozilla.org/mozilla-central/ + cd mozilla-central + +(If you are building Firefox for Android, you should now return to the +`Android build instructions <https://wiki.mozilla.org/Mobile/Fennec/Android#Mac_OS_X>`_.) + +4. Configure the build options +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In your checked out source tree create a new file, ``mozconfig``, which +will contain your build options. For more on this file, see :ref:`Configuring Build Options`. + +To get started quickly, create the file with the following contents: + +.. code:: + + # Define where build files should go. This places them in the directory + # "obj-ff-dbg" under the current source directory + mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-ff-dbg + + # Enable debug builds + ac_add_options --enable-debug + + # Use the local copy of specific version of macOS SDK compatible with Mozilla source code + ac_add_options --with-macos-sdk=$HOME/SDK-archive/MacOSX10.12.sdk + +Firefox no longer builds with gcc 4.8 or earlier, but the build system +should automatically select clang if it is available in the PATH. If +that is not the case, you need to set CC and CXX. For instance, if you +installed Clang 9 via Homebrew, then you need to have this in your +``mozconfig``: + +.. code:: + + CC=clang-9 + CXX=clang++-9 + +5. Build +~~~~~~~~ + +Once you have your ``mozconfig`` file in place, you should be able to +build! + +.. code-block:: shell + + ./mach build + +If the build step works, you should be able to find the built +application inside ``obj-ff-dbg/dist/``. If building the browser with +``--enable-debug``, the name of the application is ``NightlyDebug.app``. +To launch the application, try running the following: + +.. code-block:: shell + + ./mach run + +**Note:** The compiled application may also be named after the branch +you're building; for example, if you changed these instructions to fetch +the ``mozilla-1.9.2`` branch, the application will be named +``Namoroka.app`` or ``NamorokaDebug.app``. + +Hardware requirements +--------------------- + +There are no specific hardware requirements, provided that the hardware +accommodates all of the `software <#Software_Requirements>`_ required +to build Firefox. Firefox can take a long time to build, so more CPU, +more RAM and lots of fast disk space are always recommended. + +- **Processor:** Intel CPUs are required. Building for PowerPC chips is + not supported. +- **Memory:** 2GB RAM minimum, 8GB recommended. +- **Disk Space:** At least 30GB of free disk space. + +Software requirements +--------------------- + +- **Operating System:** macOS 10.12 or later. It is advisable to + upgrade to the latest “point” release by running Software Update, + found in the Apple menu. You will need administrative privileges to + set up your development environment +- **Development Environment:** Xcode. You can obtain this from the App + Store. +- **Package Management:** Either Homebrew or + `MacPorts <http://www.macports.org/>`_. + +These options are specific to Mozilla builds for macOS. For a more +general overview of build options and the ``mozconfig`` file, see +:ref:`Configuring Build Options`. + +- **Compiler:** Firefox releases are no longer built with gcc-4.8 or + earlier. A recent copy of clang is needed. + + - There are some options on where to get clang: + + - Newer versions of Xcode. + - Following the instructions in the `clang + website <http://clang.llvm.org/get_started.html>`__ for + information on how to get it. + - Using some of the package managers (see above). + + - Once clang is installed, make sure it is on the PATH and configure + should use it. + +The following options, specified with ``ac_add_options``, are lines that +are intended to be added to your ``mozconfig`` file. + +- **macOS SDK:** This selects the version of the system headers and + libraries to build against, ensuring that the product you build will + be able to run on older systems with less complete APIs available. + Selecting an SDK with this option overrides the default headers and + libraries in ``/usr/include``, ``/usr/lib``, and ``/System/Library``. + + .. code-block:: shell + + ac_add_options --with-macos-sdk=/path/to/SDK + + Official trunk builds use `MacOSX10.12.sdk`. Check + `build/macosx/universal/mozconfig.common <https://searchfox.org/mozilla-central/source/build/macosx/cross-mozconfig.common>`__ + for the SDK version used for official builds of any particular source + release. + + Applications built against a particular SDK will usually run on + earlier versions of macOS as long as they are careful not to use + features or frameworks only available on later versions. Note that + some frameworks (notably AppKit) behave differently at runtime + depending on which SDK was used at build time. This may be the source + of bugs that only appear on certain platforms or in certain builds. + +For macOS builds, defines are set up as follows: + +- ``XP_MACOSX`` is defined +- ``XP_UNIX`` is defined +- ``XP_MAC`` is **not** defined. ``XP_MAC`` is obsolete and has been + removed from the source tree (see {{ Bug(281889) }}). It was used for + CFM (non-Mach-O) builds for the classic (pre-X) Mac OS. + +This requires care when writing code for Unix platforms that exclude +Mac: + +.. code-block:: shell + + #if defined(XP_UNIX) && !defined(XP_MACOSX) + +Troubleshooting +--------------- + +- **If configure (or generally building with clang) fails with + ``fatal error: 'stdio.h' file not found``:** Make sure the Xcode + command line tools are installed by running. + ``xcode-select --install``. +- **For inexplicable errors in the configure phase:** Review all + modifications of your PATH in .bash\_profile, .bash\_rc or whatever + configuration file you're using for your chosen shell. Removing all + modifications and then re-adding them one-by-one can narrow down + problems. diff --git a/docs/setup/windows_build.rst b/docs/setup/windows_build.rst new file mode 100644 index 0000000000..945d38c349 --- /dev/null +++ b/docs/setup/windows_build.rst @@ -0,0 +1,476 @@ +Building Firefox On Windows +=========================== + +Thank you for helping us build the world's best browser on the world's +most popular OS. This document will help you get set up to build and +hack on your own version of Firefox on your local machine. + +Getting set up won't be difficult, but it can take a while - we need to +download a lot of bytes! Even on a fast connection, this can take ten to +fifteen minutes of work, spread out over an hour or two. + +The details are further down this page, but this quick start guide +should get you up and running: + +Getting ready +------------- + +To build Firefox on Windows, you need a 64-bit version of Windows 7 or +later and about 40 GB of free space on your hard drive. You can make +sure your version of Windows is 64-bit on Windows 7 by right-clicking on +“Computer” in your “Start” menu, clicking “Properties” and then +“System”. On Windows 8.1 and Windows 10, right-clicking the “Windows” +menu button and choosing “System” will show you the same information. +You can alternatively press the “Windows” and “Pause Break” buttons +simultaneously on your keyboard on any version of Windows. + +Next, we want to start on a solid foundation: make sure you’re up to +date with Windows Update and then we’ll get moving. + +.. note:: + + Microsoft is providing some Windows images with Visual Studio and code + https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ + + +Visual Studio 2019 +~~~~~~~~~~~~~~~~~~ + + +.. note:: + + As of `bug + 1483835 <https://bugzilla.mozilla.org/show_bug.cgi?id=1483835>`_, local + Windows builds `use clang-cl by + default <https://groups.google.com/d/topic/mozilla.dev.platform/MdbLAcvHC0Y/discussion>`_ + as compiler. Visual Studio is still necessary for build tools, headers, + and SDK. + +.. note:: + + Automation builds still use Visual Studio 2017, so there may be some + divergence until we upgrade. `Bug + 1581930 <https://bugzilla.mozilla.org/show_bug.cgi?id=1581930>`_ tracks + various issues building with 2019. Please file your issue there and + downgrade in the interim if you encounter build failures. + +`Download and install the Community +edition <https://visualstudio.microsoft.com/downloads/>`_ of Visual +Studio 2019. Professional and Enterprise are also supported if you have +either of those editions. + +.. note:: + + When installing, the following workloads must be checked: + + - "Desktop development with C++" (under the Windows group) + - "Game development with C++" (under the Mobile & Gaming group) + + In addition, go to the Individual Components tab and make sure the + following components are selected under the "SDKs, libraries, and + frameworks" group: + + - "Windows 10 SDK" (at least version **10.0.17134.0**) + - "C++ ATL for v142 build tools (x86 and x64)" (also select ARM64 + if you'll be building for ARM64) + +Make sure you run Visual Studio once after installing, so it finishes +any first-run tasks and associates the installation with your account. + +If you already have Visual Studio 2019 installed, you can get to the +installer via its menu "Tools" → "Get Tools and Features". + +Other Required Tools +~~~~~~~~~~~~~~~~~~~~ + +MozillaBuild +^^^^^^^^^^^^ + +Finally, download the `MozillaBuild +Package <https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe>`__ +from Mozilla. Accept the default settings, in particular the default +installation directory: ``c:\mozilla-build\``. On some versions of +Windows an error dialog will give you the option to ‘reinstall with the +correct settings’ - you should agree and proceed. + +Once this is done, creating a shortcut to +``c:\mozilla-build\start-shell.bat`` on your desktop will make your life +easier. + +Getting the source +~~~~~~~~~~~~~~~~~~ + +This is the last big step. Double-clicking **start-shell.bat** in +``c:\mozilla-build`` (or the shortcut you’ve created above) will open a +terminal window. + +Start by creating a "mozilla-source" directory off of ``C:\`` and cd +into it like so + +.. code-block:: shell + + cd c:/ + + mkdir mozilla-source + + cd mozilla-source + +Next, you can get the Firefox source code with Mercurial. There is a +wiki page on how to :ref:`get the source using +Mercurial <Mercurial Overview>` +but, in summary, if your network connection is good enough to download +1+ GB without interuption and you want the main development repository, +then you can just use: + +.. code-block:: shell + + curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -o bootstrap.py + + python3 bootstrap.py + +... and follow the prompts. This will use mercurial to checkout the +source code. If you prefer to work with git, use this command instead (you'll +have to have `Git for Windows <https://git-scm.com/download/win>`_ installed): + +.. code-block:: shell + + python3 bootstrap.py --vcs=git + +While you’re waiting for that process to finish, take a look at `our +Mercurial +documentation <http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html>`_. +It explains how we use version control at Mozilla to manage our code and +land changes to our source tree. + +Build Firefox! +~~~~~~~~~~~~~~ + +Now we tie it all together. In your terminal window, ``cd`` to your +source directory as before and type + +.. code-block:: shell + + cd mozilla-unified # ... or the name of the repo you chose earlier + + ./mach bootstrap + + ./mach build + +The ``./mach bootstrap`` step is a catch-all for any dependencies not +covered in this documentation. Note that, bootstrap works **only with +the Mercurial repo of the source**, not with source tar balls, nor the +github mirror. If you are working on Firefox or Firefox for Android +frontends or building Firefox without any changes, select :ref:`Artifact Builds +<Understanding Artifact Builds>` in +the first question in ``./mach bootstrap``. Artifact builds will +complete more quickly! Artifact builds are unsuitable for those working +on C++ or Rust code. + +You’re on your way. Don’t be discouraged if this takes a while; it takes +some time even on the fastest modern machines and as much as two hours +or more on older hardware. Firefox is pretty big, because the Web is +big. + + +You're ready +~~~~~~~~~~~~ + +When mach build completes, you'll have your own version of Firefox built +from the source code on your hard drive, ready to run. You can run it +with + +.. code-block:: shell + + ./mach run + +Now you have your own home-built version of Firefox. + +If you saw an error here, look further down in this document for the +"Troubleshooting" section - some antivirus software quarantine some of +our tests, so you need to create exceptions for the "mozilla-source" and +"mozilla-build" directories. Don't turn your antivirus off! Just add the +exceptions. + + +Details and troubleshooting +--------------------------- + +Hardware and software requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Firefox build process is both I/O and CPU-intensive, and can take a +long time to build even on modern hardware. The minimum and recommended +hardware requirements for Mozilla development are: + +- At least 4 GB of RAM. 8 GB or more is recommended, and more is always + better. +- 35 GB free disk space. This amount of disk space accommodates Visual + Studio 2019 Community Edition, the required SDKs, the MozillaBuild + package, the Mercurial source repository and enough free disk space + to compile. A solid-state hard disk is recommended as the Firefox + build process is I/O-intensive. +- A 64-bit version of Windows 7 (Service Pack 1) or later. You can + still build 32-bit Firefox on a 64-bit Windows installation. + +Overview +~~~~~~~~ + +The Mozilla build process requires many tools that are not pre-installed +on most Windows systems. In addition to Visual Studio, install +MozillaBuild - a software bundle that includes the required versions of +bash, GNU make, Mercurial, and much more. + +Firefox 61+ require Visual Studio 2017 Update 6 or newer to build. + +Firefox 48 to 60 build with Visual Studio 2015. Visual Studio 2017 also +works for building Firefox 58 or newer. + +Firefox 37 through to 47 build with Visual Studio 2013 (VC12) and +possibly Visual Studio 2015 (although Visual Studio 2015 may not build +every revision). + +Earlier versions of Firefox build with older versions of Visual Studio. + +Installing the build prerequisites +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Complete each of these steps otherwise, you may not be able to build +successfully. There are notes on these software requirements below. + +#. Make sure your system is up-to-date through Windows Update. +#. Install `Visual Studio Community + 2019 <https://www.visualstudio.com/downloads/>`_ (free). + Alternatively, you can also use a paid version of Visual Studio. Some + additional components may be required to build Firefox, as noted in + the "Visual Studio 2019" section above. Earlier versions of Visual + Studio are not supported; the Firefox codebase relies on C++ features + that are not supported in earlier releases. +#. Optionally, in addition to VS2019, you may want to install `Visual + C++ 2008 Express <http://go.microsoft.com/?linkid=7729279>`_ (free) + to compile some Python extensions used in the build system as Python + 2.7.x for Windows is built with that compiler by default. Note, if + you want to use "mach resource-usage", "mach doctor", "mach + android-emulator", or run talos tests locally, you should install it + for building psutil. +#. Download and install the + `MozillaBuild <https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe>`__ + package, containing additional build tools. If you have Cygwin + installed, read the note in the tips section. If you see a Windows + error dialog giving you the option to re-install with the 'correct + settings', after the MozillaBuild's installer exits, choose the + option and after that all should be well. More information about + MozillaBuild and links to newer versions are available at + https://wiki.mozilla.org/MozillaBuild. + +Troubleshooting +~~~~~~~~~~~~~~~ + +In some circumstances, the following problems can arise: + +Antivirus performance +^^^^^^^^^^^^^^^^^^^^^ + +- Windows Defender and some scanning antivirus products are known to + have a major impact on build times. For example, if you have cloned + ``mozilla-unified`` successfully but ``./mach build`` fails, reporting + a missing file, you are likely experiencing this problem. Our + regression tests, for well-known security bugs, can include code + samples that some antivirus software will identify as a threat, and + will either quarantine or otherwise corrupt the files involved. To + resolve this you will need to add your source and object directories + (the ``mozilla-source`` and ``mozilla-build`` + directories) to the + `exclusion list in Windows Defender <https://support.microsoft.com/en-ca/help/4028485/windows-10-add-an-exclusion-to-windows-security>`_ + or your antivirus software. If you are missing files, revert your source + tree with the ``hg update -C`` command. Once this is done your next + ``./mach build`` should complete successfully. + +Installing Visual Studio in a different language than the system can cause issues +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- For example, having Visual Studio in French when the system is in + English causes the build to spew a lot of include errors and finishes + with a link error. + +.. note:: + + **Note:** **Mozilla will not build** if the path to the installation + tool folders contains **spaces** or other breaking characters such as + pluses, quotation marks, or metacharacters. The Visual Studio tools and + SDKs are an exception - they may be installed in a directory which + contains spaces. It is strongly recommended that you accept the default + settings for all installation locations. + +MozillaBuild package +~~~~~~~~~~~~~~~~~~~~ + +The MozillaBuild package contains other software prerequisites necessary +for building Mozilla, including the MSYS build environment, +`Mercurial <https://www.mercurial-scm.org/>`_, CVS, Python, YASM, NSIS, and UPX, +as well as optional but useful tools such as wget and emacs. + +`Download the current MozillaBuild +package. <https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe>`_ + +By default, the package installs to ``c:\mozilla-build`` and it is +recommended to use the default path. Don't use a path that contains +spaces. The installer does not modify the Windows registry. Note that +some binaries may require `Visual C++ Redistributable +package <https://www.microsoft.com/downloads/en/details.aspx?FamilyID=a5c84275-3b97-4ab7-a40d-3802b2af5fc2&displaylang=en>`_ to +run. + +.. note:: + + **MozillaBuild command prompt expectation setting:** Note that the + "UNIX-like" environment provided by MozillaBuild is only really useful + for building and committing to the Mozilla source. Most command line + tools you would expect in a modern Linux distribution are not present, + and those tools that are provided can be as much as a decade or so old + (especially those provided by MSYS). It's the old tools in particular + that can cause problems since they often don't behave as expected, are + buggy, or don't support command line arguments that have been taken for + granted for years. For example, copying a source tree using + ``cp -rf src1 src2`` does not work correctly because of an old version + of cp (it gives "cp: will not create hard link" errors for some files). + In short, MozillaBuild supports essential developer interactions with + the Mozilla code, but beyond that don't be surprised if it trips you up + in all sorts of exciting and unexpected ways. + +Opening a MozillaBuild command prompt +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After the prerequisites are installed, launch +the ``start-shell.bat`` batch file using the Windows command +prompt in the directory to which you installed MozillaBuild +(``c:\mozilla-build`` by default). This will launch an MSYS/BASH command +prompt properly configured to build Firefox. All further commands should +be executed in this command prompt window. (Note that this is not the +same as what you get with the Windows CMD.EXE shell.) + +.. note:: + + Note: This is not the same as what you get with the Windows CMD.EXE + shell. + +Create a directory for the source +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Note:** You won't be able to build the Firefox source code if it's +under a directory with spaces in the path such as "Documents and +Settings". You can pick any other location, such as a new directory +c:/mozilla-source or c:/thunderbird-src. The build command prompt also +tolerates "c:\\" and "/c/", but the former gives confusion in the +Windows command prompt, and the latter is misinterpreted by some tools +(at least MOZ\_OBJDIR). The "C:/" syntax helps draw attention that the +**MozillaBuild** command prompt is assumed from here on out since it +provides configured environment and tools. + + +It's a sensible idea to create a new shallow directory, like +"c:/mozilla-source" dedicated solely to the +code: + +.. code-block:: shell + + cd c:/; mkdir mozilla-source; cd mozilla-source + +Keeping in mind the diagnostic hints below should you have issues. You +are now ready to get the Firefox source and build. + +Command prompt tips and caveats +------------------------------- + +- To paste into this window, you must right-click on the window's title + bar, move your cursor to the “Edit” menu, and click “Paste”. You can + also set “Quick Edit Mode” in the “Properties” menu and right-click + the window to paste your selection. +- If you have Cygwin installed, make sure that the MozillaBuild + directories come before any Cygwin directories in the search path + enhanced by ``start-shell-msvc2015.bat`` (use ``echo $PATH`` to see + your search path). +- In the MSYS / BASH shell started by ``start-shell-msvc2015.bat``, + UNIX-style forward slashes (/) are used as path separators instead of + the Windows-style backward slashes (\\). So if you want to change to + the directory ``c:\mydir``, in the MSYS shell to improve clarity, you + would use ``cd /c/mydir ``though both ``c:\mydir`` and ``c:/mydir`` + are supported. +- The MSYS root directory is located in ``/c/mozilla-build/msys`` if + you used the default installation directory. It's a good idea not to + build anything under this directory. Instead use something like + ``/c/mydir``. + +Common problems, hints, and restrictions +---------------------------------------- + +- :ref:`Debugging Firefox on Windows FAQ <Debugging On Windows>`: + Tips on how to debug Mozilla on Windows. +- Your installed MozillaBuild may be too old. The build system may + assume you have new features and bugfixes that are only present in + newer versions of MozillaBuild. Instructions for how to update + MozillaBuild `can be found + here <https://wiki.mozilla.org/MozillaBuild>`_. +- If the bootstrapping script ``bootstrap.py`` fails, you can also try running + ``hg clone https://hg.mozilla.org/mozilla-unified`` followed by + ``cd mozilla-unified; ./mach bootstrap`` yourself. +- The build may fail if your machine is configured with the wrong + architecture. If you want to build 64-bit Firefox, add the two lines + below to your mozconfig file: + +.. code-block:: shell + + ac_add_options --target=x86_64-pc-mingw32 + ac_add_options --host=x86_64-pc-mingw32 + +- The build may fail if your ``PATH`` environment variable contains + quotation marks("). Quotes are not properly translated when passed + down to MozillaBuild sub-shells and they are usually not needed so + they can be removed. +- The build may fail if you have a ``PYTHON`` environment variable set. + It displays an error almost immediately that says + "``The system cannot find the file specified``." Typing + "``unset PYTHON``" before running the Mozilla build tools in the same + command shell should fix this. Make sure that ``PYTHON`` is unset, + rather than set to an empty value. +- The build may fail if you have Cygwin installed. Make sure that the + MozillaBuild directories (``/c/mozilla-build`` and subdirectories) + come before any Cygwin directories in your PATH environment variable. + If this does not help, remove the Cygwin directories from PATH, or + try building on a clean PC with no Cygwin. +- Building with versions of NSIS other than the version that comes with + the latest supported version of MozillaBuild is not supported and + will likely fail. +- If you intend to distribute your build to others, set + ``WIN32_REDIST_DIR=$VCINSTALLDIR\redist\x86\Microsoft.VC141.CRT`` in + your mozconfig to get the Microsoft CRT DLLs packaged along with the + application. Note the exact .CRT file may depend on your Visual + Studio version. +- The Microsoft Antimalware service can interfere with compilation, + often manifesting as an error related to ``conftest.exe`` during + build. To remedy this, add at your object directory at least to the + exclusion settings. +- Errors like "second C linkage of overloaded function + '\_interlockedbittestandset' not allowed", are encountered when + intrin.h and windows.h are included together. Use a *#define* to + redefine one instance of the function's name. +- Parallel builds (``-jN``) do not work with GNU makes on Windows. You + should use the ``mozmake`` command included with current versions of + MozillaBuild. Building with the ``mach`` command will always use the + best available make command. +- If you encounter a build failure like "ERROR: Cannot find + makecab.exe", try applying the patch from `bug + 1383578 <https://bugzilla.mozilla.org/show_bug.cgi?id=1383578>`_, + i.e. change: ``SET PATH="%PATH%;!LLVMDIR!\bin"`` to + ``SET "PATH=%PATH%;!LLVMDIR!\bin"``. +- If you encounter a build failure with + ``LINK: fatal error LNK1181: cannot open input file ..\..\..\..\..\security\nss3.lib``, + it may be related to your clone of ``mozilla-unified`` being located + in the Users folder (possibly encrypted). Try moving it outside of + the Users folder. The docs recommend + ``C:\mozilla-source\mozilla-unified`` which should work. +- If you encounter a build failure with + ``ERROR: GetShortPathName returned a long path name.``.You need + create a 8dot3name short name for the path which has space.For + example : fsutil file setshortname "C:\\Program Files (x86)" + PROGRA~2. If you got "access denied", try to restart your computer + to safe mode and try again. + diff --git a/docs/testing-rust-code/index.md b/docs/testing-rust-code/index.md new file mode 100644 index 0000000000..d7fb87ada9 --- /dev/null +++ b/docs/testing-rust-code/index.md @@ -0,0 +1,123 @@ +# Testing & Debugging Rust Code + +This page explains how to test and debug Rust code in Firefox. + +The [build documentation](../build/buildsystem/rust.html) explains how to add +new Rust code to Firefox. The [code documentation](../writing-rust-code) +explains how to write and work with Rust code in Firefox. + +## Testing Mozilla crates + +Rust code will naturally be tested as part of system tests such as Mochitests. +This section describes the two methods for unit testing of individual Rust +crates. Which method should be used depends on the circumstances. + +### Rust tests + +If a Mozilla crate has "normal" Rust tests (i.e. `#[test]` functions that run +with `cargo test`), you can add the crate's name to `RUST_TESTS` in +[toolkit/library/rust/moz.build](https://searchfox.org/mozilla-central/source/toolkit/library/rust/moz.build). +(Cargo features can be activated for Rust tests by adding them to +`RUST_TEST_FEATURES` in the same file.) + +Rust tests are run with `./mach rusttests`. They run on automation in a couple +of `rusttests` jobs, but not on all platforms. + +Rust tests have one major restriction: they cannot link against Gecko symbols. +Therefore, Rust tests cannot be used for crates that use Gecko crates like +`nsstring` and `xpcom`. + +It's also possible to use `RUST_TESTS` in a different `moz.build` file. See +`testing/geckodriver/moz.build` and the [geckodriver testing docs] for an +example. + +[geckodriver testing docs]: ../testing/geckodriver/Testing.html + +### GTests + +Another way to unit test a Mozilla crate is by writing a GTest that uses FFI to +call into Rust code. This requires the following steps. +- Create a new test crate whose name is the same as the name of crate being + tested, with a `-gtest` suffix. +- Add to the test crate a Rust file, a C++ file containing GTest `TEST()` + functions that use FFI to call into the Rust file, a `Cargo.toml` file that + references the Rust file, and a `moz.build` file that references the C++ + file. +- Add an entry to the `[dependencies]` section in + [toolkit/library/gtest/rust/Cargo.toml](https://searchfox.org/mozilla-central/source/toolkit/library/gtest/rust/Cargo.toml). +- Add an `extern crate` entry to + [toolkit/library/gtest/rust/lib.rs](https://searchfox.org/mozilla-central/source/toolkit/library/gtest/rust/lib.rs). + +See +[xpcom/rust/gtest/nsstring/](https://searchfox.org/mozilla-central/source/xpcom/rust/gtest/nsstring) +for a simple example. (Note that the `moz.build` file is in the parent +directory for that crate.) + +A Rust GTest can be run like any other GTest via `./mach gtest`, using the C++ +`TEST()` functions as the starting point. + +Unlike Rust tests, GTests can be used when linking against Gecko symbols is required. + +## Testing third-party crates + +In general we don't run tests for third-party crates. The assumption is that +these crates are sufficiently well-tested elsewhere. + +## Debugging Rust code + +In theory, Rust code is debuggable much like C++ code, using standard tools +like `gdb`, `rr`, and the Microsoft Visual Studio Debugger. In practice, the +experience can be worse, because shortcomings such as the following can occur. +- Inability to print local variables, even in non-optimized builds. +- Inability to call generic functions. +- Missing line numbers and stack frames. +- Printing of basic types such as `Option` and `Vec` is sometimes sub-optimal. + If you see a warning "Missing auto-load script at offset 0 in section + `.debug_gdb_scripts`" when starting `gdb`, the `rust-gdb` wrapper may give + better results. + +## Logging from Rust code + +### Rust logging + +The `RUST_LOG` environment variable (from the `env_logger` crate) can be used +to enable logging to stderr from Rust code in Firefox. The logging macros from +the `log` crate can be used. In order of importance, they are: `error!`, +`warn!`, `info!`, `debug!`, `trace!`. + +For example, to show all log messages of `info` level or higher, run: +``` +RUST_LOG=info firefox +``` +Module-level logging can also be specified, see the [documentation] for the +`env_logger` crate for details. + +To restrict logging to child processes, use `RUST_LOG_CHILD` instead of +`RUST_LOG`. + +[documentation]: https://docs.rs/env_logger/ + +### Gecko logging + +Rust logging can also be forwarded to the [Gecko logger] for capture via +`MOZ_LOG` and `MOZ_LOG_FILE`. + +[Gecko logger]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Gecko_Logging + +- When parsing modules from `MOZ_LOG`, modules containing `::` are considered + to be Rust modules. To log everything in a top-level module like + `neqo_transport`, specify it as `neqo_transport::*`. For example: +``` +MOZ_LOG=timestamp,sync,nsHostResolver:5,neqo_transport::*:5,proxy:5 firefox +``` +- When logging from a submodule the `::*` is allowed but isn't necessary. + So these two lines are equivalent: +``` +MOZ_LOG=timestamp,sync,neqo_transport::recovery:5 firefox +MOZ_LOG=timestamp,sync,neqo_transport::recovery::*:5 firefox +``` +- `debug!` and `trace!` logs will not appear in non-debug builds. This is due + to our use of the `release_max_level_info` feature in the `log` crate. + +- When using both `MOZ_LOG` and `RUST_LOG`, modules that are specified in + `MOZ_LOG` will not appear in `RUST_LOG`. diff --git a/docs/writing-rust-code/basics.md b/docs/writing-rust-code/basics.md new file mode 100644 index 0000000000..1d3732de4e --- /dev/null +++ b/docs/writing-rust-code/basics.md @@ -0,0 +1,83 @@ +# Basics + +## Formatting Rust code + +To format all the Rust code within a directory `$DIR`, run: +``` +./mach lint -l rustfmt --fix $DIR +``` + +## Using Cargo + +Many Cargo commands can be run on individual crates. Change into the directory +containing the crate's `Cargo.toml` file, and then run the command with +`MOZ_TOPOBJDIR` set appropriately. For example, to generate and view rustdocs +for the `xpcom` crate, run these commands: + +``` +cd xpcom/rust/xpcom +MOZ_TOPOBJDIR=$OBJDIR cargo doc +cd - +firefox target/doc/xpcom/index.html +``` +where `$OBJDIR` is the path to the object directory. + +## Using static prefs + +Static boolean/integer prefs can be easily accessed from Rust code. Add a +`rust: true` field to the pref definition in +[modules/libpref/init/StaticPrefList.yaml](https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml), +like this: +```yaml +- name: my.lucky.pref + type: RelaxedAtomicBool + value: true + mirror: always + rust: true +``` +The pref can then be accessed via the `pref!` macro, like this: +``` +let my_lucky_pref = static_prefs::pref!("my.lucky.pref"); +``` + +## Helper crates + +The following in-tree helper crates provide idiomatic support for some common patterns. +- [nserror](https://searchfox.org/mozilla-central/source/xpcom/rust/nserror/src/lib.rs) +reflects `nsresult` codes into Rust. +- [nsstring](https://searchfox.org/mozilla-central/source/xpcom/rust/nsstring/src/lib.rs) + exposes bindings for XPCOM string types. You can use the same `ns{A,C}String` + types as C++ for owned strings and pass them back and forth over the + boundary. There is also `ns{A,C}Str` for dependent or borrowed strings. +- [xpcom](https://searchfox.org/mozilla-central/source/xpcom/rust/xpcom/src) + provides multiple building blocks for a component's implementation. + - The `RefPtr` type is for managing reference-counted pointers. + - XPCOM service getters are generated by + [xpcom/build/Services.py](https://searchfox.org/mozilla-central/source/xpcom/build/Services.py) + and can be called like this: + ``` + let pref_service = xpcom::services::get_PrefService(); + ``` + - There is also a `get_service` function that works like `do_GetService` in + C++, as an alternative. + - A set of `derive` macros help with declaring interface implementations. The + [docs](https://searchfox.org/mozilla-central/source/xpcom/rust/xpcom/xpcom_macros/src/lib.rs) + have details and examples. +- [moz_task](https://searchfox.org/mozilla-central/source/xpcom/rust/moz_task/src/lib.rs) + wraps XPCOM's threading functions in order to make it easy and safe to write + threaded code. It has helpers for getting and creating threads, dispatching + async runnables, and thread-safe handles. +- [storage](https://searchfox.org/mozilla-central/source/storage/rust/src/lib.rs) + is an interface to mozStorage, our wrapper for SQLite. It can wrap an + existing storage connection, and prepare and execute statements. This crate + wraps the synchronous connection API, and lets you execute statements + asynchronously via `moz_task`. +- [storage_variant](https://searchfox.org/mozilla-central/source/storage/variant/src/lib.rs) + is for working with variants. It also provides a `HashPropertyBag` type + that's useful for passing hash maps over XPCOM to JS. + +Unfortunately, rustdocs are [not yet generated and +hosted](https://bugzilla.mozilla.org/show_bug.cgi?id=1428139) for crates within +mozilla-central. Therefore, the crate links shown above link to files +containing the relevant rustdocs source where possible. However, you can +generate docs locally using the `cargo doc` command described above. diff --git a/docs/writing-rust-code/ffi.md b/docs/writing-rust-code/ffi.md new file mode 100644 index 0000000000..f10fad4221 --- /dev/null +++ b/docs/writing-rust-code/ffi.md @@ -0,0 +1,240 @@ +# Rust/C++ interop + +This document describes how to use FFI in Firefox to get Rust code and C++ code to interoperate. + +## Transferable types + +Generally speaking, the more complicated is the data you want to transfer, the +harder it'll be to transfer across the FFI boundary. + +Booleans, integers, and pointers cause little trouble. +- C++ `bool` matches Rust `bool` +- C++ `uint8_t` matches Rust `u8`, `int32_t` matches Rust `i32`, etc. +- C++ `const T*` matches Rust `*const T`, `T*` matches Rust `*mut T`. + +Lists are handled by C++ `nsTArray` and Rust `ThinVec`. + +For strings, it is best to use the `nsstring` helper crate. Using a raw pointer +plus length is also possible for strings, but more error-prone. + +If you need a hashmap, you'll likely want to decompose it into two lists (keys +and values) and transfer them separately. + +Other types can be handled with tools that generate bindings, as the following +sections describe. + +## Accessing C++ code and data from Rust + +To call a C++ function from Rust requires adding a function declaration to Rust. +For example, for this C++ function: + +``` +extern "C" { +bool UniquelyNamedFunction(const nsCString* aInput, nsCString* aRetVal) { + return true; +} +} +``` +add this declaration to the Rust code: +```rust +extern "C" { + pub fn UniquelyNamedFunction(input: &nsCString, ret_val: &mut nsCString) -> bool; +} +``` + +Rust code can now call `UniquelyNamedFunction()` within an `unsafe` block. Note +that if the declarations do not match (e.g. because the C++ function signature +changes without the Rust declaration being updated) crashes are likely. (Hence +the `unsafe` block.) + +Because of this unsafety, for non-trivial interfaces (in particular when C++ +structs and classes must be accessed from Rust code) it's common to use +[rust-bindgen](https://github.com/rust-lang/rust-bindgen), which generates Rust +bindings. The documentation is +[here](https://rust-lang.github.io/rust-bindgen/). + +## Accessing Rust code and data from C++ + +A common option for accessing Rust code and data from C++ is to use +[cbindgen](https://github.com/eqrion/cbindgen), which generates C++ header +files. for Rust crates that expose a public C API. cbindgen is a very powerful +tool, and this section only covers some basic uses of it. + +### Basics + +First, add suitable definitions to your Rust. `#[no_mangle]` and `extern "C"` +are required. + +```rust +#[no_mangle] +pub unsafe extern "C" fn unic_langid_canonicalize( + langid: &nsCString, + ret_val: &mut nsCString +) -> bool { + ret_val.assign("new value"); + true +} +``` + +Then, add a `cbindgen.toml` file in the root of your crate. It may look like this: + +```toml +header = """/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */""" +autogen_warning = """/* DO NOT MODIFY THIS MANUALLY! This file was generated using cbindgen. See RunCbindgen.py */ +#ifndef mozilla_intl_locale_MozLocaleBindings_h +#error "Don't include this file directly, instead include MozLocaleBindings.h" +#endif +""" +include_version = true +braces = "SameLine" +line_length = 100 +tab_width = 2 +language = "C++" +# Put FFI calls in the `mozilla::intl::ffi` namespace. +namespaces = ["mozilla", "intl", "ffi"] + +# Export `ThinVec` references as `nsTArray`. +[export.rename] +"ThinVec" = "nsTArray" +``` + +Next, extend the relevant `moz.build` file to invoke cbindgen. + +```python +if CONFIG['COMPILE_ENVIRONMENT']: + CbindgenHeader('unic_langid_ffi_generated.h', + inputs=['/intl/locale/rust/unic-langid-ffi']) + + EXPORTS.mozilla.intl += [ + '!unic_langid_ffi_generated.h', + ] +``` + +This tells the build system to run cbindgen on +`intl/locale/rust/unic-langid-ffi` to generate `unic_langid_ffi_generated.h`, +which will be placed in `$OBJDIR/dist/include/mozilla/intl/`. + +Finally, include the generated header into a C++ file and call the function. + +```c++ +#include "mozilla/intl/unic_langid_ffi_generated.h" + +using namespace mozilla::intl::ffi; + +void Locale::MyFunction(nsCString& aInput) const { + nsCString result; + unic_langid_canonicalize(aInput, &result); +} +``` + +### Complex types + +Many complex Rust types can be exposed to C++, and cbindgen will generate +appropriate bindings for all `pub` types. For example: + +```rust +#[repr(C)] +pub enum FluentPlatform { + Linux, + Windows, + Macos, + Android, + Other, +} + +extern "C" { + pub fn FluentBuiltInGetPlatform() -> FluentPlatform; +} +``` + +```c++ +ffi::FluentPlatform FluentBuiltInGetPlatform() { + return ffi::FluentPlatform::Linux; +} +``` + +For an example using cbindgen to expose much more complex Rust types to C++, +see [this blog post]. + +[this blog post]: https://crisal.io/words/2020/02/28/C++-rust-ffi-patterns-1-complex-data-structures.html + +### Instances + +If you need to create and destroy a Rust struct from C++ code, the following +example may be helpful. + +First, define constructor, destructor and getter functions in Rust. (C++ +declarations for these will be generated by cbindgen.) + +```rust +#[no_mangle] +pub unsafe extern "C" fn unic_langid_new() -> *mut LanguageIdentifier { + let langid = LanguageIdentifier::default(); + Box::into_raw(Box::new(langid)) +} + +#[no_mangle] +pub unsafe extern "C" fn unic_langid_destroy(langid: *mut LanguageIdentifier) { + drop(Box::from_raw(langid)); +} + +#[no_mangle] +pub unsafe extern "C" fn unic_langid_as_string( + langid: &mut LanguageIdentifier, + ret_val: &mut nsACString, +) { + ret_val.assign(&langid.to_string()); +} +``` + +Next, in a C++ header define a destructor via `DefaultDelete`. + +```c++ +#include "mozilla/intl/unic_langid_ffi_generated.h" +#include "mozilla/UniquePtr.h" + +namespace mozilla { + +template <> +class DefaultDelete<intl::ffi::LanguageIdentifier> { + public: + void operator()(intl::ffi::LanguageIdentifier* aPtr) const { + unic_langid_destroy(aPtr); + } +}; + +} // namespace mozilla +``` + +(This definition must be visible any place where +`UniquePtr<intl::ffi::LanguageIdentifier>` is used, otherwise C++ will try to +free the code, which might lead to strange behaviour!) + +Finally, implement the class. + +```c++ +class Locale { +public: + explicit Locale(const nsACString& aLocale) + : mRaw(unic_langid_new()) {} + + const nsCString Locale::AsString() const { + nsCString tag; + unic_langid_as_string(mRaw.get(), &tag); + return tag; + } + +private: + UniquePtr<ffi::LanguageIdentifier> mRaw; +} +``` + +This makes it possible to instantiate a `Locale` object and call `AsString()`, +all from C++ code. + +## Other examples + +For a detailed explanation of an interface in Firefox that doesn't use cbindgen +or rust-bindgen, see [this blog post](https://hsivonen.fi/modern-cpp-in-rust/). diff --git a/docs/writing-rust-code/index.md b/docs/writing-rust-code/index.md new file mode 100644 index 0000000000..f8a6c2ad74 --- /dev/null +++ b/docs/writing-rust-code/index.md @@ -0,0 +1,17 @@ +# Writing Rust Code + +This page explains how to write and work with Rust code in Firefox, with an +emphasis on interoperation with C++ code. + +The [build documentation](../build/buildsystem/rust.html) explains how to add +new Rust code to Firefox. The [test documentation](../testing-rust-code) +explains how to test and debug Rust code in Firefox. + +```eval_rst +.. toctree:: + :titlesonly: + :maxdepth: 1 + :glob: + + * +``` diff --git a/docs/writing-rust-code/xpcom.md b/docs/writing-rust-code/xpcom.md new file mode 100644 index 0000000000..bf900e03f8 --- /dev/null +++ b/docs/writing-rust-code/xpcom.md @@ -0,0 +1,123 @@ +# XPCOM components in Rust + +XPCOM components can be written in Rust. + +## A tiny example + +The following example shows a new type that implements `nsIObserver`. + +First, create a new empty crate (e.g. with `cargo init --lib`), and add the +following dependencies in its `Cargo.toml` file. + +```toml +[dependencies] +libc = "0.2" +nserror = { path = "../../../xpcom/rust/nserror" } +nsstring = { path = "../../../xpcom/rust/nsstring" } +xpcom = { path = "../../../xpcom/rust/xpcom" } +``` + +(The number of `../` occurrences will depend on the depth of the crate in the +file hierarchy.) + +Next hook it into the build system according to the [build +documentation](../build/buildsystem/rust.html). + +The Rust code will need to import some basic types. `xpcom::interfaces` +contains all the usual `nsI` interfaces. + +```rust +use libc::c_char; +use nserror::nsresult; +use std::sync::atomic::{AtomicBool, Ordering}; +use xpcom::{interfaces::nsISupports, RefPtr}; +``` + +The next part declares the implementation. + +```rust +#[derive(xpcom)] +#[xpimplements(nsIObserver)] +#[refcnt = "atomic"] +struct InitMyObserver { + ran: AtomicBool, +} +``` + +It defines an initializer struct, prefixed with `Init`, with three attributes. +- Some `derive` magic. +- An `xpimplements` declaration naming the interface(s) being implemented. +- The reference count type. + +Next, all interface methods are declared in the `impl` block as `unsafe` methods. + +```rust +impl MyObserver { + #[allow(non_snake_case)] + unsafe fn Observe( + &self, + _subject: *const nsISupports, + _topic: *const c_char, + _data: *const i16, + ) -> nsresult { + self.ran.store(true, Ordering::SeqCst); + nserror::NS_OK + } +} +``` + +These methods always take `&self`, not `&mut self`, so we need to use interior +mutability: `AtomicBool`, `RefCell`, `Cell`, etc. + +XPCOM methods are unsafe by default, but the +[xpcom_method!](https://searchfox.org/mozilla-central/source/xpcom/rust/xpcom/src/method.rs) +macro can be used to clean this up. It also takes care of null-checking and +hiding pointers behind references, lets you return a `Result` instead of an +`nsresult,` and so on. + +To use this type within Rust code, do something like the following. + +```rust +let observer = MyObserver::allocate(InitMyObserver { + ran: AtomicBool::new(false), +}); +let rv = unsafe { + observer.Observe(x.coerce(), + cstr!("some-topic").as_ptr(), + ptr::null()) +}; +assert!(rv.succeeded()); +``` + +The implementation has an (auto-generated) `allocate` method that takes in an +initialization struct, and returns a `RefPtr` to the instance. + +`coerce` casts any XPCOM object to one of its base interfaces; in this case, +the base interface is `nsISupports`. In C++, this would be handled +automatically through inheritance, but Rust doesn’t have inheritance, so the +conversion must be explicit. + +## Bigger examples + +The following XPCOM components are written in Rust. +- [kvstore](https://searchfox.org/mozilla-central/source/toolkit/components/kvstore), + which exposes the LMDB key-value store (via the [Rkv + library](https://docs.rs/rkv)) The API is asynchronous, using `moz_task` to + schedule all I/O on a background thread, and supports getting, setting, and + iterating over keys. +- [cert_storage](https://searchfox.org/mozilla-central/source/security/manager/ssl/cert_storage), + which stores lists of [revoked intermediate certificates](https://blog.mozilla.org/security/2015/03/03/revoking-intermediate-certificates-introducing-onecrl/). +- [bookmark_sync](https://searchfox.org/mozilla-central/source/toolkit/components/places/bookmark_sync), + which [merges](https://mozilla.github.io/dogear) bookmarks from Firefox Sync + with bookmarks in the Places database. +- [webext_storage_bridge](https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge), + which powers the WebExtension storage.sync API. It's a self-contained example + that pulls in a crate from application-services for the heavy lifting, wraps + that up in a Rust XPCOM component, and then wraps the component in a JS + interface. There's also some boilerplate there around adding a + `components.conf` file, and a dummy C++ header that declares the component + constructor. +- [firefox-accounts-bridge](https://searchfox.org/mozilla-central/source/services/fxaccounts/rust-bridge/firefox-accounts-bridge), + which wraps the Rust Firefox Accounts client with which we eventually want to + replace our creaky JS implementation. It has a lot of the same patterns as + webext_storage_bridge. |
