From a90a5cba08fdf6c0ceb95101c275108a152a3aed Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Wed, 12 Jun 2024 07:35:37 +0200 Subject: Merging upstream version 127.0. Signed-off-by: Daniel Baumann --- docs/code-quality/lint/create.rst | 2 +- docs/code-quality/lint/linters/condprof-addons.rst | 10 +-- docs/code-quality/lint/linters/perfdocs.rst | 71 ++++++++++++++++++---- docs/code-quality/static-analysis/existing.rst | 2 +- docs/conf.py | 2 + docs/config.yml | 3 +- docs/contributing/contribution_quickref.rst | 2 +- docs/contributing/data-review.rst | 41 +++++++++++++ docs/contributing/how_to_submit_a_patch.rst | 10 +-- docs/contributing/index.rst | 1 + docs/contributing/reviews.rst | 53 ++++++++++++++-- docs/contributing/signing/signing_macos_build.rst | 4 +- docs/overview/ios.rst | 2 +- docs/setup/common_build_errors.md | 34 +++++++++++ docs/setup/index.rst | 1 + docs/setup/macos_build.rst | 2 +- docs/writing-rust-code/update-policy.md | 4 +- 17 files changed, 207 insertions(+), 37 deletions(-) create mode 100644 docs/contributing/data-review.rst create mode 100644 docs/setup/common_build_errors.md (limited to 'docs') diff --git a/docs/code-quality/lint/create.rst b/docs/code-quality/lint/create.rst index 4da59b83a0..f3afe59ab5 100644 --- a/docs/code-quality/lint/create.rst +++ b/docs/code-quality/lint/create.rst @@ -347,7 +347,7 @@ Adding the linter to the CI First, the job will have to be declared in Taskcluster. -This should be done in the `mozlint Taskcluster configuration `_. +This should be done in the `mozlint Taskcluster configuration `_. You will need to define a symbol, how it is executed and on what kind of change. For example, for ruff, the configuration is the following: diff --git a/docs/code-quality/lint/linters/condprof-addons.rst b/docs/code-quality/lint/linters/condprof-addons.rst index c302eef5be..ca9c869d09 100644 --- a/docs/code-quality/lint/linters/condprof-addons.rst +++ b/docs/code-quality/lint/linters/condprof-addons.rst @@ -5,7 +5,7 @@ CondProf Addons is a linter for condprof customization JSON files (see :searchfo it reports linting errors if: - any of the addons required by the customization files (e.g. see :searchfox:`testing/condprofile/condprof/customization/webext.json`) - is not found in the tar file fetched through the `firefox-addons` fetch task (see :searchfox:`taskcluster/ci/fetch/browsertime.yml`) + is not found in the tar file fetched through the `firefox-addons` fetch task (see :searchfox:`taskcluster/kinds/fetch/browsertime.yml`) - or the expected `firefox-addons` fetch task has not been found Run Locally @@ -37,7 +37,7 @@ XPI file is missing from the firefox-addons.tar archive This linting errors is expected to be reported if the linter detected that a confprof customization file requires an addon but the related xpi filename is not included in the firefox-addons.tar file fetched -through the `firefox-addons` fetch task (see :searchfox:`taskcluster/ci/fetch/browsertime.yml`). +through the `firefox-addons` fetch task (see :searchfox:`taskcluster/kinds/fetch/browsertime.yml`). If the patch or phabricator revision is not meant to be landed, but only used as a temporary patch pushed on try or only applied locally (e.g. to run the tp6/tp6m webextensions perftests with a given @@ -50,14 +50,14 @@ the linting error have to be fixed before or along landing the change, either by - removing the addition to the customization file if it wasn't intended to include that addon to all runs of the tp6/tp6m webextensions perftests -- updating the `firefox-addons` fetch task as defined in :searchfox:`taskcluster/ci/fetch/browsertime.yml` +- updating the `firefox-addons` fetch task as defined in :searchfox:`taskcluster/kinds/fetch/browsertime.yml` by creating a pull request in the github repository where the asset is stored, and ask a review from a peer of the `#webextensions-reviewer` and `#perftests-reviewers` review groups. firefox-addons taskcluster config 'add-prefix' attribute should be set to 'firefox-addons/' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If this linting error is hit, then the `firefox-addons` task defined in :searchfox:`taskcluster/ci/fetch/browsertime.yml` +If this linting error is hit, then the `firefox-addons` task defined in :searchfox:`taskcluster/kinds/fetch/browsertime.yml` is missing the `add-prefix` attribute or its value is not set to the expected 'firefox-addons/' subdir name. This is enforced as a linting rule because when the condprof utility is going to build a conditioned profile @@ -69,7 +69,7 @@ names are already available in `$MOZ_FETCHES_DIR/firefox-addons`. firefox-addons taskcluser fetch config section not found ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This linting error is hit if the linter does not find the expected `firefox-addons` task defined in :searchfox:`taskcluster/ci/fetch/browsertime.yml` +This linting error is hit if the linter does not find the expected `firefox-addons` task defined in :searchfox:`taskcluster/kinds/fetch/browsertime.yml` or it is missing the expected `fetch` attribute. Configuration diff --git a/docs/code-quality/lint/linters/perfdocs.rst b/docs/code-quality/lint/linters/perfdocs.rst index 1ec2e33e68..9a5c6f1d2d 100644 --- a/docs/code-quality/lint/linters/perfdocs.rst +++ b/docs/code-quality/lint/linters/perfdocs.rst @@ -14,21 +14,42 @@ The mozlint integration of PerfDocs can be run using mach: $ mach lint --linter perfdocs . +Documentation can be regenerated for performance tests by including the ``--fix`` flag: + +.. parsed-literal:: + + $ mach lint --linter perfdocs . --fix + Configuration ------------- -There are no configuration options available for this linter. It scans the full source tree under ``testing``, looking for folders named ``perfdocs`` and then validates their content. This has only been implemented for Raptor so far, but Talos will be added in the future. We also hope to expand this to search outside the ``testing`` directory. +There are no configuration options available for this linter. It scans the full source tree under ``testing``, looking for folders named ``perfdocs``, validates their content, and regenerates the documentation (if ``--fix`` is provided). This has been implemented for all performance testing harnesses, and the documentation generated gets displayed in :ref:`Performance Testing`. -The ``perfdocs`` folders, there needs to be an ``index.rst`` file and it needs to contain the string ``{documentation}`` in some location in the file which is where the test documentation will be placed. The folders must also have a ``config.yml`` file following this schema: +In the ``perfdocs`` folders, there needs to be an ``index.rst`` file and it needs to contain the string ``{documentation}`` in some location in the file which is where the test documentation will be placed. The folders must also have a ``config.yml`` file following this schema: .. code-block:: python CONFIG_SCHEMA = { + "definitions": { + "metrics_schema": { + "metric_name": { + "type": "object", + "properties": { + "aliases": {"type": "array", "items": {"type": "string"}}, + "description": {"type": "string"}, + "matcher": {"type": "string"}, + }, + "required": ["description", "aliases"], + }, + }, + }, "type": "object", "properties": { "name": {"type": "string"}, "manifest": {"type": "string"}, + "static-only": {"type": "boolean"}, + "metrics": {"$ref": "#/definitions/metrics_schema"}, "suites": { "type": "object", "properties": { @@ -39,22 +60,17 @@ The ``perfdocs`` folders, there needs to be an ``index.rst`` file and it needs t "type": "object", "properties": { "test_name": {"type": "string"}, - } + }, }, "description": {"type": "string"}, + "owner": {"type": "string"}, }, - "required": [ - "description" - ] + "required": ["description"], } - } - } + }, + }, }, - "required": [ - "name", - "manifest", - "suites" - ] + "required": ["name", "manifest", "static-only", "suites"], } Here is an example of a configuration file for the Raptor framework: @@ -75,6 +91,35 @@ Here is an example of a configuration file for the Raptor framework: tests: wasm: "All wasm tests." +Metrics that produce alerts can also be documented like so: + +.. parsed-literal:: + + name: raptor + manifest: testing/raptor/raptor/raptor.toml + metrics: + "First Paint": + description: "The description of the metric." + aliases: + - fcp + - anAliasForFCP + # Optional regex to match the metrics found in the tests with this + # documented metric + matcher: f.* + suites: + desktop: + description: "Desktop tests." + tests: + raptor-tp6: "Raptor TP6 tests." + mobile: + description: "Mobile tests" + benchmarks: + description: "Benchmark tests." + tests: + wasm: "All wasm tests." + +The documented metrics must exist in the tests for the suite. If they are not, then validation will fail. The same is true if a metric in a test is not documented. Also, if ``metrics`` are defined, then a ``metrics.rst`` file is expected to be found in the ``perfdocs`` folder for the given suite. It must contain the string ``{metrics_documentation}`` where the documentation should be added. The ``metrics.rst`` is renamed ``{suite-name}-metrics.rst`` in the generated folder, so if it needs to be linked to in the ``index.rst`` file, it should contain a ``{metrics_rst_name}`` string for where the link should be added - it's expected to be found in a toctree section. + Note that there needs to be a FrameworkGatherer implemented for the framework being documented since each of them may have different ways of parsing test manifests for the tests. See `RaptorGatherer `_ for an example gatherer that was implemented for Raptor. Sources diff --git a/docs/code-quality/static-analysis/existing.rst b/docs/code-quality/static-analysis/existing.rst index 9b20650ad0..dd38051dc7 100644 --- a/docs/code-quality/static-analysis/existing.rst +++ b/docs/code-quality/static-analysis/existing.rst @@ -96,7 +96,7 @@ Regression Testing In order to prevent regressions in our clang-tidy based static analysis, we have created a -:searchfox:`task ` +:searchfox:`task ` on automation. This task runs on each commit and launches a test suite that is integrated into mach. diff --git a/docs/conf.py b/docs/conf.py index 72c25f464b..9b0c6cf41a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -56,6 +56,7 @@ extensions = [ # excluded from valid-jsdoc in the top-level .eslintrc.js. js_source_path = [ "../browser/components/backup", + "../browser/components/backup/actors", "../browser/components/backup/resources", "../browser/components/extensions", "../browser/components/migration", @@ -72,6 +73,7 @@ js_source_path = [ "../toolkit/components/extensions", "../toolkit/components/extensions/parent", "../toolkit/components/featuregates", + "../toolkit/components/ml/content/ONNXPipeline.mjs", "../toolkit/mozapps/extensions", "../toolkit/components/prompts/src", "../toolkit/components/pictureinpicture", diff --git a/docs/config.yml b/docs/config.yml index cf3cea87e9..249a92d911 100644 --- a/docs/config.yml +++ b/docs/config.yml @@ -67,6 +67,7 @@ categories: - testing/webrender - testing/mochitest-plain - testing/xpcshell + - testing/tps - web-platform - gtest - tools/fuzzing @@ -123,4 +124,4 @@ fatal warnings: - "WARNING: '([^']*)' reference target not found:((?!.rst).)*$" - "WARNING: Include file '([^']*)' not found or reading it failed" -max_num_warnings: 837 +max_num_warnings: 845 diff --git a/docs/contributing/contribution_quickref.rst b/docs/contributing/contribution_quickref.rst index f6f80e0cdf..9a9e5240e9 100644 --- a/docs/contributing/contribution_quickref.rst +++ b/docs/contributing/contribution_quickref.rst @@ -77,7 +77,7 @@ To Setup Firefox for Android $ curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O $ python3 bootstrap.py -More information on :ref:`building Firefox for Android ` +More information on :ref:`building Firefox for Android ` To set up your editor --------------------- diff --git a/docs/contributing/data-review.rst b/docs/contributing/data-review.rst new file mode 100644 index 0000000000..93cc4067a6 --- /dev/null +++ b/docs/contributing/data-review.rst @@ -0,0 +1,41 @@ +Data Review +----------- + +**Everything that lands in mozilla-central that adds or expands data +collection must go through the data review process.** + +This will require assessing the sensitivity of the data that is being +collected, and going through the `sensitive data collection +process `__ +if necessary. All data collection is subject to our `overall data +collection policy `__. + +Documentation for the data collection request process and the +expectations we have for people following it `lives on the +wiki `__. +This document describes the technical implementation in Phabricator +using tags. + +1. Any change that touches metrics will be automatically flagged with a + ``needs-data-classification`` tag by Phabricator, using `this herald + rule `__. If a change + adds/updates data collection in a way that doesn’t automatically + trigger this rule, this tag should be added manually (and if + appropriate, please file a bug to update the herald rule so it + happens automatically next time). + +2. After assessing data sensitivity, the tag can be replaced with either + ``data-classification-low`` or ``data-classification-high`` depending + on that sensitivity. + +3. Adding ``data-classification-high`` will auto-add the ``#data-stewards`` + reviewer group as a blocking reviewer for the change and initiate the + `sensitive data review process `__. + +4. For patches making mechanical changes that happen to trigger the + herald rule linked above, but that do not actually add or update any + data collection, the ``data-classification-unnecessary`` tag can be used. + +Patches with the ``needs-data-classification`` tag will not be landable in +Lando. The process linked above must be followed in order to land the +change. diff --git a/docs/contributing/how_to_submit_a_patch.rst b/docs/contributing/how_to_submit_a_patch.rst index d8ef62f742..7c5b56a7e1 100644 --- a/docs/contributing/how_to_submit_a_patch.rst +++ b/docs/contributing/how_to_submit_a_patch.rst @@ -105,11 +105,6 @@ simple commit message should look like this: Bug 123456 - Change this thing to work better by doing something. r=reviewers -The ``r=reviewers`` part is optional; if you are using Phabricator, -Lando will add it automatically based on who actually granted review, -and in any case the person who does the final check-in of the patch will -make sure it's added. - The text of the message should be what you did to fix the bug, not a description of what the bug was. If it is not obvious why this change is appropriate, then `explain why in the commit @@ -117,6 +112,11 @@ message ` or `git log `) +* Finally if you are still unable to identify someone, try asking in the `#introduction channel on Matrix `_. + + +Getting attention +----------------- + +Generally most reviews will happen within roughly a week. If, however, a reviewer doesn't respond within a week or so of the review request: * Contact the reviewer directly (either via e-mail or on Matrix). * Join developers on `Mozilla's Matrix server `_, and ask if anyone knows why a review may be delayed. Please link to the bug too. * If the review is still not addressed, mail the reviewer directly, asking if/when they'll have time to review the patch, or might otherwise be able to review it. +Remember that reviewers are human too, and may have complex reasons that prevent them from reviewing your patch in a timely manner. Be confident in reaching out to your reviewer, but be mindful of the `Mozilla Community Participation Guidelines `_ while doing so. + For simple documentation changes, reviews are not required. For more information about the review process, see the :ref:`Code Review FAQ`. diff --git a/docs/contributing/signing/signing_macos_build.rst b/docs/contributing/signing/signing_macos_build.rst index 688a935bbe..ef5e5f3ab3 100644 --- a/docs/contributing/signing/signing_macos_build.rst +++ b/docs/contributing/signing/signing_macos_build.rst @@ -127,7 +127,7 @@ Example: Re-Signing Official Nightly 0:00.20 Using ad-hoc signing identity 0:00.20 Using nightly channel signing configuration 0:00.20 Using developer entitlements - 0:00.20 Reading build config file /Users/me/r/mc/taskcluster/ci/config.yml + 0:00.20 Reading build config file /Users/me/r/mc/taskcluster/config.yml 0:00.23 Stripping existing xattrs and signatures 0:01.91 Signing with codesign 0:02.72 Verification of signed app /Users/me/Desktop/FirefoxNightly.app OK @@ -149,7 +149,7 @@ can be exported from Keychain Access in .p12 format. 0:00.26 Using pkcs12 signing identity 0:00.26 Using devedition channel signing configuration 0:00.26 Using developer entitlements - 0:00.26 Reading build config file /Users/me/r/mc/taskcluster/ci/config.yml + 0:00.26 Reading build config file /Users/me/r/mc/taskcluster/config.yml 0:00.29 Stripping existing xattrs and signatures 0:02.09 Signing with rcodesign 0:11.16 Verification of signed app /Users/me/Desktop/DevEdition.app OK diff --git a/docs/overview/ios.rst b/docs/overview/ios.rst index a16e0676bd..e52a3953ee 100644 --- a/docs/overview/ios.rst +++ b/docs/overview/ios.rst @@ -20,6 +20,6 @@ WKWebView --------- WKWebView is part of Apple`s WebKit framework. It supports a complete web browsing experience, -rendering HTML, CSS, and JavaScrip content alongside an app`s native views. It can also be thought +rendering HTML, CSS, and JavaScript content alongside an app`s native views. It can also be thought of as an API to help render web pages on Apple platforms. For more information on WKWebView itself, please see `Apple's documentation `_. diff --git a/docs/setup/common_build_errors.md b/docs/setup/common_build_errors.md new file mode 100644 index 0000000000..3267c96937 --- /dev/null +++ b/docs/setup/common_build_errors.md @@ -0,0 +1,34 @@ +# Common Build Errors + +When setting up Firefox, you may encounter some other build errors or +warnings that are not fatal. This document is to help you determine +if the error you're running into is a fatal one or not. + +## Watchman unavailable + +This is a warning and can be ignored. +[Watchman is a file watching service](https://facebook.github.io/watchman/) +that can speed up some interactions with Git and Mercurial. + +## VSCode Java extension + +If you happen to have the Java extension installed in VSCode, there's +a chance that the "Problems" tab in the integrated terminal will +display an error about `Cannot run program [...]/mozilla-unified/mach`. +This is because the extension does not know how to parse the mozilla-central +repository and specifically the `mach` command runner. +This will not prevent you from building and running Firefox. + +## ERROR glean_core + +This is a non-fatal error and will not prevent you from building and +running Firefox. You might see this text in the console after running +`./mach run`. The specific text might appear as: +`Error setting metrics feature config: [...]`. + +## Region.sys.mjs + +This is a non-fatal error, this will not prevent you from building and +running Firefox. You might see this text in the console after running +`./mach run` and the specific text might look like: +`console.error: Region.sys.mjs: Error fetching region`. diff --git a/docs/setup/index.rst b/docs/setup/index.rst index eb0c68aa4e..3cd6d3f229 100644 --- a/docs/setup/index.rst +++ b/docs/setup/index.rst @@ -18,6 +18,7 @@ Don't hesitate to look at the :ref:`Firefox Contributors Quick Reference