diff options
Diffstat (limited to 'testing/perfdocs')
23 files changed, 38550 insertions, 0 deletions
diff --git a/testing/perfdocs/generated/DAMP.rst b/testing/perfdocs/generated/DAMP.rst new file mode 100644 index 0000000000..71a764f081 --- /dev/null +++ b/testing/perfdocs/generated/DAMP.rst @@ -0,0 +1,8 @@ +DAMP Performance Tests +###################### + +================================== +(DevTools at Maximum Performance) +================================== + +Documentation for DAMP tests is located in `devtools source docs <https://firefox-source-docs.mozilla.org/devtools/tests/performance-tests-damp.html>`_.
\ No newline at end of file diff --git a/testing/perfdocs/generated/android-chrome-try-perf.png b/testing/perfdocs/generated/android-chrome-try-perf.png Binary files differnew file mode 100644 index 0000000000..f9b1172ea5 --- /dev/null +++ b/testing/perfdocs/generated/android-chrome-try-perf.png diff --git a/testing/perfdocs/generated/awsy.rst b/testing/perfdocs/generated/awsy.rst new file mode 100644 index 0000000000..327b4388c1 --- /dev/null +++ b/testing/perfdocs/generated/awsy.rst @@ -0,0 +1,304 @@ +==== +AWSY +==== + +Are We Slim Yet project (commonly known as AWSY) tracks memory usage across builds. + +On treeherder, the AWSY builds are listed in subgroups of `SY`. + +AWSY tests consist of three types: TP5*, TP6, and Base Memory Usage. + +*\*TP5 tests are out of date and no longer maintained. These tests are scheduled to be removed:* https://bugzilla.mozilla.org/show_bug.cgi?id=1712406 + +Awsy tests +---------- + + +.. dropdown:: base (FF) + :class-container: anchor-id-base-Awsy-tests + + **Owner**: :mccr8 and Perftest Team + + * **Test Task**: + * test-linux1804-64-clang-trunk-qr/opt + * awsy-base: None + * awsy-base-dmd: None + + * test-linux1804-64-qr/opt + * awsy-base: None + * awsy-base-dmd: None + + * test-linux1804-64-shippable-qr/opt + * awsy-base: integration, mozilla-beta, mozilla-central, mozilla-release + * awsy-base-dmd: None + + * test-macosx1015-64-shippable-qr/opt + * awsy-base: integration, mozilla-beta, mozilla-central, mozilla-release + * awsy-base-dmd: None + + * test-windows11-32-2009-qr/opt + * awsy-base: None + * awsy-base-dmd: None + + * test-windows11-32-2009-shippable-qr/opt + * awsy-base: None + * awsy-base-dmd: None + + * test-windows11-64-2009-qr/opt + * awsy-base: None + * awsy-base-dmd: None + + * test-windows11-64-2009-shippable-qr/opt + * awsy-base: integration, mozilla-beta, mozilla-central, mozilla-release + * awsy-base-dmd: None + + +.. dropdown:: dmd (FF) + :class-container: anchor-id-dmd-Awsy-tests + + **Owner**: :mccr8 and Perftest Team + + * **Test Task**: + * test-linux1804-64-clang-trunk-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-linux1804-64-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-linux1804-64-shippable-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-macosx1015-64-shippable-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-windows11-32-2009-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-windows11-32-2009-shippable-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-windows11-64-2009-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + * test-windows11-64-2009-shippable-qr/opt + * awsy-base-dmd: None + * awsy-dmd: None + + +.. dropdown:: tp5 (FF) + :class-container: anchor-id-tp5-Awsy-tests + + **Owner**: :mccr8 and Perftest Team + + * **Test Task**: + * test-linux1804-64-clang-trunk-qr/opt + + * test-linux1804-64-qr/opt + + * test-linux1804-64-shippable-qr/opt + + * test-macosx1015-64-shippable-qr/opt + + * test-windows11-32-2009-qr/opt + + * test-windows11-32-2009-shippable-qr/opt + + * test-windows11-64-2009-qr/opt + + * test-windows11-64-2009-shippable-qr/opt + + +.. dropdown:: tp6 (FF) + :class-container: anchor-id-tp6-Awsy-tests + + **Owner**: :mccr8 and Perftest Team + + * **Test Task**: + * test-linux1804-64-clang-trunk-qr/opt + * awsy-tp6: None + + * test-linux1804-64-qr/opt + * awsy-tp6: None + + * test-linux1804-64-shippable-qr/opt + * awsy-tp6: integration, mozilla-beta, mozilla-central, mozilla-release + + * test-macosx1015-64-shippable-qr/opt + * awsy-tp6: integration, mozilla-beta, mozilla-central, mozilla-release + + * test-windows11-32-2009-qr/opt + * awsy-tp6: None + + * test-windows11-32-2009-shippable-qr/opt + * awsy-tp6: None + + * test-windows11-64-2009-qr/opt + * awsy-tp6: None + + * test-windows11-64-2009-shippable-qr/opt + * awsy-tp6: integration, mozilla-beta, mozilla-central, mozilla-release + + + + +Running AWSY Locally +********************* + +Running tests locally is most likely only useful for debugging what is going on in a test, +as the test output is only reported as raw JSON. The CLI is documented via: + +.. code-block:: bash + + ./mach awsy-test --help + +*Currently all tests will download TP5 even if it is not used, see:* https://bugzilla.mozilla.org/show_bug.cgi?id=1683920 + +TP5 tests +========= + +.. code-block:: bash + + ./mach awsy-test + +TP6 tests +========= + +.. code-block:: bash + + ./mach awsy-test --tp6 + +Base Memory Usage tests +======================== + +.. code-block:: bash + + ./mach awsy-test testing/awsy/awsy/test_base_memory_usage.py + +Running AWSY on Try +******************** + +AWSY runs can be generated through the mach try fuzzy finder: + +.. code-block:: bash + + ./mach try fuzzy + +A query for "awsy" will return all AWSY tests. The default test is TP5, TP6 and Base test names will contain `tp6` and `base`, respectively. + +The following documents all tests we currently run for AWSY. +*The following content was migrated from* https://wiki.mozilla.org/AWSY/Tests *and will be updated to clarify TP5/TP6 tests vs Base tests:* https://bugzilla.mozilla.org/show_bug.cgi?id=1714600 + + +TP5/TP6 Tests +************** + +The following tests exist for both TP5 and TP6. Running `./mach awsy-test` by default will run TP5 tests*. +To run TP6 tests, add the `--tp6` flag: `./mach awsy-test --tp6` + +*\*TP5 tests are out of date and no longer maintained. These tests are scheduled to be removed:* https://bugzilla.mozilla.org/show_bug.cgi?id=1712406 + + +Explicit Memory +================ + +* This is memory explicitly reported by a memory reporter. It includes all the memory allocated via explicit calls to heap allocation functions (such as malloc and new), and some (only that covered by memory reporters) of the memory allocated via explicit calls to non-heap allocations functions (such as mmap and VirtualAlloc). + +**Possible regression causes** + +* A regression in this usually means a new feature is using or retaining more memory and should be looked at. These are easier to diagnose as we can compare memory reports. + +See the `about:memory` `mdn page <https://developer.mozilla.org/docs/Mozilla/Performance/about:memory#Explicit_Allocations>`__ for more details. + + +Heap Unclassified +================== + +*to do: add test definition* + + +Images +======= + +* This is a subset of the "explicit" measurement that focuses on memory used to render images. + +**Possible regression causes** + +* A regressions in this can indicate leaks or poor memory usage in the image subsystem. In the past this was persistent problem. + + +JS +==== + +*to do: add test definition* + + +Resident Memory +================ + +* This is a higher level measurement provided by the operating system. We sum the "resident" memory (`RSS <https://en.wikipedia.org/wiki/Resident_set_size>`_) with the `resident-unique <https://en.wikipedia.org/wiki/Unique_set_size>`_ memory of the content processes. It's pretty noisy and large so it's not very useful in detecting smaller regressions. + +**Possible regression causes** + +* Regressions in this often track regressions in explicit and heap unclassified. If we see a regression in resident, but not in other reports this can indicate we are leaking untracked memory (perhaps through shared memory, graphics allocations, file handles, etc). + + +Base Content Tests +******************* + +* An updated test focused on supporting Fission. This measures the base overhead of an empty content process. It tracks resident unique, heap unclassified, JS, and explicit memory metrics as well as storing full memory reports as artifacts. The median value for each metric is used from across all content processes. It has much lower thresholds for alerting and is recorded in `Perfherder <https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder>`_. + + +Base Content Explicit +====================== + +**Possible regression causes** + +A change has caused more JavaScript to load at startup or into blank pages + +* **Common solution**: lazily load any new modules you rely on +* **Common solution**: Split your code out to only load what is minimally needed initially. You modified the JS engine and it's using more memory +* **Common solution**: Attempt to reduce your object size for the common case, these tend to add up! You implemented a new feature in JavaScript +* **Common solution**: Write the majority (or all of it) in compiled code (C++/Rust). This will reduce overhead and generally improve performance. + + +Base Content Heap Unclassified +=============================== + +* The "heap-unclassified" value represents heap-allocated memory that is not measured by any memory reporter. This is typically 10--20% of "explicit". + + +**Possible regression causes** + +* A regression in this can indicate that we're leaking memory or that additional memory reporters should be added. +* An improvement can indicate that leaks have been fixed or that we added new memory reporters. + +See the `about:memory` `mdn page <https://developer.mozilla.org/docs/Mozilla/Performance/about:memory#Explicit_Allocations>`__ for more details. + + +Base Content JS +================ + +* This is the "js-main-runtime/" value in `about:memory` which is all the memory attributed to the javascript engine. + +**Possible regression causes** + +* A regression in this number can indicate leaks in the JS engine, optimizations that take performance into consideration at the expense of more memory, or problems with the garbage collector. + + +Base Content Resident Unique Memory +==================================== + +*to do: add test definition* + + +Other references +----------------- + +`Are We Slim Yet MDN web docs <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/AWSY>`_ diff --git a/testing/perfdocs/generated/browsertime.rst b/testing/perfdocs/generated/browsertime.rst new file mode 100644 index 0000000000..d491676edf --- /dev/null +++ b/testing/perfdocs/generated/browsertime.rst @@ -0,0 +1,244 @@ +################## +Raptor Browsertime +################## + +.. contents:: + :depth: 2 + :local: + +Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles. + +Source code: + +- Our current Browsertime version uses the canonical repo. +- In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh + +Running Locally +--------------- + +**Prerequisites** + +- A local mozilla repository clone with a `successful Firefox build </setup>`_ completed + +Running on Firefox Desktop +-------------------------- + +Vanilla Browsertime tests +------------------------- + +If you want to run highly customized tests, you can make use of our customizable ``browsertime`` test. + +With this test, you can customize the page to test, test script to use, and anything else required. It will make use of default settings that Raptor uses in browsertime but these can be overridden with ``--browsertime-arg`` settings. + +For example, here's a test on ``https://www.sitespeed.io`` using this custom test: + +:: + + ./mach raptor --browsertime -t browsertime --browsertime-arg test_script=pageload --browsertime-arg browsertime.url=https://www.sitespeed.io --browsertime-arg iterations=3 + +That test will perform 3 iterations of the given url. Note also that we can use simplified names to make use of test scripts that are built into raptor. You can use ``pageload``, ``interactive``, or provide a path to another test script. + +This custom test is only available locally. + +Page-load tests +--------------- +There are two ways to run performance tests through browsertime listed below. + +**Note that** ``./mach browsertime`` **should not be used when debugging performance issues with profiles as it does not do symbolication.** + +* Raptor-Browsertime (recommended): + +:: + + ./mach raptor --browsertime -t google-search + +* Browsertime-"native": + +:: + + ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox' + +Benchmark tests +--------------- +* Raptor-wrapped: + +:: + + ./mach raptor -t raptor-speedometer --browsertime + +Running on Android +------------------ +Running on Raptor-Browsertime (recommended): + +* Running on Fenix + +:: + + ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix + +* Running on Geckoview + +:: + + ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example + +Running on vanilla Browsertime: + +* Running on Fenix/Firefox Preview + +:: + + ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io + +* Running on the GeckoView Example app + +:: + + ./mach browsertime --android --browser firefox https://www.sitespeed.io + +Running on Google Chrome +------------------------ +Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned. + +There are two ways of doing this: + +* Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path: + +:: + + ./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER> + +* Upgrade the ChromeDriver version in ``tools/browsertime/package-lock.json`` (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions). + +Run ``npm install``. + +Launch vanilla Browsertime as follows: + +:: + + ./mach browsertime https://www.sitespeed.io -b chrome + +Or for Raptor-Browsertime (use ``chrome`` for desktop, and ``chrome-m`` for mobile): + +:: + + ./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER> + +More Examples +------------- + +`Browsertime docs <https://github.com/sitespeedio/browsertime/tree/main/docs/examples>`_ + +Passing Additional Arguments to Browsertime +------------------------------------------- + +Browsertime has many command line flags to configure its usage, see `Browsertime configuration <https://www.sitespeed.io/documentation/browsertime/configuration/>`_. + +There are multiple ways of adding additional arguments to Browsertime from Raptor. The primary method is to use ``--browsertime-arg``. For example: ``./mach raptor --browsertime -t amazon --browsertime-arg iterations=10`` + +Other methods for adding additional arguments are: + +* Define additional arguments in `testing/raptor/raptor/browsertime/base.py <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/browsertime/base.py#220-252>`_. + +* Add a ``browsertime_args`` entry to the appropriate manifest with the desired arguments, i.e. `browsertime-tp6.ini <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/tp6/desktop/browsertime-tp6.ini>`_ for desktop page load tests. `Example of browsertime_args format <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/custom/browsertime-process-switch.ini#27>`_. + +Running Browsertime on Try +-------------------------- +You can run all of our browsertime pageload tests through ``./mach try fuzzy --full``. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time. + +For example: + +:: + + ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable" + +Retriggering Browsertime Visual Metrics Tasks +--------------------------------------------- + +You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.). + +The following metrics are collected each time: ``fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange`` + +Further information regarding these metrics can be viewed at `visual-metrics <https://www.sitespeed.io/documentation/sitespeed.io/metrics/#visual-metrics>`_ + +Gecko Profiling with Browsertime +-------------------------------- + +To run gecko profiling using Raptor-Browsertime you can add the ``--gecko-profile`` flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example: + +:: + + ./mach raptor --browsertime -t amazon --gecko-profile + +Note that vanilla Browsertime does support Gecko Profiling but **it does not symbolicate the profiles** so it is **not recommended** to use for debugging performance regressions/improvements. + +Upgrading Browsertime In-Tree +----------------------------- +To upgrade the browsertime version used in-tree you can run, then commit the changes made to ``package.json`` and ``package-lock.json``: + +:: + + ./mach browsertime --update-upstream-url <TARBALL-URL> + +Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47 + +To test the upgrade, run a raptor test locally (with and without visual-metrics ``--browsertime-visualmetrics`` if possible) and test it on try with at least one test on desktop and mobile. + +Updating Benchmark Tests +------------------------ +To upgrade any of our benchmark tests, you will need to change the revision used in the test manifest. There are three fields that you have available to use there: ``repository_revision`` to denote the revision, ``repository_branch`` to denote the branch name, and ``repository`` to provide the link of the Github repo that contains the benchmark. + +For instance, with Speedometer 3 (sp3), we can update the revision `by changing the repository_revision field found here <https://searchfox.org/mozilla-central/rev/aa3ccd258b64abfd4c5ce56c1f512bc7f65b844c/testing/raptor/raptor/tests/benchmarks/speedometer-desktop.ini#29>`_. If the change isn't found on the default branch (master/main branch), then you will need to add an entry for ``repository_branch`` to specify this. + +If the path to the test file changes (the file that is invoked to run the test), then the ``test_url`` will need to be changed. + +Finding the Geckodriver Being Used +---------------------------------- +If you're looking for the latest geckodriver being used there are two ways: +* Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver +* Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the ``Task`` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in. + +If you're trying to test Browsertime with a new geckodriver, you can do either of the following: +* Request a new geckodriver build in your try run (i.e. through ``./mach try fuzzy``). +* Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push. + +Comparing Before/After Browsertime Videos +----------------------------------------- + +We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/mozilla/mozperftest-tools#browsertime-side-by-side-video-comparisons + +Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video. + +Mach Browsertime Setup +---------------------- + +**WARNING** + Raptor-Browsertime (i.e. ``./mach raptor --browsertime -t <TEST>``) is currently required to be ran first in order to acquire the Node-16 binary. In general, it is also not recommended to use ``./mach browsertime`` for testing as it will be deprecated soon. + +Note that if you are running Raptor-Browsertime then it will get installed automatically and also update itself. Otherwise, you can run: + +- ``./mach browsertime --clobber --setup --install-vismet-reqs`` + +This will automatically check your setup and install the necessary dependencies if required. If successful, the output should read as something similar to: + +:: + + browsertime installed successfully! + + NOTE: Your local browsertime binary is at <...>/mozilla-unified/tools/browsertime/node_modules/.bin/browsertime + +- To manually check your setup, you can also run ``./mach browsertime --check`` + +Known Issues +^^^^^^^^^^^^ + +With the replacement of ImageMagick, former cross platform installation issues have been resolved. The details of this can be viewed in the meta bug tracker +`Bug 1735410 <https://bugzilla.mozilla.org/show_bug.cgi?id=1735410>`_ + + + +- For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command or a Raptor-Browsertime test + +- If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the :ref: `Running on Android` section) + +- **If you encounter any issues not mentioned here, please** `file a bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Raptor>`_ **in the** ``Testing::Raptor`` **component.** diff --git a/testing/perfdocs/generated/contributing.rst b/testing/perfdocs/generated/contributing.rst new file mode 100644 index 0000000000..30beac81c3 --- /dev/null +++ b/testing/perfdocs/generated/contributing.rst @@ -0,0 +1,42 @@ +############ +Contributing +############ + +Raptor on Mobile projects (Fenix, Reference-Browser) +**************************************************** + +Add new tests +------------- + +For mobile projects, Raptor tests are on the following repositories: + +**Fenix**: + +- Repository: `Github <https://github.com/mozilla-mobile/fenix/>`__ +- Tests results: `Treeherder view <https://treeherder.mozilla.org/#/jobs?repo=fenix>`__ +- Schedule: Every 24 hours `Taskcluster force hook <https://tools.taskcluster.net/hooks/project-releng/cron-task-mozilla-mobile-fenix%2Fraptor>`_ + +**Reference-Browser**: + +- Repository: `Github <https://github.com/mozilla-mobile/reference-browser/>`__ +- Tests results: `Treeherder view <https://treeherder.mozilla.org/#/jobs?repo=reference-browser>`__ +- Schedule: On each push + +Tests are now defined in a similar fashion compared to what exists in mozilla-central. Task definitions are expressed in Yaml: + +* https://github.com/mozilla-mobile/fenix/blob/1c9c5317eb33d92dde3293dfe6a857c279a7ab12/taskcluster/ci/raptor/kind.yml +* https://github.com/mozilla-mobile/reference-browser/blob/4560a83cb559d3d4d06383205a8bb76a44336704/taskcluster/ci/raptor/kind.yml + +If you want to test your changes on a PR, before they land, you need to apply a patch like this one: https://github.com/mozilla-mobile/fenix/pull/5565/files. Don't forget to revert it before merging the patch. Note that the checks will run but the results aren't currently available on treeherder (`bug 1593252 <https://bugzilla.mozilla.org/show_bug.cgi?id=1593252>`_ is expected to address this). + +On Fenix and Reference-Browser, the raptor revision is tied to the latest nightly of mozilla-central + +For more information, please reach out to :mhentges in #cia + +Code formatting on Raptor +************************* +As Raptor is a Mozilla project we follow the general Python coding style: + +* https://firefox-source-docs.mozilla.org/tools/lint/coding-style/coding_style_python.html + +`black <https://github.com/psf/black/>`_ is the tool used to reformat the Python code. diff --git a/testing/perfdocs/generated/debugging.rst b/testing/perfdocs/generated/debugging.rst new file mode 100644 index 0000000000..755d231d8e --- /dev/null +++ b/testing/perfdocs/generated/debugging.rst @@ -0,0 +1,132 @@ +######### +Debugging +######### + +.. contents:: + :depth: 2 + :local: + +Debugging Desktop Product Failures +********************************** + +As of now, there is no easy way to do this. Raptor was not built for debugging functional failures. Hitting these in Raptor is indicative that we lack functional test coverage so regression tests should be added for those failures after they are fixed. + +To debug a functional failure in Raptor you can follow these steps: + +#. If bug 1653617 has not landed yet, apply the patch. +#. Add the --verbose flag to the extra-options list `here <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#98-101>`__. +#. If the --setenv doesn't exist yet (`bug 1494669 <https://bugzilla.mozilla.org/show_bug.cgi?id=1494669>`_), then add your MOZ_LOG environment variables to give you additional logging `here <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/webextension/desktop.py#42>`_. +#. If the flag does exist, then you can add the MOZ_LOG variables to the `raptor.yml <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml>`_ configuration file. +#. Push to try if you can't reproduce the failure locally. + +You can follow `bug 1655554 <https://bugzilla.mozilla.org/show_bug.cgi?id=1655554>`_ as we work on improving this workflow. + +In some cases, you might not be able to get logging for what you are debugging (browser console logging for instance). In this case, you should make your own debug prints with printf or something along those lines (`see :agi's debugging work for an example <https://matrix.to/#/!LfXZSWEroPFPMQcYmw:mozilla.org/$r_azj7OipkgDzQ75SCns2QIayp4260PIMHLWLApJJNg?via=mozilla.org&via=matrix.org&via=rduce.org>`_). + +Debugging the Raptor Web Extension +********************************** + +When developing on Raptor and debugging, there's often a need to look at the output coming from the `Raptor Web Extension <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor>`_. Here are some pointers to help. + +Raptor Debug Mode +----------------- + +The easiest way to debug the Raptor web extension is to run the Raptor test locally and invoke debug mode, i.e. for Firefox: + +:: + + ./mach raptor --test raptor-tp6-amazon-firefox --debug-mode + +Or on Chrome, for example: + +:: + + ./mach raptor --test raptor-tp6-amazon-chrome --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --debug-mode + +Running Raptor with debug mode will: + +* Automatically set the number of test page-cycles to 2 maximum +* Reduce the 30 second post-browser startup delay from 30 seconds to 3 seconds +* On Firefox, the devtools browser console will automatically open, where you can view all of the console log messages generated by the Raptor web extension +* On Chrome, the devtools console will automatically open +* The browser will remain open after the Raptor test has finished; you will be prompted in the terminal to manually shutdown the browser when you're finished debugging. + +Manual Debugging on Firefox Desktop +----------------------------------- + +The main Raptor runner is '`runner.js <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/runner.js>`_' which is inside the web extension. The code that actually captures the performance measures is in the web extension content code '`measure.js <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/measure.js>`_'. + +In order to retrieve the console.log() output from the Raptor runner, do the following: + +#. Invoke Raptor locally via ``./mach raptor`` +#. During the 30 second Raptor pause which happens right after Firefox has started up, in the ALREADY OPEN current tab, type "about:debugging" for the URL. +#. On the debugging page that appears, make sure "Add-ons" is selected on the left (default). +#. Turn ON the "Enable add-on debugging" check-box +#. Then scroll down the page until you see the Raptor web extension in the list of currently-loaded add-ons. Under "Raptor" click the blue "Debug" link. +#. A new window will open in a minute, and click the "console" tab + +To retrieve the console.log() output from the Raptor content 'measure.js' code: + +#. As soon as Raptor opens the new test tab (and the test starts running / or the page starts loading), in Firefox just choose "Tools => Web Developer => Web Console", and select the "console' tab. + +Raptor automatically closes the test tab and the entire browser after test completion; which will close any open debug consoles. In order to have more time to review the console logs, Raptor can be temporarily hacked locally in order to prevent the test tab and browser from being closed. Currently this must be done manually, as follows: + +#. In the Raptor web extension runner, comment out the line that closes the test tab in the test clean-up. That line of `code is here <https://searchfox.org/mozilla-central/rev/3c85ea2f8700ab17e38b82d77cd44644b4dae703/testing/raptor/webext/raptor/runner.js#357>`_. +#. Add a return statement at the top of the Raptor control server method that shuts-down the browser, the browser shut-down `method is here <https://searchfox.org/mozilla-central/rev/924e3d96d81a40d2f0eec1db5f74fc6594337128/testing/raptor/raptor/control_server.py#120>`_. + +For **benchmark type tests** (i.e. speedometer, motionmark, etc.) Raptor doesn't inject 'measure.js' into the test page content; instead it injects '`benchmark-relay.js <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/benchmark-relay.js>`_' into the benchmark test content. Benchmark-relay is as it sounds; it basically relays the test results coming from the benchmark test, to the Raptor web extension runner. Viewing the console.log() output from benchmark-relay is done the same was as noted for the 'measure.js' content above. + +Note, `Bug 1470450 <https://bugzilla.mozilla.org/show_bug.cgi?id=1470450>`_ is on file to add a debug mode to Raptor that will automatically grab the web extension console output and dump it to the terminal (if possible) that will make debugging much easier. + +Debugging TP6 and Killing the Mitmproxy Server +---------------------------------------------- + +Regarding debugging Raptor pageload tests that use Mitmproxy (i.e. tp6, gdocs). If Raptor doesn't finish naturally and doesn't stop the Mitmproxy tool, the next time you attempt to run Raptor it might fail out with this error: + +:: + + INFO - Error starting proxy server: OSError(48, 'Address already in use') + NFO - raptor-mitmproxy Aborting: mitmproxy playback process failed to start, poll returned: 1 + +That just means the Mitmproxy server was already running before so it couldn't startup. In this case, you need to kill the Mitmproxy server processes, i.e: + +:: + + mozilla-unified rwood$ ps -ax | grep mitm + 5439 ttys000 0:00.09 /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/mitmdump -k -q -s /Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/amazon.mp + 440 ttys000 0:01.64 /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/mitmdump -k -q -s /Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/amazon.mp + 5509 ttys000 0:00.01 grep mitm + +Then just kill the first mitm process in the list and that's sufficient: + +:: + + mozilla-unified rwood$ kill 5439 + +Now when you run Raptor again, the Mitmproxy server will be able to start. + +Manual Debugging on Firefox Android +----------------------------------- + +Be sure to read the above section first on how to debug the Raptor web extension when running on Firefox Desktop. + +When running Raptor tests on Firefox on Android (i.e. geckoview), to see the console.log() output from the Raptor web extension, do the following: + +#. With your android device (i.e. Google Pixel 2) all set up and connected to USB, invoke the Raptor test normally via ``./mach raptor`` +#. Start up a local copy of the Firefox Nightly Desktop browser +#. In Firefox Desktop choose "Tools => Web Developer => WebIDE" +#. In the Firefox WebIDE dialog that appears, look under "USB Devices" listed on the top right. If your device is not there, there may be a link to install remote device tools - if that link appears click it and let that install. +#. Under "USB Devices" on the top right your android device should be listed (i.e. "Firefox Custom on Android Pixel 2" - click on your device. +#. The debugger opens. On the left side click on "Main Process", and click the "console" tab below - and the Raptor runner output will be included there. +#. On the left side under "Tabs" you'll also see an option for the active tab/page; select that and the Raptor content console.log() output should be included there. + +Also note: When debugging Raptor on Android, the 'adb logcat' is very useful. More specifically for 'geckoview', the output (including for Raptor) is prefixed with "GeckoConsole" - so this command is very handy: + +:: + + adb logcat | grep GeckoConsole + +Manual Debugging on Google Chrome +--------------------------------- + +Same as on Firefox desktop above, but use the Google Chrome console: View ==> Developer ==> Developer Tools. diff --git a/testing/perfdocs/generated/developing.rst b/testing/perfdocs/generated/developing.rst new file mode 100644 index 0000000000..97125d8729 --- /dev/null +++ b/testing/perfdocs/generated/developing.rst @@ -0,0 +1,154 @@ +Developing in mozperftest +========================= + +Architecture overview +--------------------- + +`mozperftest` implements a mach command that is a thin wrapper on the +top of `runner.py`, which allows us to run the tool without having to go through +a mach call. Command arguments are prepared in `argparser.py` and then made +available for the runner. + +The runner creates a `MachEnvironment` instance (see `environment.py`) and a +`Metadata` instance (see `metadata.py`). These two objects are shared during the +whole test and used to share data across all parts. + +The runner then calls `MachEnvironment.run`, which is in charge of running the test. +The `MachEnvironment` instance runs a sequence of **layers**. + +Layers are classes responsible of one single aspect of a performance test. They +are organized in three categories: + +- **system**: anything that sets up and tears down some resources or services + on the system. Existing system layers: **android**, **proxy** +- **test**: layers that are in charge of running a test to collect metrics. + Existing test layers: **browsertime** and **androidlog** +- **metrics**: all layers that process the metrics to turn them into usable + metrics. Existing system layers: **perfherder** and **console** + +The MachEnvironment instance collects a series of layers for each category and +runs them sequentially. + +The goal of this organization is to allow adding new performance tests runners +that will be based on a specific combination of layers. To avoid messy code, +we need to make sure that each layer represents a single aspect of the process +and that is completely independent from other layers (besides sharing the data +through the common environment.) + +For instance, we could use `perftest` to run a C++ benchmark by implementing a +new **test** layer. + + +Layer +----- + +A layer is a class that inherits from `mozperftest.layers.Layer` and implements +a few methods and class variables. + +List of methods and variables: + +- `name`: name of the layer (class variable, mandatory) +- `activated`: boolean to activate by default the layer (class variable, False) +- `user_exception`: will trigger the `on_exception` hook when an exception occurs +- `arguments`: dict containing arguments. Each argument is following + the `argparser` standard +- `run(self, medatata)`: called to execute the layer +- `setup(self)`: called when the layer is about to be executed +- `teardown(self)`: called when the layer is exiting + +Example:: + + class EmailSender(Layer): + """Sends an email with the results + """ + name = "email" + activated = False + + arguments = { + "recipient": { + "type": str, + "default": "tarek@mozilla.com", + "help": "Recipient", + }, + } + + def setup(self): + self.server = smtplib.SMTP(smtp_server,port) + + def teardown(self): + self.server.quit() + + def __call__(self, metadata): + self.server.send_email(self.get_arg("recipient"), metadata.results()) + + +It can then be added to one of the top functions that are used to create a list +of layers for each category: + +- **mozperftest.metrics.pick_metrics** for the metrics category +- **mozperftest.system.pick_system** for the system category +- **mozperftest.test.pick_browser** for the test category + +And also added in each `get_layers` function in each of those category. +The `get_layers` functions are invoked when building the argument parser. + +In our example, adding the `EmailSender` layer will add two new options: + +- **--email** a flag to activate the layer +- **--email-recipient** + + +Important layers +---------------- + +**mozperftest** can be used to run performance tests against browsers using the +**browsertime** test layer. It leverages the `browsertime.js +<https://www.sitespeed.io/documentation/browsertime/>`_ framework and provides +a full integration into Mozilla's build and CI systems. + +Browsertime uses the selenium webdriver client to drive the browser, and +provides some metrics to measure performance during a user journey. + + +Coding style +------------ + +For the coding style, we want to: + +- Follow `PEP 257 <https://www.python.org/dev/peps/pep-0257/>`_ for docstrings +- Avoid complexity as much as possible +- Use modern Python 3 code (for instance `pathlib` instead of `os.path`) +- Avoid dependencies on Mozilla build projects and frameworks as much as possible + (mozharness, mozbuild, etc), or make sure they are isolated and documented + + +Landing patches +--------------- + +.. warning:: + + It is mandatory for each patch to have a test. Any change without a test + will be rejected. + +Before landing a patch for mozperftest, make sure you run `perftest-test`:: + + % ./mach perftest-test + => black [OK] + => flake8 [OK] + => remove old coverage data [OK] + => running tests [OK] + => coverage + Name Stmts Miss Cover Missing + ------------------------------------------------------------------------------------------ + mozperftest/metrics/notebook/analyzer.py 29 20 31% 26-36, 39-42, 45-51 + ... + mozperftest/system/proxy.py 37 0 100% + ------------------------------------------------------------------------------------------ + TOTAL 1614 240 85% + + [OK] + +The command will run `black`, `flake8` and also make sure that the test coverage has not regressed. + +You can use the `-s` option to bypass flake8/black to speed up your workflow, but make +sure you do a full tests run. You can also pass the name of one single test module. diff --git a/testing/perfdocs/generated/fxrecord.rst b/testing/perfdocs/generated/fxrecord.rst new file mode 100644 index 0000000000..313d4b74a3 --- /dev/null +++ b/testing/perfdocs/generated/fxrecord.rst @@ -0,0 +1,8 @@ +======== +Fxrecord +======== + +`Fxrecord <https://github.com/mozilla/fxrecord>`__ is a tool for measuring the +startup performance of Firefox for Desktop. It captures a video of Firefox for +desktop starting on a laptop and computes visual metrics using the same manner +as Raptor using Browsertime. diff --git a/testing/perfdocs/generated/index.rst b/testing/perfdocs/generated/index.rst new file mode 100644 index 0000000000..2f0835015d --- /dev/null +++ b/testing/perfdocs/generated/index.rst @@ -0,0 +1,98 @@ +################### +Performance Testing +################### + +.. toctree:: + :maxdepth: 2 + :hidden: + :glob: + + DAMP + awsy + fxrecord + mach-try-perf + mozperftest + raptor + talos + +Performance tests are designed to catch performance regressions before they reach our +end users. At this time, there is no unified approach for these types of tests, +but `mozperftest </testing/perfdocs/mozperftest.html>`_ aims to provide this in the future. + +For more detailed information about each test suite and project, see their documentation: + + * :doc:`DAMP` + * :doc:`awsy` + * :doc:`fxrecord` + * :doc:`mach-try-perf` + * :doc:`mozperftest` + * :doc:`raptor` + * :doc:`talos` + + +Here are the active PerfTest components/modules and their respective owners: + + * AWFY (Are We Fast Yet) - + - Owner: Beatrice A. + - Description: A public dashboard comparing Firefox and Chrome performance metrics + * AWSY (Are We Slim Yet) + - Owner: Alexandru I. + - Description: Project that tracks memory usage across builds + * Raptor + - Owner: Sparky + - Co-owner: Kash + - Description: Test harness that uses Browsertime (based on webdriver) as the underlying engine to run performance tests + * CondProf (Conditioned profiles) + - Owner: Sparky + - Co-owner: Jmaher + - Description: Provides tooling to build, and obtain profiles that are preconditioned in some way. + * fxrecord + - Owner: Sparky + - Co-owners: Kash, Andrej + - Description: Tool for measuring startup performance for Firefox Desktop + * Infrastructure + - Owner: Sparky + - Co-owners: Kash, Andrej + - Description: All things involving: TaskCluster, Youtube Playback, Bitbar, Mobile Configs, etc... + * Mozperftest + - Owner: Sparky + - Co-owners: Kash, Andrej + - Description: Testing framework used to run performance tests + * Mozperftest Tools + - Owner: Sparky + - Co-owner: Alexandru I. + - Description: Various tools used by performance testing team + * Mozproxy + - Owner: Sparky + - Co-owner: Kash + - Description: An http proxy used to run tests against third-party websites in a reliable and reproducible way + * PerfCompare + - Owner: Kimberly + - Co-owner: Beatrice A. + - Description: Performance comparison tool used to compare performance of different commits within a repository + * PerfDocs + - Owner: Sparky + - Co-owner: Alexandru I. + - Description: Automatically generated performance test engineering documentation + * PerfHerder + - Owner: Beatrice A + - Co-owner: Andra A. + - Description: The framework used by the performance sheriffs to find performance regressions and for storing, and visualizing our performance data. + * Performance Sheriffing + - Owner: Alexandru I. + - Co-owner: Andra A. + - Description: Performance sheriffs are responsible for finding commits that cause performance regressions and getting fixes from devs or backing out the changes + * Talos + - Owner: Sparky + - Co-owner: Andrej + - Description: Testing framework used to run Firefox-specific performance tests + * WebPageTest + - Owner: Andrej + - Co-owner: Sparky + - Description: A test running in the mozperftest framework used as a third party performance benchmark + +You can additionally reach out to our team on +the `#perftest <https://matrix.to/#/#perftest:mozilla.org>`_ channel on matrix + +For more information about the performance testing team, +`visit the wiki page <https://wiki.mozilla.org/TestEngineering/Performance>`_. diff --git a/testing/perfdocs/generated/mach-try-perf.rst b/testing/perfdocs/generated/mach-try-perf.rst new file mode 100644 index 0000000000..645c7e1dd5 --- /dev/null +++ b/testing/perfdocs/generated/mach-try-perf.rst @@ -0,0 +1,268 @@ +############# +Mach Try Perf +############# + +.. contents:: + :depth: 2 + :local: + +To make it easier for developers to find the tests they need to run we built a perf-specific try selector called `./mach try perf`. With this tool, you no longer need to remember the obfuscated platform and test names that you need to target for your tests. Instead, the new interface shows test categories along with a simplified name of the platform that they will run on. + +When you trigger a try run from the perf selector, two try runs will be created. One with your changes, and one without. In your console, after you trigger the try runs, you'll find a PerfCompare link that will bring you directly to a comparison of the two pushes when they have completed. + +The tool is built to be conservative about the number of tests to run, so if you are looking for something that is not listed, it's likely hidden behind a flag found in the `--help`. Here's a small sample of what you'll find there which highlights the most relevant flags:: + + $ ./mach try perf --help + + perf arguments: + --show-all Show all available tasks. + --android Show android test categories (disabled by default). + --chrome Show tests available for Chrome-based browsers (disabled by default). + --safari Show tests available for Safari (disabled by default). + --live-sites Run tasks with live sites (if possible). You can also use the `live-sites` variant. + --profile Run tasks with profiling (if possible). You can also use the `profiling` variant. + --single-run Run tasks without a comparison + --variants [ [ ...]] Select variants to display in the selector from: no-fission, bytecode-cached, live-sites, profiling, swr + --platforms [ [ ...]] + Select specific platforms to target. Android only available with --android. Available platforms: android-a51, android, + windows, linux, macosx, desktop + --apps [ [ ...]] Select specific applications to target from: firefox, chrome, chromium, geckoview, fenix, chrome-m, safari + + task configuration arguments: + --artifact Force artifact builds where possible. + + +Standard Usage +-------------- + +To use mach try perf simply call `./mach try perf`. This will open an interface for test selection like so: + + +.. image:: ./standard-try-perf.png + :alt: Mach try perf with default options + :scale: 75% + :align: center + + +Select the categories you'd like to run, hit enter, and wait for the tool to finish the pushes. **Note that it can take some time to do both pushes, and you might not see logging for some time.** + +Chrome and Android +------------------ + +Android and chrome tests are disabled by default as they are often unneeded and waste our limited resources. If you need either of these, you can add `--chrome` and/or `--android` to the command like so `./mach try perf --android --chrome`: + + +.. image:: ./android-chrome-try-perf.png + :alt: Mach try perf with android, and chrome options + :scale: 75% + :align: center + + +Variants +-------- + +If you are looking for any variants (e.g. no-fission, bytecode-cached, live-sites), use the `--variants` options like so `./mach try perf --variants live-sites`. This will select all possible categories that could have live-sites tests. + + +.. image:: ./variants-try-perf.png + :alt: Mach try perf with variants + :scale: 75% + :align: center + + +Note that it is expected that the offered categories have extra variants (such as bytecode-cached) as we are showing all possible combinations that can include live-sites. + +Platforms +--------- + +To target a particular platform you can use `--platforms` to only show categories with the given platforms. + +Categories +---------- + +In the future, this section will be populated dynamically. If you are wondering what the categories you selected will run, you can use `--no-push` to print out a list of tasks that will run like so:: + + $ ./mach try perf --no-push + + Artifact builds enabled, pass --no-artifact to disable + Gathering tasks for Benchmarks desktop category + Executing queries: 'browsertime 'benchmark, !android 'shippable !-32 !clang, !live, !profil, !chrom + estimates: Runs 66 tasks (54 selected, 12 dependencies) + estimates: Total task duration 8:45:58 + estimates: In the top 62% of durations + estimates: Should take about 1:04:58 (Finished around 2022-11-22 15:08) + Commit message: + Perf selections=Benchmarks desktop (queries='browsertime 'benchmark&!android 'shippable !-32 !clang&!live&!profil&!chrom) + Pushed via `mach try perf` + Calculated try_task_config.json: + { + "env": { + "TRY_SELECTOR": "fuzzy" + }, + "tasks": [ + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-ares6", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-assorted-dom", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-jetstream2", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-matrix-react-bench", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-motionmark-animometer", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-motionmark-htmlsuite", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-speedometer", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-stylebench", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-sunspider", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-twitch-animation", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-unity-webgl", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-firefox-webaudio", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot-baseline", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot-optimizing", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc-baseline", + "test-linux1804-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc-optimizing", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-ares6", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-assorted-dom", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-jetstream2", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-matrix-react-bench", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-motionmark-animometer", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-motionmark-htmlsuite", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-speedometer", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-stylebench", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-sunspider", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-twitch-animation", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-unity-webgl", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-firefox-webaudio", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot-baseline", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot-optimizing", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc-baseline", + "test-macosx1015-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc-optimizing", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-ares6", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-assorted-dom", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-jetstream2", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-matrix-react-bench", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-motionmark-animometer", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-motionmark-htmlsuite", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-speedometer", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-stylebench", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-sunspider", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-twitch-animation", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-unity-webgl", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-firefox-webaudio", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot-baseline", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-godot-optimizing", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc-baseline", + "test-windows10-64-shippable-qr/opt-browsertime-benchmark-wasm-firefox-wasm-misc-optimizing" + ], + "use-artifact-builds": true, + "version": 1 + } + + +Adding a New Category +--------------------- + +It's very easy to add a new category if needed, and you can do so by modifying the `PerfParser categories attribute here <https://searchfox.org/mozilla-central/source/tools/tryselect/selectors/perf.py#179>`_. The following is an example of a complex category that gives a good idea of what you have available:: + + "Resource Usage": { + "query": { + "talos": ["'talos 'xperf | 'tp5"], + "raptor": ["'power 'osx"], + "awsy": ["'awsy"], + }, + "suites": ["talos", "raptor", "awsy"], + "platform-restrictions": ["desktop"], + "variant-restrictions": { + "raptor": [], + "talos": [], + }, + "app-restrictions": { + "raptor": ["firefox"], + "talos": ["firefox"], + }, + "tasks": [], + }, + +The following fields are available: + * **query**: Set the queries to use for each suite you need. + * **suites**: The suites that are needed for this category. + * **tasks**: A hard-coded list of tasks to select. + * **platform-restrictions**: The platforms that it can run on. + * **app-restrictions**: A list of apps that the category can run. + * **variant-restrictions**: A list of variants available for each suite. + +Note that setting the App/Variant-Restriction fields should be used to restrict the available apps and variants, not expand them as the suites, apps, and platforms combined already provide the largest coverage. The restrictions should be used when you know certain things definitely won't work, or will never be implemented for this category of tests. For instance, our `Resource Usage` tests only work on Firefox even though they may exist in Raptor which can run tests with Chrome. + +Comparators +----------- + +If the standard/default push-to-try comparison is not enough, you can build your own "comparator" that can setup the base, and new revisions. The default comparator `BasePerfComparator` runs the standard mach-try-perf comparison, and there also exists a custom comparator called `BenchmarkComparator` for running custom benchmark comparisons on try (using Github PR links). + +If you'd like to add a custom comparator, you can either create it in a separate file and pass it in the `--comparator`, or add it to the `tools/tryselect/selectors/perfselector/perfcomparators.py` and use the name of the class as the `--comparator` argument (e.g. `--comparator BenchmarkComparator`). You can pass additional arguments to it using the `--comparator-args` option that accepts arguments in the format `NAME=VALUE`. + +The custom comparator needs to be a subclass of `BasePerfComparator`, and optionally overrides its methods. See the comparators file for more information about the interface available. Here's the general interface for it (subject to change), note that the `@comparator` decorator is required when making a builtin comparator:: + + @comparator + class BasePerfComparator: + def __init__(self, vcs, compare_commit, current_revision_ref, comparator_args): + """Initialize the standard/default settings for Comparators. + + :param vcs object: Used for updating the local repo. + :param compare_commit str: The base revision found for the local repo. + :param current_revision_ref str: The current revision of the local repo. + :param comparator_args list: List of comparator args in the format NAME=VALUE. + """ + + def setup_base_revision(self, extra_args): + """Setup the base try run/revision. + + The extra_args can be used to set additional + arguments for Raptor (not available for other harnesses). + + :param extra_args list: A list of extra arguments to pass to the try tasks. + """ + + def teardown_base_revision(self): + """Teardown the setup for the base revision.""" + + def setup_new_revision(self, extra_args): + """Setup the new try run/revision. + + Note that the extra_args are reset between the base, and new revision runs. + + :param extra_args list: A list of extra arguments to pass to the try tasks. + """ + + def teardown_new_revision(self): + """Teardown the new run/revision setup.""" + + def teardown(self): + """Teardown for failures. + + This method can be used for ensuring that the repo is cleaned up + when a failure is hit at any point in the process of doing the + new/base revision setups, or the pushes to try. + """ + +Frequently Asked Questions (FAQ) +-------------------------------- + +If you have any questions which aren't already answered below please reach out to us in the `perftest matrix channel <https://matrix.to/#/#perftest:mozilla.org>`_. + + * **How can I tell what a category or a set of selections will run?** + + At the moment, you need to run your command with an additional option to see what will be run: `./mach try perf --no-push`. See the `Categories`_ section for more information about this. In the future, we plan on having an dynamically updated list for the tasks in the `Categories`_ section of this document. + + * **What's the difference between `Pageload desktop`, and `Pageload desktop firefox`?** + + If you simply ran `./mach try perf` with no additional options, then there is no difference. If you start adding additional browsers to the try run with commands like `./mach try perf --chrome`, then `Pageload desktop` will select all tests available for ALL browsers available, and `Pageload desktop firefox` will only select Firefox tests. When `--chrome` is provided, you'll also see a `Pageload desktop chrome` option. + + * **Help! I can't find a test in any of the categories. What should I do?** + + Use the option `--show-all`. This will let you select tests from the `./mach try fuzzy --full` interface directly instead of the categories. You will always be able to find your tests this way. Please be careful with your task selections though as it's easy to run far too many tests in this way! + +Future Work +----------- + +The future work for this tool can be `found in this bug <https://bugzilla.mozilla.org/show_bug.cgi?id=1799178>`_. Feel free to file improvments, and bugs against it. diff --git a/testing/perfdocs/generated/metrics.rst b/testing/perfdocs/generated/metrics.rst new file mode 100644 index 0000000000..931456467c --- /dev/null +++ b/testing/perfdocs/generated/metrics.rst @@ -0,0 +1,82 @@ +################ +Metrics Gathered +################ + +.. contents:: + :depth: 2 + :local: + +**WARNING: This page is still being actively developed.** + +This document contains information about the metrics gathered in Browsertime tests, as well as detailed information of how they are gathered. + +Pageload Tests +-------------- + +For browsertime pageload tests, there is a limited set of metrics that we collect (which can easily be expanded). Currently we divide these into two sets of metrics: (i) visual metrics, and (ii) technical metrics. These are gathered through two types of tests called warm and cold pageload tests. We have combined these two into a single "chimera" mode which you'll find in the Treeherder tasks. + +Below, you can find the process of how we run Warm, Cold, and Chimera pageload tests. + +Warm Pageload +============== + +In this pageload test type, we open the browser, then repeatedly navigate to the same page without restarting the browser in between cycles. + +* A new, or conditioned, browser profile is created +* The browser is started up +* Post-startup browser settle pause of 30 seconds or 1 second if using a conditioned profile +* A new tab is opened +* The test URL is loaded; measurements taken +* The tab is reloaded ``X`` more times (for ``X`` replicates); measurements taken each time + +NOTES: +- The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts +- The bytecode cache gets populated on the first test cycle, and subsequent iterations will already have the cache built to reduce noise. + +Cold Pageload +============== + +In this pageload test type, we open the browser, navigate to the page, then restart the browser before performing the next cycle. + +* A new, or conditioned, browser profile is created +* The browser is started up +* Post-startup browser settle pause of 30 seconds or 1 second if using a conditioned profile +* A new tab is opened +* The test URL is loaded; measurements taken +* The browser is shut down +* Entire process is repeated for the remaining browser cycles + +NOTE: The measurements from all browser cycles are used to calculate overall results + +Chimera Pageload +================ + +A new mode for pageload testing is called chimera mode. It combines the warm and cold variants into a single test. This test mode is used in our Taskcluster tasks. + +* A new, or conditioned, browser profile is created +* The browser is started up +* Post-startup browser settle pause of 30 seconds or 1 second if using a conditioned profile +* A new tab is opened +* The test URL is loaded; measurements taken for ``Cold pageload`` +* Navigate to a secondary URL (to preserve the content process) +* The test URL is loaded again; measurements taken for ``Warm pageload`` +* The desktop browser is shut down +* Entire process is repeated for the remaining browser cycles + +NOTE: The bytecode cache mentioned in Warm pageloads still applies here. + +Technical Metrics +================= + +These are metrics that are obtained from the browser. This includes metrics like First Paint, DOM Content Flushed, etc.. + +Visual Metrics +============== + +When you run Raptor Browsertime with ``--browsertime-visualmetrics``, it will record a video of the page being loaded and then process this video to build the metrics. The video is either produced using FFMPEG (with ``--browsertime-no-ffwindowrecorder``) or the Firefox Window Recorder (default). + + +Benchmarks +---------- + +Benchmarks gather their own custom metrics unlike the pageload tests above. Please ping the owners of those benchmarks to determine what they mean and how they are produced, or reach out to the Performance Test and Tooling team in #perftest on Element. diff --git a/testing/perfdocs/generated/mozperftest.rst b/testing/perfdocs/generated/mozperftest.rst new file mode 100644 index 0000000000..5859486f0e --- /dev/null +++ b/testing/perfdocs/generated/mozperftest.rst @@ -0,0 +1,246 @@ +=========== +Mozperftest +=========== + +**Mozperftest** can be used to run performance tests. + + +.. toctree:: + + running + tools + writing + developing + vision + +The following documents all testing we have for mozperftest. +If the owner does not specify the Usage and Description, it's marked N/A. + +browser/base/content/test +------------------------- +Performance tests from the 'browser/base/content/test' folder. + +perftest_browser_xhtml_dom.js +============================= + +:owner: Browser Front-end team +:name: Dom-size + +**Measures the size of the DOM** + + +netwerk/test/perf +----------------- +Performance tests from the 'network/test/perf' folder. + +perftest_http3_cloudflareblog.js +================================ + +:owner: Network Team +:name: cloudflare + +**User-journey live site test for cloudflare blog.** + +perftest_http3_controlled.js +============================ + +:owner: Network Team +:name: controlled +:tags: throttlable + +**User-journey live site test for controlled server** + +perftest_http3_facebook_scroll.js +================================= + +:owner: Network Team +:name: facebook-scroll + +**Measures the number of requests per second after a scroll.** + +perftest_http3_google_image.js +============================== + +:owner: Network Team +:name: g-image + +**Measures the number of images per second after a scroll.** + +perftest_http3_google_search.js +=============================== + +:owner: Network Team +:name: g-search + +**User-journey live site test for google search** + +perftest_http3_lucasquicfetch.js +================================ + +:owner: Network Team +:name: lq-fetch + +**Measures the amount of time it takes to load a set of images.** + +perftest_http3_youtube_watch.js +=============================== + +:owner: Network Team +:name: youtube-noscroll + +**Measures quality of the video being played.** + +perftest_http3_youtube_watch_scroll.js +====================================== + +:owner: Network Team +:name: youtube-scroll + +**Measures quality of the video being played.** + + +netwerk/test/unit +----------------- +Performance tests from the 'netwerk/test/unit' folder. + +test_http3_perf.js +================== + +:owner: Network Team +:name: http3 raw +:tags: network,http3,quic +:Default options: + +:: + + --perfherder + --perfherder-metrics name:speed,unit:bps + --xpcshell-cycles 13 + --verbose + --try-platform linux, mac + +**XPCShell tests that verifies the lib integration against a local server** + + +testing/performance +------------------- +Performance tests from the 'testing/performance' folder. + +perftest_bbc_link.js +==================== + +:owner: Performance Team +:name: BBC Link + +**Measures time to load BBC homepage** + +perftest_facebook.js +==================== + +:owner: Performance Team +:name: Facebook + +**Measures time to log in to Facebook** + +perftest_jsconf_cold.js +======================= + +:owner: Performance Team +:name: JSConf (cold) + +**Measures time to load JSConf page (cold)** + +perftest_jsconf_warm.js +======================= + +:owner: Performance Team +:name: JSConf (warm) + +**Measures time to load JSConf page (warm)** + +perftest_politico_link.js +========================= + +:owner: Performance Team +:name: Politico Link + +**Measures time to load Politico homepage** + +perftest_android_view.js +======================== + +:owner: Performance Team +:name: VIEW + +**Measures cold process view time** + +This test launches the appropriate android app, simulating a opening a link through VIEW intent workflow. The application is launched with the intent action android.intent.action.VIEW loading a trivially simple website. The reported metric is the time from process start to navigationStart, reported as processLaunchToNavStart + +perftest_youtube_link.js +======================== + +:owner: Performance Team +:name: YouTube Link + +**Measures time to load YouTube video** + +perftest_android_startup.js +=========================== + +:owner: Performance Team +:name: android-startup + +**Measures android startup times** + +This test consists of 2 main tests, cold main first frame(cmff) and cold view nav start(cvns). cold main first frame is the measurement from when you click the app icon & get duration to first frame from 'am start -W'. cold view nav start is the measurement from when you send a VIEW intent & get duration from logcat: START proc to PageStart. + +perftest_android_main.js +======================== + +:owner: Performance Team +:name: main + +**Measures the time from process start until the Fenix main activity (HomeActivity) reports Fully Drawn** + +This test launches Fenix to its main activity (HomeActivity). The application logs "Fully Drawn" when the activity is drawn. Using the android log transformer we measure the time from process start to this event. + +perftest_pageload.js +==================== + +:owner: Performance Team +:name: pageload + +**Measures time to load mozilla page** + +perftest_perfstats.js +===================== + +:owner: Performance Team +:name: perfstats + +**Collect perfstats for the given site** + +This test launches browsertime with the perfStats option (will collect low-overhead timings, see Bug 1553254). The test currently runs a short user journey. A selection of popular sites are visited, first as cold pageloads, and then as warm. + +perftest_WPT_chrome_init_file.js +================================ + +:owner: Performance Testing Team +:name: webpagetest-chrome + +**Run webpagetest performance pageload tests on Chrome against Alexa top 50 websites** + +This mozperftest gets webpagetest to run pageload tests on Chrome against the 50 most popular websites and provide data. The full list of data returned from webpagetest: firstContentfulPaint, visualComplete90, firstPaint, visualComplete99, visualComplete, SpeedIndex, bytesIn,bytesOut, TTFB, fullyLoadedCPUms, fullyLoadedCPUpct, domElements, domContentLoadedEventStart, domContentLoadedEventEnd, loadEventStart, loadEventEnd + +perftest_WPT_firefox_init_file.js +================================= + +:owner: Performance Testing Team +:name: webpagetest-firefox + +**Run webpagetest performance pageload tests on Firefox against Alexa top 50 websites** + +This mozperftest gets webpagetest to run pageload tests on Firefox against the 50 most popular websites and provide data. The full list of data returned from webpagetest: firstContentfulPaint, timeToContentfulPaint, visualComplete90, firstPaint, visualComplete99, visualComplete, SpeedIndex, bytesIn, bytesOut, TTFB, fullyLoadedCPUms, fullyLoadedCPUpct, domElements, domContentLoadedEventStart, domContentLoadedEventEnd, loadEventStart, loadEventEnd + + +If you have any questions, please see this `wiki page <https://wiki.mozilla.org/TestEngineering/Performance#Where_to_find_us>`_. diff --git a/testing/perfdocs/generated/raptor.rst b/testing/perfdocs/generated/raptor.rst new file mode 100644 index 0000000000..6a2dc3180b --- /dev/null +++ b/testing/perfdocs/generated/raptor.rst @@ -0,0 +1,28404 @@ +###### +Raptor +###### + +Raptor is a performance-testing framework for running browser pageload and browser benchmark tests. Raptor is cross-browser compatible and is currently running in production on Firefox Desktop, Firefox Android GeckoView, Fenix, Reference Browser, Chromium, and Chrome. + +- Contact: Dave Hunt [:davehunt] +- Source code: https://searchfox.org/mozilla-central/source/testing/raptor +- Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor + +Raptor currently supports three test types: 1) page-load performance tests, 2) standard benchmark-performance tests, and 3) "scenario"-based tests, such as power, CPU, and memory-usage measurements on Android (and desktop?). + +Locally, Raptor can be invoked with the following command: + +:: + + ./mach raptor + +**We're in the process of migrating away from webextension to browsertime. Currently, raptor supports both of them, but at the end of the migration, the support for webextension will be removed.** + +.. toctree:: + :titlesonly: + + browsertime + webextension + debugging + contributing + test-list + metrics + +.. contents:: + :depth: 2 + :local: + +Raptor tests +************ + +The following documents all testing we have for Raptor. + +Benchmarks +---------- +Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI. To update any of these benchmarks, see `Updating Benchmark Tests <browsertime.html#updating-benchmark-tests>`_. + + +.. dropdown:: ares6 + :class-container: anchor-id-ares6-b + + **Owner**: :jandem and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 4 + * **page timeout**: 270000 + * **subtest lower is better**: true + * **subtest unit**: ms + * **test url**: `<http://\<host\>:\<port\>/ARES-6/index.html?raptor>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-ares6** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-ares6** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-ares6** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-ares6** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: assorted-dom + :class-container: anchor-id-assorted-dom-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 2000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 60000 + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/assorted-dom + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **screen capture**: true + * **test url**: `<http://\<host\>:\<port\>/assorted/driver.html?raptor>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-assorted-dom** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-assorted-dom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-assorted-dom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-assorted-dom** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: jetstream2 + :class-container: anchor-id-jetstream2-b + + **Owner**: :jandem and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 4 + * **page timeout**: 2000000 + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/JetStream2 + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **subtest lower is better**: false + * **subtest unit**: score + * **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-jetstream2** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-jetstream2** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-jetstream2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-jetstream2** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: matrix-react-bench + :class-container: anchor-id-matrix-react-bench-b + + **Owner**: :jandem and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 30 + * **page timeout**: 2000000 + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/matrix-react-bench + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **subtest lower is better**: true + * **subtest unit**: ms + * **test url**: `<http://\<host\>:\<port\>/matrix_demo.html>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-matrix-react-bench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-matrix-react-bench** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: motionmark-animometer + :class-container: anchor-id-motionmark-animometer-b + + **Owner**: :jgilbert and Graphics(gfx) Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 1 + * **page timeout**: 600000 + * **test url**: `<http://\<host\>:\<port\>/MotionMark/developer.html?test-interval=15&display=minimal&tiles=big&controller=fixed&frame-rate=30&kalman-process-error=1&kalman-measurement-error=4&time-measurement=performance&suite-name=Animometer&raptor=true&oskey={platform}>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-animometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-animometer** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: motionmark-htmlsuite + :class-container: anchor-id-motionmark-htmlsuite-b + + **Owner**: :jgilbert and Graphics(gfx) Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 600000 + * **test url**: `<http://\<host\>:\<port\>/MotionMark/developer.html?test-interval=15&display=minimal&tiles=big&controller=fixed&frame-rate=30&kalman-process-error=1&kalman-measurement-error=4&time-measurement=performance&suite-name=HTMLsuite&raptor=true&oskey={platform}>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-motionmark-htmlsuite** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: speedometer + :class-container: anchor-id-speedometer-b + + **Owner**: SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 180000 + * **subtest lower is better**: true + * **subtest unit**: ms + * **test url**: `<http://\<host\>:\<port\>/Speedometer/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ✅ + - ✅ + - ✅ + - ✅ + + + **Owner**: SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: fenix, geckoview, refbrow, chrome-m + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 420000 + * **subtest lower is better**: true + * **subtest unit**: ms + * **test url**: `<http://\<host\>:\<port\>/Speedometer/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: speedometer3 + :class-container: anchor-id-speedometer3-b + + **Owner**: Performance Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browsertime args**: --browsertime.speedometer_iterations=5 + * **custom data**: true + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **host from parent**: false + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 180000 + * **repository**: https://github.com/WebKit/Speedometer + * **repository revision**: 75cddea7ce4a95c5738a0dd47a2dea7fa56d8db9 + * **submetric summary method**: median + * **subtest lower is better**: true + * **subtest name filters**: tests/,s3/ + * **subtest unit**: ms + * **test script**: speedometer3.js + * **test summary**: flatten + * **test url**: `<http://\<host\>:\<port\>/InteractiveRunner.html>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ✅ + - ✅ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ✅ + - ✅ + - ✅ + - ❌ + * - **browsertime-benchmark-safari-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ✅ + - ✅ + - ✅ + - ❌ + + + **Owner**: Performance Team + + * **alert threshold**: 2.0 + * **apps**: fenix, geckoview, refbrow, chrome-m + * **browsertime args**: --browsertime.speedometer_iterations=5 + * **custom data**: true + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **host from parent**: false + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 420000 + * **repository**: https://github.com/WebKit/Speedometer + * **repository revision**: 75cddea7ce4a95c5738a0dd47a2dea7fa56d8db9 + * **submetric summary method**: median + * **subtest lower is better**: true + * **subtest name filters**: tests/,s3/ + * **subtest unit**: ms + * **test script**: speedometer3.js + * **test summary**: flatten + * **test url**: `<http://\<host\>:\<port\>/InteractiveRunner.html>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ✅ + - ✅ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ✅ + - ✅ + - ✅ + - ❌ + * - **browsertime-benchmark-safari-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-speedometer3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-speedometer3** + - ✅ + - ✅ + - ✅ + - ❌ + + + +.. dropdown:: stylebench + :class-container: anchor-id-stylebench-b + + **Owner**: :emelio and Layout Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 140000 + * **subtest lower is better**: true + * **subtest unit**: ms + * **test url**: `<http://\<host\>:\<port\>/StyleBench/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-stylebench** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-stylebench** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-stylebench** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-stylebench** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: sunspider + :class-container: anchor-id-sunspider-b + + **Owner**: :jandem and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 5 + * **page timeout**: 55000 + * **test url**: `<http://\<host\>:\<port\>/SunSpider/sunspider-1.0.1/sunspider-1.0.1/driver.html?raptor>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-sunspider** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-sunspider** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-benchmark-safari-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-sunspider** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-sunspider** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: twitch-animation + :class-container: anchor-id-twitch-animation-b + + **Owner**: :jrmuizel + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2000000 + * **perfstats**: true + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/twitch-animation + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **subtest lower is better**: true + * **subtest unit**: ms + * **test url**: `<http://\<host\>:\<port\>/index.html>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-twitch-animation** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: unity-webgl + :class-container: anchor-id-unity-webgl-b + + **Owner**: :jgilbert and Graphics(gfx) Team + + * **alert threshold**: 2.0 + * **apps**: geckoview, refbrow, fenix, chrome-m + * **expected**: pass + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 1 + * **page timeout**: 420000 + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/unity-webgl + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ✅ + - ✅ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ✅ + - ✅ + - ✅ + - ❌ + * - **browsertime-benchmark-safari-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ✅ + - ✅ + - ✅ + - ❌ + + + **Owner**: :jgilbert and Graphics(gfx) Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **page cycles**: 5 + * **page timeout**: 420000 + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/unity-webgl + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ✅ + - ✅ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ✅ + - ✅ + - ✅ + - ❌ + * - **browsertime-benchmark-safari-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-unity-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-unity-webgl** + - ✅ + - ✅ + - ✅ + - ❌ + + + +.. dropdown:: wasm-godot + :class-container: anchor-id-wasm-godot-b + + **Owner**: :lth and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **newtab per cycle**: true + * **page cycles**: 5 + * **page timeout**: 120000 + * **test url**: `<http://localhost:\<port\>/wasm-godot/index.html>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-godot** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-godot** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: wasm-godot-baseline + :class-container: anchor-id-wasm-godot-baseline-b + + **Owner**: :lth and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **newtab per cycle**: true + * **page cycles**: 5 + * **page timeout**: 120000 + * **preferences**: {"javascript.options.wasm_baselinejit": true, "javascript.options.wasm_optimizingjit": false} + * **test url**: `<http://localhost:\<port\>/wasm-godot/index.html>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-baseline** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: wasm-godot-optimizing + :class-container: anchor-id-wasm-godot-optimizing-b + + **Owner**: :lth and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 8000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **newtab per cycle**: true + * **page cycles**: 5 + * **page timeout**: 120000 + * **preferences**: {"javascript.options.wasm_baselinejit": false, "javascript.options.wasm_optimizingjit": true} + * **test url**: `<http://localhost:\<port\>/wasm-godot/index.html>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-godot-optimizing** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: wasm-misc + :class-container: anchor-id-wasm-misc-b + + **Owner**: :lth and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 4000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 5 + * **page timeout**: 1200000 + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/wasm-misc + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-chrome-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-chromium-wasm-misc** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-wasm-firefox-wasm-misc** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: wasm-misc-baseline + :class-container: anchor-id-wasm-misc-baseline-b + + **Owner**: :lth and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 4000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 5 + * **page timeout**: 1200000 + * **preferences**: {"javascript.options.wasm_baselinejit": true, "javascript.options.wasm_optimizingjit": false} + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/wasm-misc + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-baseline** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: wasm-misc-optimizing + :class-container: anchor-id-wasm-misc-optimizing-b + + **Owner**: :lth and SpiderMonkey Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 4000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 5 + * **page timeout**: 1200000 + * **preferences**: {"javascript.options.wasm_baselinejit": false, "javascript.options.wasm_optimizingjit": true} + * **repository**: https://github.com/mozilla/perf-automation + * **repository path**: benchmarks/wasm-misc + * **repository revision**: 61332db584026b73e37066d717a162825408c36b + * **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__ + * **type**: benchmark + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-wasm-firefox-wasm-misc-optimizing** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: webaudio + :class-container: anchor-id-webaudio-b + + **Owner**: :padenot and Media Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 4000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 5 + * **page timeout**: 360000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **test url**: `<http://\<host\>:\<port\>/webaudio/index.html?raptor>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-webaudio** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-webaudio** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-chromium-webaudio** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-webaudio** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: youtube-playback + :class-container: anchor-id-youtube-playback-b + + **Owner**: PerfTest Team + + * **alert on**: H264.1080p30@1X_dropped_frames, H264.1080p60@1X_dropped_frames, H264.1440p30@1X_dropped_frames, H264.144p15@1X_dropped_frames, H264.2160p30@1X_dropped_frames, H264.240p30@1X_dropped_frames, H264.360p30@1X_dropped_frames, H264.480p30@1X_dropped_frames, H264.720p30@1X_dropped_frames, H264.720p60@1X_dropped_frames, VP9.1080p30@1X_dropped_frames, VP9.1080p60@1X_dropped_frames, VP9.1440p30@1X_dropped_frames, VP9.1440p60@1X_dropped_frames, VP9.144p30@1X_dropped_frames, VP9.2160p30@1X_dropped_frames, VP9.2160p60@1X_dropped_frames, VP9.240p30@1X_dropped_frames, VP9.360p30@1X_dropped_frames, VP9.480p30@1X_dropped_frames, VP9.720p30@1X_dropped_frames, VP9.720p60@1X_dropped_frames + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix,refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + +.. dropdown:: youtube-playback-av1-sfr + :class-container: anchor-id-youtube-playback-av1-sfr-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-av1-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: youtube-playback-h264-1080p30 + :class-container: anchor-id-youtube-playback-h264-1080p30-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-h264-1080p60 + :class-container: anchor-id-youtube-playback-h264-1080p60-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-h264-full-1080p30 + :class-container: anchor-id-youtube-playback-h264-full-1080p30-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "full-screen-api.allow-trusted-requests-only": false, "full-screen-api.warning.timeout": 0} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&fullscreen=true&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-h264-full-1080p60 + :class-container: anchor-id-youtube-playback-h264-full-1080p60-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "full-screen-api.allow-trusted-requests-only": false, "full-screen-api.warning.timeout": 0} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&fullscreen=true&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-h264-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-h264-sfr + :class-container: anchor-id-youtube-playback-h264-sfr-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-h264-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: youtube-playback-hfr + :class-container: anchor-id-youtube-playback-hfr-b + + **Owner**: PerfTest Team + + * **alert on**: H2641080p60fps@1X_dropped_frames, H264720p60fps@1X_dropped_frames + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-hfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: youtube-playback-v9-1080p30 + :class-container: anchor-id-youtube-playback-v9-1080p30-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-v9-1080p60 + :class-container: anchor-id-youtube-playback-v9-1080p60-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-v9-full-1080p30 + :class-container: anchor-id-youtube-playback-v9-full-1080p30-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "full-screen-api.allow-trusted-requests-only": false, "full-screen-api.warning.timeout": 0} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&fullscreen=true&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p30** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-v9-full-1080p60 + :class-container: anchor-id-youtube-playback-v9-full-1080p60-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **expose gecko profiler**: true + * **gather cpuTime**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 20 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "full-screen-api.allow-trusted-requests-only": false, "full-screen-api.warning.timeout": 0} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&fullscreen=true&exclude=1,2>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-power-firefox-youtube-playback-v9-full-1080p60** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-playback-vp9-sfr + :class-container: anchor-id-youtube-playback-vp9-sfr-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-vp9-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: youtube-playback-widevine-h264-sfr + :class-container: anchor-id-youtube-playback-widevine-h264-sfr-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "media.eme.enabled": true, "media.gmp-manager.updateEnabled": true, "media.eme.require-app-approval": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-h264-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: youtube-playback-widevine-hfr + :class-container: anchor-id-youtube-playback-widevine-hfr-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "media.eme.enabled": true, "media.gmp-manager.updateEnabled": true, "media.eme.require-app-approval": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-hfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + +.. dropdown:: youtube-playback-widevine-vp9-sfr + :class-container: anchor-id-youtube-playback-widevine-vp9-sfr-b + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, geckoview, fenix, refbrow, chrome + * **expected**: pass + * **expose gecko profiler**: true + * **gecko profile entries**: 50000000 + * **gecko profile interval**: 1000 + * **gecko profile threads**: MediaPlayback + * **lower is better**: true + * **page cycles**: 1 + * **page timeout**: 2700000 + * **preferences**: {"media.autoplay.default": 0, "media.autoplay.ask-permission": false, "media.autoplay.blocking_policy": 0, "media.autoplay.block-webaudio": false, "media.allowed-to-play.enabled": true, "media.block-autoplay-until-in-foreground": false, "media.eme.enabled": true, "media.gmp-manager.updateEnabled": true, "media.eme.require-app-approval": false} + * **subtest lower is better**: true + * **subtest unit**: score + * **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run>`__ + * **type**: benchmark + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-firefox-youtube-playback-widevine-vp9-sfr** + - ✅ + - ❌ + - ✅ + - ❌ + + + + +Custom +------ +Browsertime tests that use a custom pageload test script. These use the pageload type, but may have other intentions. + +.. dropdown:: browsertime + :class-container: anchor-id-browsertime-c + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime + * **alert threshold**: 2.0 + * **apps**: firefox + * **browser cycles**: 1 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **measure**: fnbpaint, fcp, dcf, loadtime + * **page cycles**: 1 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: null.manifest + * **playback version**: 8.1.1 + * **test script**: None + * **test url**: `<None>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: constant-regression + :class-container: anchor-id-constant-regression-c + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, geckoview, fenix + * **browser cycles**: 1 + * **browsertime args**: --browsertime.constant_value=1500 + * **custom data**: true + * **expected**: pass + * **fore window**: 1 + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **max back window**: 3 + * **min back window**: 2 + * **output timeout**: 2000000 + * **page cycles**: 1 + * **page timeout**: 1800000 + * **subtest unit**: ms + * **test script**: constant_regression_test.js + * **test summary**: flatten + * **test url**: `<None>`__ + * **type**: pageload + * **unit**: score + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-regression-tests-firefox-constant-regression** + - ❌ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-regression-tests-firefox-constant-regression** + - ❌ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: grandprix + :class-container: anchor-id-grandprix-c + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 1 + * **browsertime args**: --browsertime.grandprix_iterations=25 + * **custom data**: true + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **output timeout**: 2000000 + * **page cycles**: 1 + * **page timeout**: 1800000 + * **subtest name filters**: tests/,iterations/ + * **subtest unit**: ms + * **test script**: grandprix.js + * **test summary**: flatten + * **test url**: `<None>`__ + * **type**: pageload + * **unit**: score + * **use live sites**: true + +.. dropdown:: process-switch + :class-container: anchor-id-process-switch-c + + **Owner**: PerfTest Team + + * **accept zero vismet**: true + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium + * **browser cycles**: 25 + * **browsertime args**: --pageCompleteWaitTime=1000 --pageCompleteCheckInactivity=true + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-proc-switch.manifest + * **playback version**: 5.1.1 + * **test script**: process_switch.js + * **test url**: `<https://mozilla.seanfeng.dev/files/red.html,https://mozilla.pettay.fi/moztests/blue.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-custom-firefox-process-switch** + - ✅ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: sample-python-support + :class-container: anchor-id-sample-python-support-c + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: firefox + * **browser cycles**: 1 + * **browsertime args**: --browsertime.constant_value={replace-with-constant-value} + * **custom data**: true + * **expected**: pass + * **fore window**: 1 + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **max back window**: 3 + * **min back window**: 2 + * **output timeout**: 2000000 + * **page cycles**: 1 + * **page timeout**: 1800000 + * **subtest unit**: ms + * **support class**: sample_python_support.py + * **test script**: constant_regression_test.js + * **test summary**: flatten + * **test url**: `<None>`__ + * **type**: pageload + * **unit**: score + * **use live sites**: true + +.. dropdown:: upload + :class-container: anchor-id-upload-c + + **Owner**: Network Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 1 + * **browsertime args**: --browsertime.upload_iterations=10 --firefox.preference=network.http.http3.enable:false --chrome.args disable-quic + * **custom data**: true + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **output timeout**: 2000000 + * **page cycles**: 1 + * **page timeout**: 1800000 + * **subtest lower is better**: false + * **subtest name filters**: tests/,iterations/ + * **subtest unit**: mbps + * **test script**: upload.js + * **test url**: `<None>`__ + * **type**: pageload + * **unit**: mbps + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-upload-safari-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: upload-h3 + :class-container: anchor-id-upload-h3-c + + **Owner**: Network Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 1 + * **browsertime args**: --browsertime.upload_iterations=10 --firefox.preference=network.http.http3.enable:true + * **custom data**: true + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: false + * **output timeout**: 2000000 + * **page cycles**: 1 + * **page timeout**: 1800000 + * **subtest lower is better**: false + * **subtest name filters**: tests/,iterations/ + * **subtest unit**: mbps + * **test script**: upload.js + * **test url**: `<None>`__ + * **type**: pageload + * **unit**: mbps + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload-h3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload-h3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-upload-safari-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-firefox-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-upload-chrome-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-chromium-upload-h3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-upload-firefox-upload-h3** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: welcome + :class-container: anchor-id-welcome-c + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-welcome.manifest + * **playback version**: 8.1.1 + * **test script**: welcome.js + * **test url**: `<about:welcome>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-first-install-firefox-welcome** + - ✅ + - ✅ + - ❌ + - ❌ + + + + +Desktop +------- +Tests for page-load performance. The links direct to the actual websites that are being tested. + +.. dropdown:: amazon + :class-container: anchor-id-amazon-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-amazon.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.amazon.com/Acer-A515-46-R14K-Quad-Core-Processor-Backlit/dp/B08VKNVDDR/ref=sr_1_3?dchild=1&keywords=laptop&qid=1627047187&sr=8-3>`__ + * **test url**: `<https://www.amazon.com/s?k=laptop&ref=nb_sb_noss_1>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-profiling-firefox-amazon** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-profiling-firefox-amazon** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: bing-search + :class-container: anchor-id-bing-search-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-bing-search.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.bing.com/search?q=barack+obama>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-bing-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-bing-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-bing-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: buzzfeed + :class-container: anchor-id-buzzfeed-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-buzzfeed.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.buzzfeed.com/quizzes>`__ + * **test url**: `<https://www.buzzfeed.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-buzzfeed** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-buzzfeed** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-buzzfeed** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-buzzfeed** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: cnn + :class-container: anchor-id-cnn-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-cnn.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.cnn.com/weather>`__ + * **test url**: `<https://www.cnn.com/2021/03/22/weather/climate-change-warm-waters-lake-michigan/index.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: ebay + :class-container: anchor-id-ebay-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-ebay.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.ebay.com/deals>`__ + * **test url**: `<https://www.ebay.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-ebay** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-ebay** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-ebay** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-ebay** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: espn + :class-container: anchor-id-espn-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-espn.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.espn.com/nba/draft/news>`__ + * **test url**: `<http://www.espn.com/nba/story/_/page/allstarweekend25788027/the-comparison-lebron-james-michael-jordan-their-own-words>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: expedia + :class-container: anchor-id-expedia-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-expedia.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://groups.expedia.com/Group-Rate/?locale=en_US&ol=1>`__ + * **test url**: `<https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470®ionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: facebook + :class-container: anchor-id-facebook-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-facebook.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.facebook.com/marketplace/?ref=bookmark>`__ + * **test url**: `<https://www.facebook.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: fandom + :class-container: anchor-id-fandom-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-fandom.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-fandom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-fandom** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-fandom** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google-docs + :class-container: anchor-id-google-docs-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page complete wait time**: 8000 + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-google-docs.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://docs.google.com/document/d/1vUnn0ePU-ynArE1OdxyEHXR2G0sl74ja_st_4OOzlgE/preview>`__ + * **test url**: `<https://docs.google.com/document/d/1US-07msg12slQtI_xchzYxcKlTs6Fp7WqIc6W5GK5M8/edit?usp=sharing>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-docs** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-docs** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-docs** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-docs** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google-mail + :class-container: anchor-id-google-mail-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-google-mail.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://mail.google.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google-search + :class-container: anchor-id-google-search-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-google-search.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.google.com/search?hl=en&q=barack+obama&cad=h>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-google-search** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google-slides + :class-container: anchor-id-google-slides-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-google-slides.manifest + * **playback version**: 6.0.2 + * **secondary url**: `<https://docs.google.com/document/d/1vUnn0ePU-ynArE1OdxyEHXR2G0sl74ja_st_4OOzlgE/preview>`__ + * **test url**: `<https://docs.google.com/presentation/d/1Ici0ceWwpFvmIb3EmKeWSq_vAQdmmdFcWqaiLqUkJng/edit?usp=sharing>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-google-slides** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-google-slides** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-google-slides** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: imdb + :class-container: anchor-id-imdb-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-imdb.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.imdb.com/title/tt0084967/episodes/?ref_=tt_ov_epl>`__ + * **test url**: `<https://www.imdb.com/title/tt0084967/?ref_=nv_sr_2>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: imgur + :class-container: anchor-id-imgur-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm8-linux-firefox-imgur.manifest + * **playback version**: 8.1.1 + * **preferences**: {"media.autoplay.default": 5, "media.autoplay.ask-permission": true, "media.autoplay.blocking_policy": 1, "media.autoplay.block-webaudio": true, "media.allowed-to-play.enabled": false, "media.block-autoplay-until-in-foreground": true} + * **secondary url**: `<https://imgur.com/gallery/rCXZUil>`__ + * **test url**: `<https://imgur.com/gallery/m5tYJL6>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imgur** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imgur** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imgur** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chromium-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imgur** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: instagram + :class-container: anchor-id-instagram-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-instagram.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.instagram.com/nobelprize_org/>`__ + * **test url**: `<https://www.instagram.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: linkedin + :class-container: anchor-id-linkedin-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-linkedin.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.linkedin.com/in/thommy-harris-hk-385723106/>`__ + * **test url**: `<https://www.linkedin.com/feed/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-linkedin** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-linkedin** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-linkedin** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-linkedin** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: microsoft + :class-container: anchor-id-microsoft-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-microsoft.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://support.microsoft.com/en-us>`__ + * **test url**: `<https://www.microsoft.com/en-us/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-microsoft** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-microsoft** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-microsoft** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-microsoft** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: netflix + :class-container: anchor-id-netflix-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-netflix.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.netflix.com/title/699257>`__ + * **test url**: `<https://www.netflix.com/title/80117263>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-netflix** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-netflix** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-netflix** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: nytimes + :class-container: anchor-id-nytimes-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-nytimes.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.nytimes.com/section/opinion/columnists>`__ + * **test url**: `<https://www.nytimes.com/2020/02/19/opinion/surprise-medical-bill.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: office + :class-container: anchor-id-office-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-office.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.office.com/>`__ + * **test url**: `<https://www.office.com/launch/powerpoint?auth=1>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-office** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-office** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-office** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-office** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-office** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: outlook + :class-container: anchor-id-outlook-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-live.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://outlook.live.com/mail/inbox>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-outlook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-outlook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-outlook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-outlook** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: paypal + :class-container: anchor-id-paypal-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-paypal.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.paypal.com/myaccount/summary/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-paypal** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-paypal** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-paypal** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-paypal** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: pinterest + :class-container: anchor-id-pinterest-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-pinterest.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.pinterest.com/today/best/halloween-costumes-for-your-furry-friends/75787/>`__ + * **test url**: `<https://pinterest.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-pinterest** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-pinterest** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-pinterest** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-pinterest** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: reddit + :class-container: anchor-id-reddit-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-reddit.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.reddit.com/r/technology/>`__ + * **test url**: `<https://www.reddit.com/r/technology/comments/9sqwyh/we_posed_as_100_senators_to_run_ads_on_facebook/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: tumblr + :class-container: anchor-id-tumblr-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-linux-firefox-tumblr.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.tumblr.com/tagged/funny+cats?sort=top>`__ + * **test url**: `<https://www.tumblr.com/dashboard>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-tumblr** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-tumblr** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-tumblr** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-tumblr** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: twitch + :class-container: anchor-id-twitch-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-twitch.manifest + * **playback version**: 8.1.1 + * **preferences**: {"media.autoplay.default": 5, "media.autoplay.ask-permission": true, "media.autoplay.blocking_policy": 1, "media.autoplay.block-webaudio": true, "media.allowed-to-play.enabled": false, "media.block-autoplay-until-in-foreground": true} + * **secondary url**: `<https://www.twitch.tv/gmashley>`__ + * **test url**: `<https://www.twitch.tv/videos/894226211>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-twitch** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-twitch** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-twitch** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: twitter + :class-container: anchor-id-twitter-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-twitter.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://twitter.com/BarackObama>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-twitter** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-twitter** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-twitter** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: wikia + :class-container: anchor-id-wikia-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-wikia.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://marvel.fandom.com/wiki/Celestials>`__ + * **test url**: `<https://marvel.fandom.com/wiki/Black_Panther>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-wikia** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-wikia** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-wikia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-wikia** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: wikipedia + :class-container: anchor-id-wikipedia-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-wikipedia.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://en.wikipedia.org/wiki/Joe_Biden>`__ + * **test url**: `<https://en.wikipedia.org/wiki/Barack_Obama>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: yahoo-mail + :class-container: anchor-id-yahoo-mail-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm5-linux-firefox-yahoo-mail.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://mail.yahoo.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-yahoo-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-yahoo-mail** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-yahoo-mail** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube + :class-container: anchor-id-youtube-d + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 25 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-youtube.manifest + * **playback version**: 8.1.1 + * **secondary url**: `<https://www.youtube.com/watch?v=JrdEMERq8MA>`__ + * **test url**: `<https://www.youtube.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + + +Interactive +----------- +Browsertime tests that interact with the webpage. Includes responsiveness tests as they make use of this support for navigation. These form of tests allow the specification of browsertime commands through the test manifest. + +.. dropdown:: cnn-nav + :class-container: anchor-id-cnn-nav-i + + **Owner**: PerfTest Team + + * **accept zero vismet**: true + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 10 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **interactive**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-cnn-nav.manifest + * **playback version**: 8.1.1 + * **test cmds**: ["measure.start", "landing"], ["navigate", "https://www.cnn.com"], ["wait.byTime", 4000], ["measure.stop", ""], ["measure.start", "world"], ["click.byXpathAndWait", "/html/body/div[5]/div/div/header/div/div[1]/div/div[2]/nav/ul/li[2]/a"], ["wait.byTime", 1000], ["measure.stop", ""], + * **test url**: `<https://www.cnn.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-cnn-nav** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-cnn-nav** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-cnn-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-cnn-nav** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: facebook-nav + :class-container: anchor-id-facebook-nav-i + + **Owner**: PerfTest Team + + * **accept zero vismet**: true + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 10 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **interactive**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 90000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-windows-firefox-facebook-nav.manifest + * **playback version**: 8.1.1 + * **test cmds**: ["measure.start", "landing"], ["navigate", "https://www.facebook.com/"], ["wait.byTime", 5000], ["measure.stop", ""], ["measure.start", "marketplace"], ["navigate", "https://www.facebook.com/marketplace"], ["wait.byTime", 5000], ["measure.stop", ""], ["measure.start", "groups"], ["navigate", "https://www.facebook.com/groups/discover/"], ["wait.byTime", 5000], ["measure.stop", ""], ["measure.start", "friends"], ["navigate", "https://www.facebook.com/friends/"], ["wait.byTime", 5000], ["measure.stop", ""], + * **test url**: `<https://www.facebook.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-facebook-nav** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-facebook-nav** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-facebook-nav** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-facebook-nav** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: reddit-billgates-ama + :class-container: anchor-id-reddit-billgates-ama-i + + **Owner**: PerfTest Team + + * **accept zero vismet**: true + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 10 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **interactive**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 240000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-windows-firefox-reddit-billgates-ama.manifest + * **playback version**: 8.1.1 + * **test cmds**: ["measure.start", "billg-ama"], ["navigate", "https://www.reddit.com/r/IAmA/comments/m8n4vt/im_bill_gates_cochair_of_the_bill_and_melinda/"], ["wait.byTime", 5000], ["measure.stop", ""], ["measure.start", "members"], ["click.byXpathAndWait", "/html/body/div[1]/div/div[2]/div[2]/div/div[3]/div[2]/div/div[1]/div/div[4]/div[1]/div"], ["wait.byTime", 1000], ["measure.stop", ""], + * **test url**: `<https://www.reddit.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-ama** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-ama** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: reddit-billgates-post-1 + :class-container: anchor-id-reddit-billgates-post-1-i + + **Owner**: PerfTest Team + + * **accept zero vismet**: true + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 10 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **interactive**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 90000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-windows-firefox-reddit-billgates-post.manifest + * **playback version**: 8.1.1 + * **test cmds**: ["measure.start", "billg"], ["navigate", "https://www.reddit.com/user/thisisbillgates/"], ["wait.byTime", 500], ["measure.stop", ""], ["measure.start", "posts"], ["click.byXpathAndWait", "/html/body/div[1]/div/div[2]/div[2]/div/div/div/div[2]/div[2]/div/div/div/a[2]"], ["wait.byTime", 500], ["measure.stop", ""], ["measure.start", "comments"], ["click.byXpathAndWait", "/html/body/div[1]/div/div[2]/div[2]/div/div/div/div[2]/div[2]/div/div/div/a[3]"], ["wait.byTime", 500], ["measure.stop", ""], ["wait.byTime", 500], + * **test url**: `<https://www.reddit.com/user/thisisbillgates/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-1** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-1** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: reddit-billgates-post-2 + :class-container: anchor-id-reddit-billgates-post-2-i + + **Owner**: PerfTest Team + + * **accept zero vismet**: true + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari + * **browser cycles**: 10 + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **interactive**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 90000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm6-windows-firefox-reddit-billgates-post.manifest + * **playback version**: 8.1.1 + * **test cmds**: ["measure.start", "billg"], ["navigate", "https://www.reddit.com/user/thisisbillgates/"], ["wait.byTime", 500], ["measure.stop", ""], ["click.byXpathAndWait", "/html/body/div[1]/div/div[2]/div[2]/div/div/div/div[2]/div[2]/div/div/div/a[3]"], ["wait.byTime", 500], ["measure.start", "hot"], ["click.byXpathAndWait", "/html/body/div[1]/div/div[2]/div[2]/div/div/div/div[2]/div[4]/div[1]/div[1]/div[2]/a[2]"], ["wait.byTime", 500], ["measure.stop", ""], ["measure.start", "top"], ["click.byXpathAndWait", "/html/body/div[1]/div/div[2]/div[2]/div/div/div/div[2]/div[4]/div[1]/div[1]/div[2]/a[3]"], ["wait.byTime", 500], ["measure.stop", ""], ["wait.byTime", 500], + * **test url**: `<https://www.reddit.com/user/thisisbillgates/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-responsiveness-chrome-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-chromium-reddit-billgates-post-2** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-responsiveness-firefox-reddit-billgates-post-2** + - ✅ + - ✅ + - ❌ + - ❌ + + + + +Live +---- +A set of test pages that are run as live sites instead of recorded versions. These tests are available on all browsers, on all platforms. + +.. dropdown:: booking-sf + :class-container: anchor-id-booking-sf-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://www.booking.com/hotel/us/edwardian-san-francisco.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: cnn + :class-container: anchor-id-cnn-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://www.cnn.com/2021/03/22/weather/climate-change-warm-waters-lake-michigan/index.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: cnn-ampstories + :class-container: anchor-id-cnn-ampstories-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: discord + :class-container: anchor-id-discord-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://discordapp.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: expedia + :class-container: anchor-id-expedia-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470®ionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-expedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-expedia** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: fashionbeans + :class-container: anchor-id-fashionbeans-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://fashionbeans.com/article/coolest-menswear-stores-in-the-world>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: google-accounts + :class-container: anchor-id-google-accounts-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://accounts.google.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: imdb-firefox + :class-container: anchor-id-imdb-firefox-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://m.imdb.com/title/tt0083943/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: medium-article + :class-container: anchor-id-medium-article-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://medium.com/s/coincidences-are-a-lie/could-america-have-also-been-the-birthplace-of-impressionism-cb3d31a2e22d>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: nytimes + :class-container: anchor-id-nytimes-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://www.nytimes.com/2020/02/19/opinion/surprise-medical-bill.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-nytimes** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-nytimes** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: people-article + :class-container: anchor-id-people-article-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://people.com/amp-stories/royal-a-to-z/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: reddit-thread + :class-container: anchor-id-reddit-thread-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://www.reddit.com/r/firefox/comments/7dkq03/its_been_a_while/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: rumble-fox + :class-container: anchor-id-rumble-fox-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://rumble.com/v3c44t-foxes-jumping-on-my-trampoline.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: stackoverflow-question + :class-container: anchor-id-stackoverflow-question-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://stackoverflow.com/questions/927358/how-do-i-undo-the-most-recent-commits-in-git>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: urbandictionary-define + :class-container: anchor-id-urbandictionary-define-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://urbandictionary.com/define.php?term=awesome%20sauce>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + +.. dropdown:: wikia-marvel + :class-container: anchor-id-wikia-marvel-l + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m, firefox, chrome, chromium + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **test url**: `<https://marvel.wikia.com/wiki/Black_Panther>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: true + + +Mobile +------ +Page-load performance test suite on Android. The links direct to the actual websites that are being tested. + +.. dropdown:: allrecipes + :class-container: anchor-id-allrecipes-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-allrecipes.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.allrecipes.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + +.. dropdown:: amazon + :class-container: anchor-id-amazon-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-amazon.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.amazon.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-profiling-firefox-amazon** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-profiling-firefox-amazon** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: amazon-search + :class-container: anchor-id-amazon-search-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-amazon-search.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.amazon.com/s/ref=nb_sb_noss_2/139-6317191-5622045?url=search-alias%3Daps&field-keywords=mobile+phone>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + +.. dropdown:: bing + :class-container: anchor-id-bing-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-bing.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.bing.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-bing** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bing** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-bing** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-bing** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-bing** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bing** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-bing** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: bing-search-restaurants + :class-container: anchor-id-bing-search-restaurants-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-bing-search-restaurants.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.bing.com/search?q=restaurants+in+exton+pa+19341>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-bing-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: booking + :class-container: anchor-id-booking-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-booking.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.booking.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-booking** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-booking** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-booking** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-booking** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-booking** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-booking** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-booking** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: cnn + :class-container: anchor-id-cnn-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-cnn.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://cnn.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-cnn** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-sheriffed-firefox-cnn** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: cnn-ampstories + :class-container: anchor-id-cnn-ampstories-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-cnn-ampstories.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-cnn-ampstories** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: dailymail + :class-container: anchor-id-dailymail-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-dailymail.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.dailymail.co.uk/sciencetech/article-9749081/Experts-say-Hubble-repair-despite-NASA-insisting-multiple-options-fix.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-dailymail** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: ebay-kleinanzeigen + :class-container: anchor-id-ebay-kleinanzeigen-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-ebay-kleinanzeigen.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://m.ebay-kleinanzeigen.de>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-ebay-kleinanzeigen** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: ebay-kleinanzeigen-search + :class-container: anchor-id-ebay-kleinanzeigen-search-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-ebay-kleinanzeigen-search.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-ebay-kleinanzeigen-search** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: espn + :class-container: anchor-id-espn-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-espn.manifest + * **playback version**: 8.1.1 + * **test url**: `<http://www.espn.com/nba/story/_/page/allstarweekend25788027/the-comparison-lebron-james-michael-jordan-their-own-words>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-espn** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-espn** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-espn** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: facebook + :class-container: anchor-id-facebook-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **login**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-g5-fenix-facebook.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://m.facebook.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: facebook-cristiano + :class-container: anchor-id-facebook-cristiano-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-facebook-cristiano.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://m.facebook.com/Cristiano>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-facebook-cristiano** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google + :class-container: anchor-id-google-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **login**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-g5-fenix-google.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.google.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + +.. dropdown:: google-maps + :class-container: anchor-id-google-maps-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-google-maps.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.google.com/maps?force=pwa>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-google-maps** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google-search-restaurants + :class-container: anchor-id-google-search-restaurants-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **login**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-g5-fenix-google-search-restaurants.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.google.com/search?q=restaurants+near+me>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-google-search-restaurants** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: imdb + :class-container: anchor-id-imdb-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-imdb.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://m.imdb.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-imdb** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-imdb** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: instagram + :class-container: anchor-id-instagram-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **login**: true + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm8-android-gve-instagram.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.instagram.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-instagram** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-instagram** + - ✅ + - ✅ + - ✅ + - ✅ + + + +.. dropdown:: microsoft-support + :class-container: anchor-id-microsoft-support-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-microsoft-support.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://support.microsoft.com/en-us>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + +.. dropdown:: reddit + :class-container: anchor-id-reddit-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-reddit.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.reddit.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-reddit** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-reddit** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: sina + :class-container: anchor-id-sina-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-sina.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.sina.com.cn/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-sina** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-sina** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-sina** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-sina** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-sina** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-sina** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-sina** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: stackoverflow + :class-container: anchor-id-stackoverflow-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-stackoverflow.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://stackoverflow.com/>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-stackoverflow** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: web-de + :class-container: anchor-id-web-de-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm7-android-gve-p2-web-de.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://web.de/magazine/politik/politologe-glaubt-grossen-koalition-herbst-knallen-33563566>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-web-de** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: wikipedia + :class-container: anchor-id-wikipedia-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-wikipedia.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://en.m.wikipedia.org/wiki/Main_Page>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-wikipedia** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-wikipedia** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube + :class-container: anchor-id-youtube-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-youtube.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://m.youtube.com>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: youtube-watch + :class-container: anchor-id-youtube-watch-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm6-android-fenix-youtube-watch.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.youtube.com/watch?v=COU5T-Wafa4>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + + +Scenario +-------- +Tests that perform a specific action (a scenario), i.e. idle application, idle application in background, etc. + +.. dropdown:: idle + :class-container: anchor-id-idle-s + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: fenix, geckoview, refbrow + * **expected**: pass + * **lower is better**: true + * **measure**: fakeMeasure + * **page cycles**: 1 + * **page timeout**: 1320000 + * **scenario time**: 1200000 + * **test url**: `<about:blank>`__ + * **type**: scenario + * **unit**: scenarioComplete + +.. dropdown:: idle-bg + :class-container: anchor-id-idle-bg-s + + **Owner**: PerfTest Team + + * **alert threshold**: 2.0 + * **apps**: fenix, geckoview, refbrow + * **browsertime args**: --browsertime.scenario_time=60000 --browsertime.background_app=false + * **expected**: pass + * **lower is better**: true + * **measure**: fakeMeasure + * **page cycles**: 1 + * **page timeout**: 1320000 + * **scenario time**: 1200000 + * **test url**: `<about:blank>`__ + * **type**: scenario + * **unit**: scenarioComplete + + +Unittests +--------- +These tests aren't used in standard testing, they are only used in the Raptor unit tests (they are similar to raptor-tp6 tests though). + +.. dropdown:: amazon + :class-container: anchor-id-amazon-u + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-amazon.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.amazon.com/s?k=laptop&ref=nb_sb_noss_1>`__ + * **type**: pageload + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-profiling-firefox-amazon** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-profiling-firefox-amazon** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-bytecode-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-essential-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-custom-car-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-essential-firefox-amazon** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-amazon** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: facebook + :class-container: anchor-id-facebook-u + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-facebook.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.facebook.com>`__ + * **type**: pageload + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-facebook** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-facebook** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: google + :class-container: anchor-id-google-u + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-google.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.google.com/search?hl=en&q=barack+obama&cad=h>`__ + * **type**: pageload + * **unit**: ms + +.. dropdown:: youtube + :class-container: anchor-id-youtube-u + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime + * **alert threshold**: 2.0 + * **apps**: firefox + * **expected**: pass + * **gecko profile entries**: 14000000 + * **gecko profile interval**: 1 + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy + * **playback pageset manifest**: mitm7-linux-firefox-youtube.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.youtube.com>`__ + * **type**: pageload + * **unit**: ms + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-safari-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-firefox-youtube** + - ✅ + - ✅ + - ✅ + - ✅ + * - **browsertime-tp6-live-chrome-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-chromium-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6-live-firefox-youtube** + - ❌ + - ❌ + - ❌ + - ❌ + + + + diff --git a/testing/perfdocs/generated/running.rst b/testing/perfdocs/generated/running.rst new file mode 100644 index 0000000000..ed8d9947a9 --- /dev/null +++ b/testing/perfdocs/generated/running.rst @@ -0,0 +1,51 @@ +Running a performance test +========================== + +You can run `perftest` locally or in Mozilla's CI + +Running locally +--------------- + +Running a test is as simple as calling it using `mach perftest` in a mozilla-central source +checkout:: + + $ ./mach perftest + +The `mach` command will bootstrap the installation of all required tools for the +framework to run, and display a selection screen to pick a test. Once the +selection is done, the performance test will run locally. + +If you know what test you want to run, you can use its path explicitly:: + + $ ./mach perftest perftest_script.js + +`mach perftest` comes with numerous options, and the test script should provide +decent defaults so you don't have to bother with them. If you need to tweak some +options, you can use `./mach perftest --help` to learn about them. + + +Running in the CI +----------------- + +.. warning:: + + If you are looking for how to run performance tests in CI and ended up here, you might want to checkout :ref:`Mach Try Perf`. + +.. warning:: + + If you plan to run tests often in the CI for android, you should contact the android + infra team to make sure there's availability in our pool of devices. + +You can run in the CI directly from the `mach perftest` command by adding the `--push-to-try` option +to your locally working perftest call. + +This call will run the fuzzy selector and then send the job into our CI:: + + $ ./mach perftest --push-to-try + +We have phones on bitbar that can run your android tests. Tests are fairly fast +to run in the CI because they use sparse profiles. Depending on the +availability of workers, once the task starts, it takes around 15mn to start +the test. + + diff --git a/testing/perfdocs/generated/standard-try-perf.png b/testing/perfdocs/generated/standard-try-perf.png Binary files differnew file mode 100644 index 0000000000..d1168665fb --- /dev/null +++ b/testing/perfdocs/generated/standard-try-perf.png diff --git a/testing/perfdocs/generated/talos.rst b/testing/perfdocs/generated/talos.rst new file mode 100644 index 0000000000..fd49186ac2 --- /dev/null +++ b/testing/perfdocs/generated/talos.rst @@ -0,0 +1,7647 @@ +===== +Talos +===== + +Talos is a cross-platform Python performance testing framework that is specifically for +Firefox on desktop. New performance tests should be added to the newer framework +`mozperftest </testing/perfdocs/mozperftest.html>`_ unless there are limitations +there (highly unlikely) that make it absolutely necessary to add them to Talos. Talos is +named after the `bronze automaton from Greek myth <https://en.wikipedia.org/wiki/Talos>`_. + +.. contents:: + :depth: 1 + :local: + +Talos tests are run in a similar manner to xpcshell and mochitests. They are started via +the command :code:`mach talos-test`. A `python script <https://searchfox.org/mozilla-central/source/testing/talos>`_ +then launches Firefox, which runs the tests via JavaScript special powers. The test timing +information is recorded in a text log file, e.g. :code:`browser_output.txt`, and then processed +into the `JSON format supported by Perfherder <https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json>`_. + +Talos bugs can be filed in `Testing::Talos <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos>`_. + +Talos infrastructure is still mostly documented `on the Mozilla Wiki <https://wiki.mozilla.org/TestEngineering/Performance/Talos>`_. +In addition, there are plans to surface all of the individual tests using PerfDocs. +This work is tracked in `Bug 1674220 <https://bugzilla.mozilla.org/show_bug.cgi?id=1674220>`_. + +Examples of current Talos runs can be `found in Treeherder by searching for "Talos" <https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=Talos>`_. +If none are immediately available, then scroll to the bottom of the page and load more test +runs. The tests all share a group symbol starting with a :code:`T`, for example +:code:`T(c d damp g1)` or :code:`T-gli(webgl)`. + +Running Talos Locally +********************* + +Running tests locally is most likely only useful for debugging what is going on in a test, +as the test output is only reported as raw JSON. The CLI is documented via: + +.. code-block:: bash + + ./mach talos-test --help + +To quickly try out the :code:`./mach talos-test` command, the following can be run to do a +single run of the DevTools' simple netmonitor test. + +.. code-block:: bash + + # Run the "simple.netmonitor" test very quickly with 1 cycle, and 1 page cycle. + ./mach talos-test --activeTests damp --subtests simple.netmonitor --cycles 1 --tppagecycles 1 + + +The :code:`--print-suites` and :code:`--print-tests` are two helpful command flags to +figure out what suites and tests are available to run. + +.. code-block:: bash + + # Print out the suites: + ./mach talos-test --print-suites + + # Available suites: + # bcv (basic_compositor_video) + # chromez (about_preferences_basic:tresize) + # dromaeojs (dromaeo_css:kraken) + # ... + + # Run all of the tests in the "bcv" test suite: + ./mach talos-test --suite bcv + + # Print out the tests: + ./mach talos-test --print-tests + + # Available tests: + # ================ + # + # a11yr + # ----- + # This test ensures basic a11y tables and permutations do not cause + # performance regressions. + # + # ... + + # Run the tests in "a11yr" listed above + ./mach talos-test --activeTests a11yr + +Running Talos on Try +******************** + +Talos runs can be generated through the mach try fuzzy finder: + +.. code-block:: bash + + ./mach try fuzzy + +The following is an example output at the time of this writing. Refine the query for the +platform and test suites of your choosing. + +.. code-block:: + + | test-windows10-64-qr/opt-talos-bcv-swr-e10s + | test-linux64-shippable/opt-talos-webgl-e10s + | test-linux64-shippable/opt-talos-other-e10s + | test-linux64-shippable-qr/opt-talos-g5-e10s + | test-linux64-shippable-qr/opt-talos-g4-e10s + | test-linux64-shippable-qr/opt-talos-g3-e10s + | test-linux64-shippable-qr/opt-talos-g1-e10s + | test-windows10-64/opt-talos-webgl-gli-e10s + | test-linux64-shippable/opt-talos-tp5o-e10s + | test-linux64-shippable/opt-talos-svgr-e10s + | test-linux64-shippable/opt-talos-damp-e10s + > test-windows7-32/opt-talos-webgl-gli-e10s + | test-linux64-shippable/opt-talos-bcv-e10s + | test-linux64-shippable/opt-talos-g5-e10s + | test-linux64-shippable/opt-talos-g4-e10s + | test-linux64-shippable/opt-talos-g3-e10s + | test-linux64-shippable/opt-talos-g1-e10s + | test-linux64-qr/opt-talos-bcv-swr-e10s + + For more shortcuts, see mach help try fuzzy and man fzf + select: <tab>, accept: <enter>, cancel: <ctrl-c>, select-all: <ctrl-a>, cursor-up: <up>, cursor-down: <down> + 1379/2967 + > talos + +At a glance +*********** + +- Tests are defined in + `testing/talos/talos/test.py <https://searchfox.org/mozilla-central/source/testing/talos/talos/test.py>`__ +- Treeherder abbreviations are defined in + `taskcluster/ci/test/talos.yml <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml>`__ +- Suites are defined for production in + `testing/talos/talos.json <https://searchfox.org/mozilla-central/source/testing/talos/talos.json>`__ + +Test lifecycle +************** + +- Taskcluster schedules `talos + jobs <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml>`__ +- Taskcluster runs a Talos job on a hardware machine when one is + available - this is bootstrapped by + `mozharness <https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py>`__ + + - mozharness downloads the build, talos.zip (found in + `talos.json <https://searchfox.org/mozilla-central/source/testing/talos/talos.json>`__), + and creates a virtualenv for running the test. + - mozharness `configures the test and runs + it <https://wiki.mozilla.org/TestEngineering/Performance/Talos/Running#How_Talos_is_Run_in_Production>`__ + - After the test is completed the data is uploaded to + `Perfherder <https://treeherder.mozilla.org/perf.html#/graphs>`__ + +- Treeherder displays a green (all OK) status and has a link to + `Perfherder <https://treeherder.mozilla.org/perf.html#/graphs>`__ +- 13 pushes later, + `analyze_talos.py <http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py>`__ + is ran which compares your push to the previous 12 pushes and next 12 + pushes to look for a + `regression <https://wiki.mozilla.org/TestEngineering/Performance/Talos/Data#Regressions>`__ + + - If a regression is found, it will be posted on `Perfherder + Alerts <https://treeherder.mozilla.org/perf.html#/alerts>`__ + +Test types +********** + +There are two different species of Talos tests: + +- Startup_: Start up the browser and wait for either the load event or the paint event and exit, measuring the time +- `Page load`_: Load a manifest of pages + +In addition we have some variations on existing tests: + +- Heavy_: Run tests with the heavy user profile instead of a blank one +- `Web extension`_: Run tests with a web extension to see the perf impact extension have +- `Real-world WebExtensions`_: Run tests with a set of 5 popular real-world WebExtensions installed and enabled. + +Some tests measure different things: + +- Paint_: These measure events from the browser like moz_after_paint, etc. +- ASAP_: These tests go really fast and typically measure how many frames we can render in a time window +- Benchmarks_: These are benchmarks that measure specific items and report a summarized score + +Startup +======= + +`Startup +tests <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test>`__ +launch Firefox and measure the time to the onload or paint events. We +run this in a series of cycles (default to 20) to generate a full set of +data. Tests that currently are startup tests are: + +- `ts_paint <#ts_paint>`_ +- tpaint_ +- `tresize <#tresize>`_ +- `sessionrestore <#sessionrestore>`_ +- `sessionrestore_no_auto_restore <#sessionrestore_no_auto_restore>`_ +- `sessionrestore_many_windows <#sessionrestore_many_windows>`_ + +Page load +========= + +Many of the talos tests use the page loader to load a manifest of pages. +These are tests that load a specific page and measure the time it takes +to load the page, scroll the page, draw the page etc. In order to run a +page load test, you need a manifest of pages to run. The manifest is +simply a list of URLs of pages to load, separated by carriage returns, +e.g.: + +.. code-block:: none + + https://www.mozilla.org + https://www.mozilla.com + +Example: +`svgx.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svgx/svgx.manifest>`__ + +Manifests may also specify that a test computes its own data by +prepending a ``%`` in front of the line: + +.. code-block:: none + + % https://www.mozilla.org + % https://www.mozilla.com + +Example: +`v8.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/v8_7/v8.manifest>`__ + +The file you created should be referenced in your test config inside of +`test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l607>`__. +For example, open test.py, and look for the line referring to the test +you want to run: + +.. code-block:: python + + tpmanifest = '${talos}/page_load_test/svgx/svgx.manifest' + tpcycles = 1 # run a single cycle + tppagecycles = 25 # load each page 25 times before moving onto the next page + +Heavy +===== + +All our testing is done with empty blank profiles, this is not ideal for +finding issues for end users. We recently undertook a task to create a +daily update to a profile so it is modern and relevant. It browses a +variety of web pages, and have history and cache to give us a more +realistic scenario. + +The toolchain is documented on +`github <https://github.com/tarekziade/heavy-profile>`__ and was added +to Talos in `bug +1407398 <https://bugzilla.mozilla.org/show_bug.cgi?id=1407398>`__. + +Currently we have issues with this on windows (takes too long to unpack +the files from the profile), so we have turned this off there. Our goal +is to run this on basic pageload and startup tests. + +Web extension +============= + +Web Extensions are what Firefox has switched to and there are different +code paths and APIs used vs addons. Historically we don't test with +addons (other than our test addons) and are missing out on common +slowdowns. In 2017 we started running some startup and basic pageload +tests with a web extension in the profile (`bug +1398974 <https://bugzilla.mozilla.org/show_bug.cgi?id=1398974>`__). We +have updated the Extension to be more real world and will continue to do +that. + +Real-world WebExtensions +======================== + +We've added a variation on our test suite that automatically downloads, +installs and enables 5 popular WebExtensions. This is used to measure +things like the impact of real-world WebExtensions on start-up time. + +Currently, the following extensions are installed: + +- Adblock Plus (3.5.2) +- Cisco Webex Extension (1.4.0) +- Easy Screenshot (3.67) +- NoScript (10.6.3) +- Video DownloadHelper (7.3.6) + +Note that these add-ons and versions are "pinned" by being held in a +compressed file that's hosted in an archive by our test infrastructure +and downloaded at test runtime. To update the add-ons in this set, one +must provide a new ZIP file to someone on the test automation team. See +`this comment in +Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=1575089#c3>`__. + +Paint +===== + +Paint tests are measuring the time to receive both the +`MozAfterPaint <https://developer.mozilla.org/en-US/docs/Web/Events/MozAfterPaint>`__ +and OnLoad event instead of just the OnLoad event. Most tests now look +for this unless they are an ASAP test, or an internal benchmark + +ASAP +==== + +We have a variety of tests which we now run in ASAP mode where we render +as fast as possible (disabling vsync and letting the rendering iterate +as fast as it can using \`requestAnimationFrame`). In fact we have +replaced some original tests with the 'x' versions to make them measure. +We do this with RequestAnimationFrame(). + +ASAP tests are: + +- `basic_compositor_video <#basic_compositor_video>`_ +- `displaylist_mutate <#displaylist_mutate>`_ +- `glterrain <#glterrain>`_ +- `rasterflood_svg <#rasterflood_svg>`_ +- `rasterflood_gradient <#rasterflood_gradient>`_ +- `tsvgx <#tsvgx>`_ +- `tscrollx <#tscrollx>`_ +- `tp5o_scroll <#tp5o_scroll>`_ +- `tabswitch <#tabswitch>`_ +- `tart <#tart>`_ + +Benchmarks +========== + +Many tests have internal benchmarks which we report as accurately as +possible. These are the exceptions to the general rule of calculating +the suite score as a geometric mean of the subtest values (which are +median values of the raw data from the subtests). + +Tests which are imported benchmarks are: + +- `ARES6 <#ares6>`_ +- `dromaeo <#dromaeo>`_ +- `JetStream <#jetstream>`_ +- `kraken <#kraken>`_ +- `motionmark <#motionmark>`_ +- `stylebench <#stylebench>`_ + +Row major vs. column major +========================== + +To get more stable numbers, tests are run multiple times. There are two +ways that we do this: row major and column major. Row major means each +test is run multiple times and then we move to the next test (and run it +multiple times). Column major means that each test is run once one after +the other and then the whole sequence of tests is run again. + +More background information about these approaches can be found in Joel +Maher's `Reducing the Noise in +Talos <https://elvis314.wordpress.com/2012/03/12/reducing-the-noise-in-talos/>`__ +blog post. + +Page sets +********* + +We run our tests 100% offline, but serve pages via a webserver. Knowing +this we need to store and make available the offline pages we use for +testing. + +tp5pages +======== + +Some tests make use of a set of 50 "real world" pages, known as the tp5n +set. These pages are not part of the talos repository, but without them +the tests which use them won't run. + +- To add these pages to your local setup, download the latest tp5n zip + from `tooltool <https://mozilla-releng.net/tooltool/>`__, and extract + it such that ``tp5n`` ends up as ``testing/talos/talos/tests/tp5n``. + You can also obtain it by running a talos test locally to get the zip + into ``testing/talos/talos/tests/``, i.e ``./mach talos-test --suite damp`` +- see also `tp5o <#tp5o>`_. + +Talos Tests +*********** +For the sample commands found below, note that the capitalization used is important. Without the exact spelling, the test won't be found when running locally. + +.. dropdown:: ARES6 + :class-container: anchor-id-ARES6 + + * contact: :jandem and SpiderMonkey Team + * source: `ARES-6 <https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/ARES-6>`__ + * type: `Page load`_ + * data: 6 cycles of the entire benchmark + * `geometric mean <https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#259>`__ self reported from the benchmark + * **Lower is better** + * unit: geometric mean / benchmark score + * lower_is_better: True + * tpmanifest: ${talos}/tests/ares6/ares6.manifest + * tppagecycles: 1 + * Command + + .. code-block:: + + ./mach talos-test -a ARES6 + + * **Test Task**: + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: JetStream + :class-container: anchor-id-JetStream + + * contact: :jandem and SpiderMonkey Team + * source: `jetstream.manifest <https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/jetstream/jetstream.manifest>`__ and jetstream.zip from tooltool + * type: `Page load`_ + * measuring: JavaScript performance + * reporting: geometric mean from the benchmark + * data: internal benchmark + * suite: `geometric + mean <https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#259>`__ + provided by the benchmark + * description: + | This is the `JetStream <http://browserbench.org/JetStream/in-depth.html>`__ + javascript benchmark taken verbatim and slightly modified to fit into + our pageloader extension and talos harness. + * tpmanifest: ${talos}/tests/jetstream/jetstream.manifest + * tppagecycles: 1 + * Command + + .. code-block:: + + ./mach talos-test -a JetStream + + * **Test Task**: + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: a11yr + :class-container: anchor-id-a11yr + + * contact: :jamie and accessibility team + * source: `a11y.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/a11y>`__ + * type: `Page load`_ + * measuring: ??? + * data: we load 2 pages 25 times each, collect 2 sets of 25 data points + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 24; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627>`__ + * suite: `geometric mean`_ of the 2 subtest results. + * reporting: test time in ms (lower is better) + * description: + | This test ensures basic a11y tables and permutations do not cause performance regressions. + * **Example Data** + + .. code-block:: + + 0;dhtml.html;1584;1637;1643;1665;1741;1529;1647;1645;1692;1647;1542;1750;1654;1649;1541;1656;1674;1645;1645;1740;1558;1652;1654;1656;1654 | + 1;tablemutation.html;398;385;389;391;387;387;385;387;388;385;384;31746;386;387;384;387;389;387;387;387;388;391;386;387;388 | + + * a11y: True + * alert_threshold: 5.0 + * preferences: {'dom.send_after_paint_to_content': False} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/a11y/a11y.manifest + * tpmozafterpaint: True + * tppagecycles: 25 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a a11yr + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: about_preferences_basic + :class-container: anchor-id-about_preferences_basic + + * contact: :jaws and :gijs + * source: `about_preferences_basic.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/about-preferences/about_preferences_basic.manifest>`__ + * type: `Page load`_ + * measuring: first-non-blank-paint + * data: We load 5 urls 1 time each, and repeat for 25 cycles; collecting 25 sets of 5 data points + * summarization: + * subtest: `ignore first`_ five data points, then take the `median`_ of the rest; `source: test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627>`__ + * suite: `geometric mean`_ of the the subtest results. + * reporting: test time in ms (lower is better) + * description: + | This test measures the performance of the Firefox about:preferences + page. This test is a little different than other pageload tests in that + we are loading one page (about:preferences) but also testing the loading + of that same page's subcategories/panels (i.e. about:preferences#home). + + When simply changing the page's panel/category, that doesn't cause a new + onload event as expected; therefore we had to introduce loading the + 'about:blank' page in between each page category; that forces the entire + page to reload with the specified category panel activated. + + For that reason, when new panels/categories are added to the + 'about:preferences' page, it can be expected that a performance + regression may be introduced, even if a subtest hasn't been added for + that new page category yet. + + This test should only ever have 1 pagecycle consisting of the main + about-preferences page and each category separated by an about:blank + between. Then repeats are achieved by using 25 cycles (instead of + pagecycles). + * **Example Data** + + .. code-block:: + + 0;preferences;346;141;143;150;136;143;153;140;154;156;143;154;146;147;151;166;140;146;140;144;144;156;154;150;140 + 2;preferences#search;164;142;133;141;141;141;142;140;131;146;131;140;131;131;139;142;140;144;146;143;143;142;142;137;143 + 3;preferences#privacy;179;159;166;177;173;153;148;154;168;155;164;155;152;157;149;155;156;186;149;156;160;151;158;168;157 + 4;preferences#sync;148;156;140;137;159;139;143;145;138;130;145;142;141;133;146;141;147;143;146;146;139;144;142;151;156 + 5;preferences#home;141;111;130;131;138;128;133;122;138;138;131;139;139;132;133;141;143;139;138;135;136;128;134;140;135 + + * fnbpaint: True + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 1 + * lower_is_better: True + * pine: False + * tpcycles: 25 + * tpmanifest: ${talos}/tests/about-preferences/about_preferences_basic.manifest + * tppagecycles: 1 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a about_preferences_basic + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-chrome-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-chrome-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: basic_compositor_video + :class-container: anchor-id-basic_compositor_video + + * contact: :b0bh00d, :jeffm, and gfx + * source: `video <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/video>`__ + * type: `Page load`_ + * data: 12 cycles of the entire benchmark, each subtest will have 12 data points (see below) + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 11; `source: test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522>`__ + * suite: `geometric mean`_ of the 24 subtest results. + * **Lower is better** + * **Example Data** + + .. code-block:: + + ;0;240p.120fps.mp4_scale_fullscreen_startup;11.112;11.071;11.196;11.157;11.195;11.240;11.196;11.155;11.237;11.074;11.154;11.282 + ;1;240p.120fps.mp4_scale_fullscreen_inclip;10.995;11.114;11.052;10.991;10.876;11.115;10.995;10.991;10.997;10.994;10.992;10.993 + ;2;240p.120fps.mp4_scale_1_startup;1.686;1.690;1.694;1.683;1.689;1.692;1.686;1.692;1.689;1.704;1.684;1.686 + ;3;240p.120fps.mp4_scale_1_inclip;1.666;1.666;1.666;1.668;1.667;1.669;1.667;1.668;1.668;1.667;1.667;1.669 + ;4;240p.120fps.mp4_scale_1.1_startup;1.677;1.672;1.673;1.677;1.673;1.677;1.672;1.677;1.677;1.671;1.676;1.679 + ;5;240p.120fps.mp4_scale_1.1_inclip;1.667;1.668;1.666;1.667;1.667;1.668;1.667;1.667;1.667;1.667;1.668;1.668 + ;6;240p.120fps.mp4_scale_2_startup;1.927;1.908;1.947;1.946;1.902;1.932;1.916;1.936;1.921;1.896;1.908;1.894 + ;7;240p.120fps.mp4_scale_2_inclip;1.911;1.901;1.896;1.917;1.897;1.921;1.907;1.944;1.904;1.912;1.936;1.913 + ;8;480p.60fps.webm_scale_fullscreen_startup;11.675;11.587;11.539;11.454;11.723;11.410;11.629;11.410;11.454;11.498;11.540;11.540 + ;9;480p.60fps.webm_scale_fullscreen_inclip;11.304;11.238;11.370;11.300;11.364;11.368;11.237;11.238;11.434;11.238;11.304;11.368 + ;10;480p.60fps.webm_scale_1_startup;3.386;3.360;3.391;3.376;3.387;3.402;3.371;3.371;3.356;3.383;3.376;3.356 + ;11;480p.60fps.webm_scale_1_inclip;3.334;3.334;3.329;3.334;3.334;3.334;3.334;3.334;3.334;3.335;3.334;3.334 + ;12;480p.60fps.webm_scale_1.1_startup;3.363;3.363;3.368;3.356;3.356;3.379;3.364;3.360;3.360;3.356;3.363;3.356 + ;13;480p.60fps.webm_scale_1.1_inclip;3.329;3.334;3.329;3.334;3.333;3.334;3.334;3.334;3.340;3.335;3.329;3.335 + ;14;480p.60fps.webm_scale_2_startup;4.960;4.880;4.847;4.959;4.802;4.863;4.824;4.926;4.847;4.785;4.870;4.855 + ;15;480p.60fps.webm_scale_2_inclip;4.903;4.786;4.892;4.903;4.822;4.832;4.798;4.857;4.808;4.856;4.926;4.741 + ;16;1080p.60fps.mp4_scale_fullscreen_startup;14.638;14.495;14.496;14.710;14.781;14.853;14.639;14.637;14.707;14.637;14.711;14.636 + ;17;1080p.60fps.mp4_scale_fullscreen_inclip;13.795;13.798;13.893;13.702;13.799;13.607;13.798;13.705;13.896;13.896;13.896;14.088 + ;18;1080p.60fps.mp4_scale_1_startup;6.995;6.851;6.930;6.820;6.915;6.805;6.898;6.866;6.852;6.850;6.803;6.851 + ;19;1080p.60fps.mp4_scale_1_inclip;6.560;6.625;6.713;6.601;6.645;6.496;6.624;6.538;6.539;6.497;6.580;6.558 + ;20;1080p.60fps.mp4_scale_1.1_startup;7.354;7.230;7.195;7.300;7.266;7.283;7.196;7.249;7.230;7.230;7.212;7.264 + ;21;1080p.60fps.mp4_scale_1.1_inclip;6.969;7.222;7.018;6.993;7.045;6.970;6.970;6.807;7.118;6.969;6.997;6.972 + ;22;1080p.60fps.mp4_scale_2_startup;6.963;6.947;6.914;6.929;6.979;7.010;7.010245327102808;6.914;6.961;7.028;7.012;6.914 + ;23;1080p.60fps.mp4_scale_2_inclip;6.757;6.694;6.672;6.669;6.737;6.831;6.716;6.715;6.832;6.670;6.672;6.759 + + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 1 + * lower_is_better: True + * preferences: {'full-screen-api.allow-trusted-requests-only': False, 'layers.acceleration.force-enabled': False, 'layers.acceleration.disabled': True, 'gfx.webrender.software': True, 'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'full-screen-api.warning.timeout': 500, 'media.ruin-av-sync.enabled': True} + * timeout: 10000 + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/video/video.manifest + * tppagecycles: 12 + * unit: ms/frame + * Command + + .. code-block:: + + ./mach talos-test -a basic_compositor_video + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-bcv-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-bcv-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-bcv** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: cpstartup + :class-container: anchor-id-cpstartup + + * contact: :mconley, Firefox Desktop Front-end team, Gijs, fqueze, and dthayer + * measuring: Time from opening a new tab (which creates a new content process) to having that new content process be ready to load URLs. + * source: `cpstartup <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/cpstartup>`__ + * type: `Page load`_ + * bug: `bug 1336389 <https://bugzilla.mozilla.org/show_bug.cgi?id=1336389>`__ + * data: 20 cycles of the entire benchmark + * **Lower is better** + * **Example Data** + + .. code-block:: + + 0;content-process-startup;877;737;687;688;802;697;794;685;694;688;794;669;699;684;690;849;687;873;694;689 + + * extensions: ['${talos}/pageloader', '${talos}/tests/cpstartup/extension'] + * gecko_profile_entries: 1000000 + * preferences: {'browser.link.open_newwindow': 3, 'browser.link.open_newwindow.restriction': 2} + * timeout: 600 + * tploadnocache: True + * tpmanifest: ${talos}/tests/cpstartup/cpstartup.manifest + * tppagecycles: 20 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a cpstartup + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: cross_origin_pageload + :class-container: anchor-id-cross_origin_pageload + + * contact: :sefeng, :jesup, and perf eng team + * measuring: The time it takes to load a page which has 20 cross origin iframes + * source: `cross_origin_pageload <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/cross_origin_pageload>`__ + * type: `Page load`_ + * bug: `bug 1701989 <https://bugzilla.mozilla.org/show_bug.cgi?id=1701989>`__ + * data: 10 cycles of the entire benchmark + * **Lower is better** + * **Example Data** + + .. code-block:: + + 0;/index.html;194.42;154.12;141.38;145.88;136.92;147.64;152.54;138.02;145.5;143.62 + + * extensions: ['${talos}/pageloader'] + * preferences: {'dom.ipc.processPrelaunch.fission.number': 30} + * timeout: 100 + * tploadnocache: True + * tpmanifest: ${talos}/tests/cross_origin_pageload/cross_origin_pageload.manifest + * tppagecycles: 10 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a cross_origin_pageload + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: damp + :class-container: anchor-id-damp + + * contact: :ochameau and devtools team + * source: `damp <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/devtools>`__ + * type: `Page load`_ + * measuring: Developer Tools toolbox performance. Split in test suites covering different DevTools areas (inspector, webconsole, other). + * reporting: intervals in ms (lower is better) + * see below for details + * data: there are 36 reported subtests from DAMP which we load 25 times, resulting in 36 sets of 25 data points. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 24 data points; `source: test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l356>`__ + * suite: No value for the suite, only individual subtests are relevant. + * description: + | To run this locally, you'll need to pull down the `tp5 page + set <#page-sets>`__ and run it in a local web server. See the `tp5 + section <#tp5>`__. + * **Example Data** + + .. code-block:: + + 0;simple.webconsole.open.DAMP;1198.86;354.38;314.44;337.32;344.73;339.05;345.55;358.37;314.89;353.73;324.02;339.45;304.63;335.50;316.69;341.05;353.45;353.73;342.28;344.63;357.62;375.18;326.08;363.10;357.30 + 1;simple.webconsole.reload.DAMP;44.60;41.21;25.62;29.85;38.10;42.29;38.25;40.14;26.95;39.24;40.32;34.67;34.64;44.88;32.51;42.09;28.04;43.05;40.62;36.56;42.44;44.11;38.69;29.10;42.00 + 2;simple.webconsole.close.DAMP;27.26;26.97;25.45;27.82;25.98;26.05;38.00;26.89;24.90;26.61;24.90;27.22;26.95;25.18;24.24;25.60;28.91;26.90;25.57;26.04;26.79;27.33;25.76;26.47;27.43 + 3;simple.inspector.open.DAMP;507.80;442.03;424.93;444.62;412.94;451.18;441.39;435.83;441.27;460.69;440.93;413.13;418.73;443.41;413.93;447.34;434.69;459.24;453.60;412.58;445.41;466.34;441.89;417.59;428.82 + 4;simple.inspector.reload.DAMP;169.45;165.11;163.93;181.12;167.86;164.67;170.34;173.12;165.24;180.59;176.72;187.42;170.14;190.35;176.59;155.00;151.66;174.40;169.46;163.85;190.93;217.00;186.25;181.31;161.13 + 5;simple.inspector.close.DAMP;44.40;42.28;42.71;47.21;41.74;41.24;42.94;43.73;48.24;43.04;48.61;42.49;45.93;41.36;43.83;42.43;41.81;43.93;41.38;40.98;49.76;50.86;43.49;48.99;44.02 + 6;simple.jsdebugger.open.DAMP;642.59;464.02;540.62;445.46;471.09;466.57;466.70;511.91;424.12;480.70;448.37;477.51;488.99;437.97;442.32;459.03;421.54;467.99;472.78;440.27;431.47;454.76;436.86;453.61;485.59 + 7;simple.jsdebugger.reload.DAMP;51.65;55.46;225.46;53.32;58.78;53.23;54.39;51.59;55.46;48.03;50.70;46.34;230.94;53.71;54.23;53.01;61.03;51.23;51.45;293.01;56.93;51.44;59.85;63.35;57.44 + 8;simple.jsdebugger.close.DAMP;29.12;30.76;40.34;32.09;31.26;32.30;33.95;31.89;29.68;31.39;32.09;30.36;44.63;32.33;30.16;32.43;30.89;30.85;31.99;49.86;30.94;44.63;32.54;29.79;33.15 + 9;simple.styleeditor.open.DAMP;758.54;896.93;821.17;1026.24;887.14;867.39;927.86;962.80;740.40;919.39;741.01;925.21;807.39;1051.47;729.04;1095.78;755.03;888.70;900.52;810.30;1090.09;869.72;737.44;893.16;927.72 + 10;simple.styleeditor.reload.DAMP;57.32;178.13;59.23;60.82;71.45;78.86;74.35;60.11;66.43;77.41;61.96;69.22;65.97;45.53;67.88;74.76;124.61;60.01;36.66;59.24;65.01;165.68;34.61;69.02;71.42 + 11;simple.styleeditor.close.DAMP;28.28;56.50;36.18;30.00;36.32;34.85;35.33;36.24;25.45;36.72;26.53;36.90;28.88;30.94;26.56;31.34;47.79;30.90;30.52;27.95;30.75;56.28;26.76;30.25;37.42 + 12;simple.performance.open.DAMP;444.28;357.87;331.17;335.16;585.71;402.99;504.58;466.95;272.98;427.54;345.60;441.53;319.99;327.91;312.94;349.79;399.51;465.60;418.42;295.14;362.06;363.11;445.71;634.96;500.83 + 13;simple.performance.reload.DAMP;38.07;33.44;35.99;70.57;64.04;106.47;148.31;29.60;68.47;28.95;148.46;75.92;32.15;93.72;36.17;44.13;75.11;154.76;98.28;75.16;29.39;36.68;113.16;64.05;135.60 + 14;simple.performance.close.DAMP;23.98;25.49;24.19;24.61;27.56;40.33;33.85;25.13;22.62;25.28;41.84;25.09;26.39;25.20;23.76;25.44;25.92;30.40;40.77;25.41;24.57;26.15;43.65;28.54;30.16 + 15;simple.netmonitor.open.DAMP;438.85;350.64;318.04;329.12;341.91;352.33;344.05;334.15;514.57;327.95;471.50;334.55;344.94;364.39;727.56;374.48;339.45;344.31;345.61;329.78;325.74;334.74;350.36;342.85;344.64 + 16;simple.netmonitor.reload.DAMP;59.68;47.50;69.37;61.18;76.89;83.22;68.11;81.24;56.15;68.20;32.41;81.22;81.62;44.30;39.52;29.60;86.07;71.18;76.32;79.93;79.63;82.15;83.58;87.04;82.97 + 17;simple.netmonitor.close.DAMP;38.42;39.32;52.56;43.37;48.08;40.62;51.12;41.11;59.54;43.29;41.72;40.85;51.61;49.61;51.39;44.91;40.36;41.10;45.43;42.15;42.63;40.69;41.21;44.04;41.95 + 18;complicated.webconsole.open.DAMP;589.97;505.93;480.71;530.93;460.60;479.63;485.33;489.08;605.28;457.12;463.95;493.28;680.05;478.72;504.47;578.69;488.66;485.34;504.94;460.67;548.38;474.98;470.33;471.34;464.58 + 19;complicated.webconsole.reload.DAMP;2707.20;2700.17;2596.02;2728.09;2905.51;2716.65;2657.68;2707.74;2567.86;2726.36;2650.92;2839.14;2620.34;2718.36;2595.22;2686.28;2703.48;2609.75;2686.41;2577.93;2634.47;2745.56;2655.89;2540.09;2649.18 + 20;complicated.webconsole.close.DAMP;623.56;570.80;636.63;502.49;565.83;537.93;525.46;565.78;532.90;562.66;525.42;490.88;611.99;486.45;528.60;505.35;480.55;500.75;532.75;480.91;488.69;548.77;535.31;477.92;519.84 + 21;complicated.inspector.open.DAMP;1233.26;753.57;742.74;953.11;653.29;692.66;653.75;767.02;840.68;707.56;713.95;685.79;690.21;1020.47;685.67;721.69;1063.72;695.55;702.15;760.91;853.14;660.12;729.16;1044.86;724.34 + 22;complicated.inspector.reload.DAMP;2384.90;2436.35;2356.11;2436.58;2372.96;2558.86;2543.76;2351.03;2411.95;2358.04;2413.27;2339.85;2373.11;2338.94;2418.88;2360.87;2349.09;2498.96;2577.73;2445.07;2354.88;2424.90;2696.10;2362.39;2493.29 + 23;complicated.inspector.close.DAMP;541.96;509.38;476.91;456.48;545.48;634.04;603.10;488.09;599.20;480.45;617.93;420.39;514.92;439.99;727.41;469.04;458.59;539.74;611.55;725.03;473.36;484.60;481.46;458.93;554.76 + 24;complicated.jsdebugger.open.DAMP;644.97;578.41;542.23;595.94;704.80;603.08;689.18;552.99;597.23;584.17;682.14;758.16;791.71;738.43;640.30;809.26;704.85;601.32;696.10;683.44;796.34;657.25;631.89;739.96;641.82 + 25;complicated.jsdebugger.reload.DAMP;2676.82;2650.84;2687.78;2401.23;3421.32;2450.91;2464.13;2286.40;2399.40;2415.97;2481.48;2827.69;2652.03;2554.63;2631.36;2443.83;2564.73;2466.22;2597.57;2552.73;2539.42;2481.21;2319.50;2539.00;2576.43 + 26;complicated.jsdebugger.close.DAMP;795.68;616.48;598.88;536.77;435.02;635.61;558.67;841.64;613.48;886.60;581.38;580.96;571.40;605.34;671.00;882.02;619.01;579.63;643.05;656.78;699.64;928.99;549.76;560.96;676.32 + 27;complicated.styleeditor.open.DAMP;2327.30;2494.19;2190.29;2205.60;2198.11;2509.01;2189.79;2532.05;2178.03;2207.75;2224.96;2665.84;2294.40;2645.44;2661.41;2364.60;2395.36;2582.72;2872.03;2679.29;2561.24;2330.11;2580.16;2510.36;2860.83 + 28;complicated.styleeditor.reload.DAMP;2218.46;2335.18;2284.20;2345.05;2286.98;2453.47;2506.97;2661.19;2529.51;2289.78;2564.15;2608.24;2270.77;2362.17;2287.31;2300.19;2331.56;2300.86;2239.27;2231.33;2476.14;2286.28;2583.24;2540.29;2259.67 + 29;complicated.styleeditor.close.DAMP;302.67;343.10;313.15;305.60;317.92;328.44;350.70;370.12;339.77;308.72;312.71;320.63;305.52;316.69;324.92;306.60;313.65;312.17;326.26;321.45;334.56;307.38;312.95;350.94;339.36 + 30;complicated.performance.open.DAMP;477.99;537.96;564.85;515.05;502.03;515.58;492.80;689.06;448.76;588.91;509.76;485.39;548.17;479.14;638.67;535.86;541.61;611.52;554.72;665.37;694.04;470.60;746.16;547.85;700.02 + 31;complicated.performance.reload.DAMP;2258.31;2345.74;2509.24;2579.71;2367.94;2365.94;2260.86;2324.23;2579.01;2412.63;2540.38;2069.80;2534.91;2443.48;2193.01;2442.99;2422.42;2475.35;2076.48;2092.95;2444.53;2353.86;2154.28;2354.61;2104.82 + 32;complicated.performance.close.DAMP;334.44;516.66;432.49;341.29;309.30;365.20;332.16;311.42;370.81;301.81;381.13;299.39;317.60;314.10;372.44;314.76;306.24;349.85;382.08;352.53;309.40;298.44;314.10;315.44;405.22 + 33;complicated.netmonitor.open.DAMP;469.70;597.87;468.36;823.09;696.39;477.19;487.78;495.92;587.89;471.48;555.02;507.45;883.33;522.15;756.86;713.64;593.82;715.13;477.15;717.85;586.79;556.97;631.43;629.55;581.16 + 34;complicated.netmonitor.reload.DAMP;4033.55;3577.36;3655.61;3721.24;3874.29;3977.92;3778.62;3825.60;3984.65;3707.91;3985.24;3565.21;3702.40;3956.70;3627.14;3916.11;3929.11;3934.06;3590.60;3628.39;3618.84;3579.52;3953.04;3781.01;3682.69 + 35;complicated.netmonitor.close.DAMP;1042.98;920.21;928.19;940.38;950.25;1043.61;1078.16;1077.38;1132.91;1095.05;1176.31;1256.83;1143.14;1234.61;1248.97;1242.29;1378.63;1312.74;1371.48;1373.15;1544.55;1422.51;1549.48;1616.55;1506.58 + + * cycles: 5 + * extensions: ['${talos}/pageloader', '${talos}/tests/devtools/addon'] + * gecko_profile_entries: 10000000 + * gecko_profile_interval: 10 + * linux_counters: None + * mac_counters: None + * perfherder_framework: devtools + * preferences: {'devtools.memory.enabled': True} + * subtest_alerts: True + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/devtools/damp.manifest + * tpmozafterpaint: False + * tppagecycles: 5 + * unit: ms + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a damp + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-other** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-webconsole** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-damp-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-damp-webconsole** + - ✅ + - ✅ + - ❌ + - ✅ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-damp-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-damp-webconsole** + - ✅ + - ✅ + - ❌ + - ✅ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-other** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-webconsole** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-other** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-webconsole** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-other** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-damp-webconsole** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-damp-inspector** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-damp-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-damp-webconsole** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: displaylist_mutate + :class-container: anchor-id-displaylist_mutate + + * contact: :miko and gfx + * source: `displaylist_mutate.html <https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/layout/benchmarks/displaylist_mutate.html>`__ + * type: `Page load`_ + * data: we load the displaylist_mutate.html page five times, measuring pageload each time, generating 5 data points. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 4; `source: test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986>`__ + * description: + | This measures the amount of time it takes to render a page after + changing its display list. The page has a large number of display list + items (10,000), and mutates one every frame. The goal of the test is to + make displaylist construction a bottleneck, rather than painting or + other factors, and thus improvements or regressions to displaylist + construction will be visible. The test runs in ASAP mode to maximize + framerate, and the result is how quickly the test was able to mutate and + re-paint 600 items, one during each frame. + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 2 + * linux_counters: None + * mac_counters: None + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/layout/displaylist_mutate.manifest + * tpmozafterpaint: False + * tppagecycles: 5 + * unit: ms + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a displaylist_mutate + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g4-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g4-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: dromaeo + :class-container: anchor-id-dromaeo + + * description: + | Dromaeo suite of tests for JavaScript performance testing. See the + `Dromaeo wiki <https://wiki.mozilla.org/Dromaeo>`__ for more + information. + + This suite is divided into several sub-suites. + + Each sub-suite is divided into tests, and each test is divided into + sub-tests. Each sub-test takes some (in theory) fixed piece of work and + measures how many times that piece of work can be performed in one + second. The score for a test is then the geometric mean of the + runs/second numbers for its sub-tests. The score for a sub-suite is the + geometric mean of the scores for its tests. + * Command + + .. code-block:: + + ./mach talos-test -a dromaeo + + +.. dropdown:: dromaeo_css + :class-container: anchor-id-dromaeo_css + + * contact: :emilio, and css/layout team + * source: `css.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/dromaeo>`__ + * type: `Page load`_ + * reporting: speed in test runs per second (higher is better) + * data: Dromaeo has 6 subtests which run internal benchmarks, each benchmark reports about 180 raw data points each + * summarization: + * subtest: + Dromaeo is a custom benchmark which has a lot of micro tests + inside each subtest, because of this we use a custom `dromaeo + filter <https://wiki.mozilla.org/TestEngineering/Performance/Talos/Data#dromaeo>`__ + to summarize the subtest. Each micro test produces 5 data points and + for each 5 data points we take the mean, leaving 36 data points to + represent the subtest (assuming 180 points). These 36 micro test + means, are then run through a geometric_mean to produce a single + number for the dromaeo subtest. `source: + filter.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l527>`__ + * suite: `geometric mean`_ of the 6 subtest results. + * description: + | Each page in the manifest is part of the dromaeo css benchmark. Each + page measures the performance of searching the DOM for nodes matching + various CSS selectors, using different libraries for the selector + implementation (jQuery, Dojo, Mootools, ExtJS, Prototype, and Yahoo UI). + * **Example Data** + + .. code-block:: + + 0;dojo.html;2209.83;2269.68;2275.47;2278.83;2279.81;4224.43;4344.96;4346.74;4428.69;4459.82;4392.80;4396.38;4412.54;4414.34;4415.62;3909.94;4027.96;4069.08;4099.63;4099.94;4017.70;4018.96;4054.25;4068.74;4081.31;3825.10;3984.20;4053.23;4074.59;4106.63;3893.88;3971.80;4031.15;4046.68;4048.31;3978.24;4010.16;4046.66;4051.68;4056.37;4189.50;4287.98;4390.98;4449.89;4450.20;4536.23;4557.82;4588.40;4662.58;4664.42;4675.51;4693.13;4743.72;4758.12;4764.67;4138.00;4251.60;4346.22;4410.12;4417.23;4677.53;4702.48;4714.62;4802.59;4805.33;4445.07;4539.91;4598.93;4605.45;4618.79;4434.40;4543.09;4618.56;4683.98;4689.51;4485.26;4496.75;4511.23;4600.86;4602.08;4567.52;4608.33;4615.56;4619.31;4622.79;3469.44;3544.11;3605.80;3647.74;3658.56;3101.88;3126.41;3147.73;3159.92;3170.73;3672.28;3686.40;3730.74;3748.89;3753.59;4411.71;4521.50;4633.98;4702.72;4708.76;3626.62;3646.71;3713.07;3713.13;3718.91;3846.17;3846.25;3913.61;3914.63;3916.22;3982.88;4112.98;4132.26;4194.92;4201.54;4472.64;4575.22;4644.74;4645.42;4665.51;4120.13;4142.88;4171.29;4208.43;4211.03;4405.36;4517.89;4537.50;4637.77;4644.28;4548.25;4581.20;4614.54;4658.42;4671.09;4452.78;4460.09;4494.06;4521.30;4522.37;4252.81;4350.72;4364.93;4441.40;4492.78;4251.34;4346.70;4355.00;4358.89;4365.72;4494.64;4511.03;4582.11;4591.79;4592.36;4207.54;4308.94;4309.14;4406.71;4474.46 + 1;ext.html;479.65;486.21;489.61;492.94;495.81;24454.14;33580.33;34089.15;34182.83;34186.15;34690.83;35050.30;35051.30;35071.65;35099.82;5758.22;5872.32;6389.62;6525.38;6555.57;8303.96;8532.96;8540.91;8544.00;8571.49;8360.79;8408.79;8432.96;8447.28;8447.83;5817.71;5932.67;8371.83;8389.20;8643.44;7983.80;8073.27;8073.84;8076.48;8078.15;24596.00;32518.84;32787.34;32830.51;32861.00;2220.87;2853.84;3333.53;3345.17;3445.47;24785.75;24971.75;25044.25;25707.61;25799.00;2464.69;2481.89;2527.57;2534.65;2534.92;217793.00;219347.90;219495.00;220059.00;297168.00;40556.19;53062.47;54275.73;54276.00;54440.37;50636.75;50833.49;50983.49;51028.49;51032.74;10746.36;10972.45;11450.37;11692.18;11797.76;8402.58;8415.79;8418.66;8426.75;8428.16;16768.75;16896.00;16925.24;16945.58;17018.15;7047.68;7263.13;7313.16;7337.38;7383.22;713.88;723.72;751.47;861.35;931.00;25454.36;25644.90;25801.87;25992.61;25995.00;819.89;851.23;852.00;886.59;909.89;14325.79;15064.92;15240.39;15431.23;15510.61;452382.00;458194.00;458707.00;459226.00;459601.00;45699.54;46244.54;46270.54;46271.54;46319.00;1073.94;1080.66;1083.35;1085.84;1087.74;26622.33;27807.58;27856.72;28040.58;28217.86;37229.81;37683.81;37710.81;37746.62;37749.81;220386.00;222903.00;240808.00;247394.00;247578.00;25567.00;25568.49;25610.74;25650.74;25710.23;26466.21;28718.71;36175.64;36529.27;36556.00;26676.00;30757.69;31965.84;34521.83;34622.65;32791.18;32884.00;33194.83;33720.16;34192.66;32150.36;32520.02;32851.18;32947.18;33128.01;29472.85;30214.09;30708.54;30999.23;32879.51;23822.88;23978.28;24358.88;24470.88;24515.51 + 2;jquery.html;285.42;288.57;292.66;293.75;294.14;10313.00;10688.20;13659.11;13968.65;14003.93;13488.39;13967.51;13980.79;14545.13;15059.77;4361.37;4488.35;4489.44;4492.24;4496.69;3314.32;3445.07;4412.51;5020.75;5216.66;5113.49;5136.56;5141.31;5143.87;5156.28;5055.95;5135.02;5138.64;5215.82;5226.48;4550.98;4551.59;4553.07;4557.77;4559.16;18339.63;18731.53;18738.63;18741.16;18806.15;1474.99;1538.31;1557.52;1703.67;1772.16;12209.94;12335.44;12358.32;12516.50;12651.94;1520.94;1522.62;1541.37;1584.71;1642.50;57533.00;59169.41;59436.11;59758.70;59872.40;8669.13;8789.34;8994.37;9016.05;9064.95;11047.39;11058.39;11063.78;11077.89;11082.78;6401.81;6426.26;6504.35;6518.25;6529.61;6250.22;6280.65;6304.59;6318.91;6328.72;5144.28;5228.40;5236.21;5271.26;5273.79;1398.91;1450.05;1456.39;1494.66;1519.42;727.85;766.62;844.35;858.49;904.87;9912.55;10249.54;14936.71;16566.50;16685.00;378.04;381.34;381.44;385.67;387.23;5362.60;5392.78;5397.14;5497.12;5514.83;213309.00;318297.00;320682.00;322681.00;322707.00;56357.44;67892.66;68329.66;68463.32;69506.00;418.91;424.49;425.19;425.28;426.40;9363.39;9559.95;9644.00;9737.07;9752.80;33170.83;33677.33;34950.83;35607.47;35765.82;44079.34;44588.55;45396.00;46309.00;46427.30;6302.87;6586.51;6607.08;6637.44;6642.17;9776.17;9790.46;9931.90;10391.79;10392.43;8739.26;8838.38;8870.20;8911.50;8955.15;8422.83;8786.21;8914.00;9135.82;9145.36;8945.28;9028.37;9035.23;9116.64;9137.86;6433.90;6688.73;6822.11;6830.08;6833.90;8575.23;8599.87;8610.91;8655.65;9123.91 + 3;mootools.html;1161.69;1333.31;1425.89;1500.37;1557.37;6706.93;7648.46;8020.04;8031.36;8049.64;7861.80;7972.40;7978.12;7993.32;7993.88;1838.83;1862.93;1864.11;1866.28;1866.71;1909.93;1921.83;1928.53;1954.07;1969.98;1808.68;1820.01;1821.30;1825.92;1826.91;1849.07;1904.99;1908.26;1911.24;1912.50;1856.86;1871.78;1873.72;1878.54;1929.57;6506.67;6752.73;7799.22;7830.41;7855.18;4117.18;4262.42;4267.30;4268.27;4269.62;2720.56;2795.36;2840.08;2840.79;2842.62;699.12;703.75;774.36;791.73;798.18;11096.22;11126.39;11132.72;11147.16;11157.44;3934.33;4067.37;4140.94;4149.75;4150.38;9042.82;9077.46;9083.55;9084.41;9086.41;4431.47;4432.84;4437.33;4438.40;4440.44;3935.67;3937.31;3937.43;3940.53;3976.68;3247.17;3307.90;3319.90;3323.32;3330.60;1001.90;1016.87;1021.12;1021.67;1022.05;1016.34;1019.09;1036.62;1056.81;1057.76;7345.56;7348.56;7391.89;7393.85;7406.30;374.27;392.53;394.73;397.28;398.26;5588.58;5653.21;5655.07;5659.15;5660.66;9775.41;9860.51;9938.40;9959.85;9968.45;9733.42;9904.31;9953.05;9960.55;9967.20;6399.26;6580.11;7245.93;7336.96;7386.78;7162.00;7245.49;7249.38;7250.75;7304.63;8458.24;8583.40;8651.57;8717.39;8742.39;8896.42;8904.60;8927.96;8960.73;8961.82;7483.48;7747.77;7763.46;7766.34;7773.07;7784.00;7821.41;7827.18;7849.18;7855.49;7012.16;7141.57;7250.09;7253.13;7335.89;6977.97;7015.51;7042.40;7204.35;7237.20;7160.46;7293.23;7321.27;7321.82;7331.16;6268.69;6324.11;6325.78;6328.56;6342.40;6554.54;6625.30;6646.00;6650.30;6674.90 + 4;prototype.html;237.05;251.94;256.61;259.65;263.52;4488.53;4676.88;4745.24;4745.50;4748.81;4648.47;4660.21;4666.58;4671.88;4677.32;3602.84;3611.40;3613.69;3615.69;3619.15;3604.41;3619.44;3623.24;3627.66;3628.11;3526.59;3589.35;3615.93;3616.35;3622.80;3624.69;3626.84;3628.47;3631.22;3632.15;3184.76;3186.11;3187.16;3187.78;3189.35;4353.43;4466.46;4482.57;4616.72;4617.88;4012.18;4034.84;4047.07;4047.82;4055.29;4815.11;4815.21;4816.11;4817.08;4820.40;3300.31;3345.18;3369.55;3420.98;3447.97;5026.99;5033.82;5034.50;5034.95;5038.97;3516.72;3520.79;3520.95;3521.81;3523.47;3565.29;3574.23;3574.37;3575.82;3578.37;4045.19;4053.51;4056.76;4058.76;4059.00;4714.67;4868.66;4869.66;4873.54;4878.29;1278.20;1300.92;1301.13;1301.17;1302.47;868.94;871.16;878.50;883.40;884.85;3874.71;3878.44;3881.61;3882.67;3886.92;4959.83;4968.45;4969.50;4971.38;4972.30;3862.69;3870.15;3871.79;3873.83;3878.07;2690.15;2711.66;2714.42;2715.39;2715.89;4349.04;4349.63;4351.33;4353.59;4355.46;4950.95;5101.08;5107.69;5120.21;5120.39;4336.63;4360.76;4361.96;4362.28;4365.43;4928.75;4939.41;4939.56;4943.95;4966.78;4869.03;4886.24;4888.85;4889.14;4895.76;4362.39;4362.78;4363.96;4365.00;4365.08;3408.00;3470.03;3476.37;3546.65;3547.34;4905.73;4926.21;4926.70;4926.93;4929.43;4682.88;4694.91;4696.30;4697.06;4699.69;4688.86;4691.25;4691.46;4698.37;4699.41;4628.07;4631.31;4633.42;4634.00;4636.00;4699.44;4796.02;4808.83;4809.95;4813.52;4719.10;4720.41;4722.95;4723.03;4723.53 + 5;yui.html;569.72;602.22;627.02;647.49;692.84;9978.30;10117.54;10121.70;10129.75;10137.24;9278.68;9291.44;9349.00;9370.53;9375.86;475.79;481.92;606.51;607.42;618.73;617.68;618.89;623.30;626.58;631.85;501.81;649.76;653.22;655.69;656.71;510.62;645.56;657.42;657.88;658.39;475.53;476.77;476.80;476.92;476.96;9895.16;9976.15;9988.25;9989.85;9996.40;9483.15;9545.75;9676.37;9808.51;10360.22;8331.29;8397.87;8538.06;8714.69;8803.78;2748.93;2800.93;2802.59;2857.33;2864.46;33757.16;33804.83;33859.32;33931.00;33991.32;7818.65;7846.92;7892.09;8170.55;8217.75;13691.38;13692.86;13693.25;13698.73;13706.66;5378.70;5517.83;5615.86;5616.16;5624.00;2985.63;3002.97;3003.07;3037.73;3038.87;2459.10;2502.52;2504.91;2507.07;2507.26;396.62;405.78;411.43;412.03;412.56;543.45;550.75;568.50;578.59;592.25;6762.21;6901.72;6984.27;7064.22;7122.29;454.78;519.40;539.29;543.96;566.16;3235.39;3266.13;3453.26;3498.79;3518.54;39079.22;39722.80;41350.59;41422.38;41540.17;34435.14;34606.31;34623.31;34661.00;34672.48;29449.12;29530.11;30507.24;31938.52;31961.52;7449.33;7524.62;7629.73;7712.96;7796.42;22917.43;23319.00;23441.41;23582.88;23583.53;29780.40;30272.55;31761.00;31765.84;31839.36;6112.45;6218.35;6476.68;6603.54;6793.66;10385.79;10471.69;10518.53;10552.74;10644.95;9563.52;9571.33;9617.09;9946.35;9976.80;9406.11;9518.48;9806.46;10102.44;10173.19;9482.43;9550.28;9878.21;9902.90;9951.45;8343.17;8511.00;8606.00;8750.21;8869.29;8234.96;8462.70;8473.49;8499.58;8808.91 + + * gecko_profile_entries: 10000000 + * gecko_profile_interval: 2 + * tpmanifest: ${talos}/tests/dromaeo/css.manifest + * unit: score + * Command + + .. code-block:: + + ./mach talos-test -a dromaeo_css + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-dromaeojs-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-dromaeojs-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: dromaeo_dom + :class-container: anchor-id-dromaeo_dom + + * contact: :peterv and dom team + * source: `dom.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/dromaeo>`__ + * type: `Page load`_ + * data: see Dromaeo DOM + * reporting: speed in test runs per second (higher is better) + * description: + | Each page in the manifest is part of the dromaeo dom benchmark. These + are the specific areas that Dromaeo DOM covers: + + * **DOM Attributes**: + Measures performance of getting and setting a DOM attribute, both via + ``getAttribute`` and via a reflecting DOM property. Also throws in some + expando getting/setting for good measure. + + * **DOM Modification**: + Measures performance of various things that modify the DOM tree: + creating element and text nodes and inserting them into the DOM. + + * **DOM Query**: + Measures performance of various methods of looking for nodes in the DOM: + ``getElementById``, ``getElementsByTagName``, and so forth. + + * **DOM Traversal**: + Measures performance of various accessors (``childNodes``, + ``firstChild``, etc) that would be used when doing a walk over the DOM + tree. + + Please see `dromaeo_css <#dromaeo_css>`_ for examples of data. + * gecko_profile_entries: 10000000 + * gecko_profile_interval: 2 + * tpmanifest: ${talos}/tests/dromaeo/dom.manifest + * unit: score + * Command + + .. code-block:: + + ./mach talos-test -a dromaeo_dom + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g3** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g3-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g3-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: glterrain + :class-container: anchor-id-glterrain + + * contact: :jgilbert and gfx + * source: `glterrain <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/terrain>`__ + * type: `Page load`_ + * data: we load the perftest.html page (which generates 4 metrics to track) 25 times, resulting in 4 sets of 25 data points + * summarization: Measures average frames interval while animating a simple WebGL scene + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 24; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l381>`__ + * suite: `geometric mean`_ of the 4 subtest results. + * description: + | This tests animates a simple WebGL scene (static textured landscape, one + moving light source, rotating viewport) and measure the frames + throughput (expressed as average interval) over 100 frames. It runs in + ASAP mode (vsync off) and measures the same scene 4 times (for all + combination of antialiasing and alpha. It reports the results as 4 + values) one for each combination. Lower results are better. + * **Example Data** + + .. code-block:: + + 0;0.WebGL-terrain-alpha-no-AA-no;19.8189;20.57185;20.5069;21.09645;20.40045;20.89025;20.34285;20.8525;20.45845;20.6499;19.94505;20.05285;20.316049;19.46745;19.46135;20.63865;20.4789;19.97015;19.9546;20.40365;20.74385;20.828649;20.78295;20.51685;20.97069 + 1;1.WebGL-terrain-alpha-no-AA-yes;23.0464;23.5234;23.34595;23.40609;22.54349;22.0554;22.7933;23.00685;23.023649;22.51255;23.25975;23.65819;22.572249;22.9195;22.44325;22.95015;23.3567;23.02089;22.1459;23.04545;23.09235;23.40855;23.3296;23.18849;23.273249 + 2;2.WebGL-terrain-alpha-yes-AA-no;24.01795;23.889449;24.2683;24.34649;23.0562;24.02275;23.54819;24.1874;23.93545;23.53629;23.305149;23.62459;24.01589;24.06405;24.143449;23.998549;24.08205;24.26989;24.0736;24.2346;24.01145;23.7817;23.90785;24.7118;24.2834 + 3;3.WebGL-terrain-alpha-yes-AA-yes;25.91375;25.87005;25.64875;25.15615;25.5475;24.497449;24.56385;25.57529;25.54889;26.31559;24.143949;25.09895;24.75049;25.2087;25.52385;25.9017;25.4439;24.3495;25.9269;25.734449;26.4126;25.547449;25.667249;25.679349;25.9565 + + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 10 + * linux_counters: None + * mac_counters: None + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/webgl/glterrain.manifest + * tpmozafterpaint: False + * tppagecycles: 25 + * unit: frame interval + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a glterrain + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-webgl-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-webgl-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-ref-hw-2017-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: glvideo + :class-container: anchor-id-glvideo + + * contact: :jgilbert and gfx + * source: `glvideo <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/video>`__ + * type: `Page load`_ + * data: 5 cycles of the entire benchmark, each subtest will have 5 data points (see below) + * summarization: WebGL video texture update with 1080p video. Measures mean tick time across 100 ticks. + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 4; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522>`__ + * suite: `geometric mean`_ of the 4 subtest results. + * **Lower is better** + * **Example Data** + + .. code-block:: + + 0;Mean tick time across 100 ticks: ;54.6916;49.0534;51.21645;51.239650000000005;52.44295 + + * description: + | This test playbacks a video file and ask WebGL to draw video frames as + WebGL textures for 100 ticks. It collects the mean tick time across 100 + ticks to measure how much time it will spend for a video texture upload + to be a WebGL texture (gl.texImage2D). We run it for 5 times and ignore + the first found. Lower results are better. + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 2 + * linux_counters: None + * mac_counters: None + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/webgl/glvideo.manifest + * tpmozafterpaint: False + * tppagecycles: 5 + * unit: ms + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a glvideo + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-webgl-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-webgl-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-ref-hw-2017-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: kraken + :class-container: anchor-id-kraken + + * contact: :sdetar, jandem, and SpiderMonkey Team + * source: `kraken.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/kraken>`__ + * type: `Page load`_ + * measuring: JavaScript performance + * reporting: Total time for all tests, in ms (lower is better) + * data: there are 14 subtests in kraken, each subtest is an internal benchmark and generates 10 data points, or 14 sets of 10 data points. + * summarization: + * subtest: For all of the 10 data points, we take the + `mean <https://wiki.mozilla.org/TestEngineering/Performance/Talos/Data#mean>`__ + to report a single number. + * suite: `geometric mean`_ of the 14 subtest results. + * description: + | This is the `Kraken <https://wiki.mozilla.org/Kraken>`__ javascript + benchmark taken verbatim and slightly modified to fit into our + pageloader extension and talos harness. + * **Example Data** + + .. code-block:: + + 0;ai-astar;100;95;98;102;101;99;97;98;98;102 + 1;audio-beat-detection;147;147;191;173;145;139;186;143;183;140 + 2;audio-dft;161;156;158;157;160;158;160;160;159;158 + 3;audio-fft;82;83;83;154;83;83;82;83;160;82 + 4;audio-oscillator;96;96;141;95;95;95;129;96;95;134 + 5;imaging-gaussian-blur;116;115;116;115;115;115;115;115;117;116 + 6;imaging-darkroom;166;164;166;165;166;166;165;165;165;166 + 7;imaging-desaturate;87;87;87;87;88;87;88;87;87;87 + 8;json-parse-financial;75;77;77;76;77;76;77;76;77;77 + 9;json-stringify-tinderbox;79;79;80;79;78;79;79;78;79;79 + 10;stanford-crypto-aes;98;97;96;98;98;98;98;98;113;95 + 11;stanford-crypto-ccm;130;138;130;127;137;134;134;132;147;129 + 12;stanford-crypto-pbkdf2;176;187;183;183;176;174;181;187;175;173 + 13;stanford-crypto-sha256-iterative;86;85;83;84;86;85;85;86;83;83 + + * gecko_profile_entries: 5000000 + * gecko_profile_interval: 1 + * preferences: {'dom.send_after_paint_to_content': False} + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/kraken/kraken.manifest + * tpmozafterpaint: False + * tppagecycles: 1 + * unit: score + * Command + + .. code-block:: + + ./mach talos-test -a kraken + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-dromaeojs-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-dromaeojs-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-dromaeojs** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: motionmark_animometer + :class-container: anchor-id-motionmark_animometer + + * contact: :b0bh00d, :jeffm, and gfx + * source: `source <https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark>`__ `manifests <https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/motionmark>`__ + * type: `Page load`_ + * measuring: benchmark measuring the time to animate complex scenes + * summarization: + * subtest: FPS from the subtest, each subtest is run for 15 seconds, + repeat this 5 times and report the median value + * suite: we take a geometric mean of all the subtests (9 for + animometer, 11 for html suite) + * tpmanifest: ${talos}/tests/motionmark/animometer.manifest + * Command + + .. code-block:: + + ./mach talos-test -a motionmark_animometer + + * **Test Task**: + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: motionmark_htmlsuite + :class-container: anchor-id-motionmark_htmlsuite + + * contact: :jrmuizel and graphics(gfx) team + * tpmanifest: ${talos}/tests/motionmark/htmlsuite.manifest + * Command + + .. code-block:: + + ./mach talos-test -a motionmark_htmlsuite + + * **Test Task**: + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-motionmark-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + +.. dropdown:: motionmark_webgl + :class-container: anchor-id-motionmark_webgl + + * contact: :jgilbert and gfx + * source: `source <https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark>`__ `manifest <https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/motionmark/webgl.manifest>`__ + * type: `Page load`_ + * measuring: Draw call performance in WebGL + * summarization: + * subtest: FPS from the subtest, each subtest is run once for 15 + seconds, report the average FPS over that time. + * suite: identical to subtest + * timeout: 600 + * tpmanifest: ${talos}/tests/motionmark/webgl.manifest + * unit: fps + * Command + + .. code-block:: + + ./mach talos-test -a motionmark_webgl + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-webgl-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-webgl-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-ref-hw-2017-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: pdfpaint + :class-container: anchor-id-pdfpaint + + * contact: :calixte and CI and Quality Tools team + * source: + * type: `Page load`_ + * reporting: time from *performance.timing.navigationStart* to *pagerendered* event in ms (lower is better) + * data: load a PDF 20 times + * gecko_profile_entries: 1000000 + * pdfpaint: True + * preferences: {'pdfjs.eventBusDispatchToDOM': True} + * timeout: 600 + * tpmanifest: ${talos}/tests/pdfpaint/pdfpaint.manifest + * tppagecycles: 20 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a pdfpaint + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: perf_reftest + :class-container: anchor-id-perf_reftest + + * contact: :emilio and css/layout team + * source: `perf-reftest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest>`__ + * type: `Page load`_ + * reporting: intervals in ms (lower is better) + * data: each test loads 25 times + * summarization: + * subtest: `ignore first`_ 5 data points, then take the `median`_ of the remaining 20 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734>`__ + * suite: identical to subtest + * description: + | **Important note:** This test now requires an 'opt' build. If the + perf-reftest is ran on a non-opt build, it will time out (more + specifically on innertext-1.html, and possibly others in the future). + + Style system performance test suite. The perf-reftest suite is a unique + talos suite where each subtest loads two different test pages: a 'base' + page (i.e. bloom_basic) and a 'reference' page (i.e. bloom_basic_ref), + and then compares each of the page load times against eachother to + determine the variance. + + Talos runs each of the two pages as if they are stand-alone tests, and + then calculates and reports the variance; the test output 'replicates' + reported from bloom_basic are actually the comparisons between the + 'base' and 'reference' pages for each page load cycle. The suite + contains multiple subtests, each of which contains a base page and a + reference page. + + If you wish to see the individual 'base' and 'reference' page results + instead of just the reported difference, the 'base_replicates' and + 'ref_replicates' can be found in the PERFHERDER_DATA log file output, + and in the 'local.json' talos output file when running talos locally. In + production, both of the page replicates are also archived in the + perfherder-data.json file. The perfherder-data.json file is archived + after each run in production, and can be found on the Treeherder Job + Details tab when the perf-reftest job symbol is selected. + + This test suite was ported over from the `style-perf-tests <https://github.com/heycam/style-perf-tests>`__. + * **Example Data** + + .. code-block:: + + "replicates": [1.185, 1.69, 1.22, 0.36, 11.26, 3.835, 3.315, 1.355, 3.185, 2.485, 2.2, 1.01, 0.9, 1.22, 1.9, + 0.285, 1.52, 0.31, 2.58, 0.725, 2.31, 2.67, 3.295, 1.57, 0.3], "value": 1.7349999999999999, "unit": "ms", + "base_replicates": [166.94000000000003, 165.16, 165.64000000000001, 165.04000000000002, 167.355, 165.175, + 165.325, 165.11, 164.175, 164.78, 165.555, 165.885, 166.83499999999998, 165.76500000000001, 164.375, 166.825, + 167.13, 166.425, 169.22500000000002, 164.955, 165.335, 164.45000000000002, 164.85500000000002, 165.005, 166.035]}], + "ref_replicates": [165.755, 166.85000000000002, 166.85999999999999, 165.4, 178.615, 169.01, 168.64, 166.465, + 167.36, 167.265, 167.75500000000002, 166.895, 167.735, 166.985, 166.275, 166.54000000000002, 165.61, 166.115, + 166.64499999999998, 165.68, 167.64499999999998, 167.12, 168.15, 166.575, 166.335], + + * alert_threshold: 5.0 + * base_vs_ref: True + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 1 + * lower_is_better: True + * subtest_alerts: True + * tpcycles: 1 + * tpmanifest: ${talos}/tests/perf-reftest/perf_reftest.manifest + * tppagecycles: 10 + * tptimeout: 30000 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a perf_reftest + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-perf-reftest-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-perf-reftest-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: perf_reftest_singletons + :class-container: anchor-id-perf_reftest_singletons + + * contact: :emelio and Layout team + * source: `perf-reftest-singletons <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest-singletons>`__ + * type: `Page load`_ + * reporting: intervals in ms (lower is better) + * data: each test loads 25 times + * summarization: + * subtest: `ignore first`_ 5 data points, then take the `median`_ of the remaining 20 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734>`__ + * suite: identical to subtest + * description: + | Individual style system performance tests. The perf-reftest-singletons + suite runs the perf-reftest 'base' pages (i.e. bloom_basic) test + individually, and reports the values for that single test page alone, + NOT the comparison of two different pages. There are multiple subtests + in this suite, each just containing the base page on its own. + + This test suite was ported over from the `style-perf-tests <https://github.com/heycam/style-perf-tests>`__. + * **Example Data** + + .. code-block:: + + bloombasic.html;88.34000000000003;88.66499999999999;94.815;92.60500000000002;95.30000000000001; + 98.80000000000001;91.975;87.73500000000001;86.925;86.965;93.00500000000001;98.93;87.45000000000002; + 87.14500000000001;92.78500000000001;86.96499999999999;98.32000000000001;97.485;90.67000000000002; + 86.72500000000001;95.665;100.67;101.095;94.32;91.87 + + * alert_threshold: 5.0 + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 1 + * lower_is_better: True + * subtest_alerts: True + * suite_should_alert: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/perf-reftest-singletons/perf_reftest_singletons.manifest + * tppagecycles: 15 + * tptimeout: 30000 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a perf_reftest_singletons + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-perf-reftest-singletons-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-perf-reftest-singletons-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-perf-reftest-singletons** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: rasterflood_gradient + :class-container: anchor-id-rasterflood_gradient + + * contact: :jrmuizel, :jimm, and gfx + * source: `rasterflood_gradient.html <https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_gradient.html>`__ + * type: `Page load`_ + * data: we load the rasterflood_gradient.html page ten times, computing a score each time, generating 10 data points. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 9; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986>`__ + * description: + | This page animates some complex gradient patterns in a + requestAnimationFrame callback. However, it also churns the CPU during + each callback, spinning an empty loop for 14ms each frame. The intent is + that, if we consider the rasterization costs to be 0, then the animation + should run close to 60fps. Otherwise it will lag. Since rasterization + costs are not 0, the lower we can get them, the faster the test will + run. The test runs in ASAP mode to maximize framerate. + + The test runs for 10 seconds, and the resulting score is how many frames + we were able to render during that time. Higher is better. Improvements + (or regressions) to general painting performance or gradient rendering + will affect this benchmark. + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 2 + * linux_counters: None + * lower_is_better: False + * mac_counters: None + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/gfx/rasterflood_gradient.manifest + * tpmozafterpaint: False + * tppagecycles: 10 + * unit: score + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a rasterflood_gradient + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g4-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g4-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: rasterflood_svg + :class-container: anchor-id-rasterflood_svg + + * contact: :jrmuizel, :jimm, and gfx + * source: `rasterflood_svg.html <https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_svg.html>`__ + * type: `Page load`_ + * data: we load the rasterflood_svg.html page ten times, measuring pageload each time, generating 10 data points. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 9; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986>`__ + * description: + | This page animates some complex SVG patterns in a requestAnimationFrame + callback. However, it also churns the CPU during each callback, spinning + an empty loop for 14ms each frame. The intent is that, if we consider + the rasterization costs to be 0, then the animation should run close to + 60fps. Otherwise it will lag. Since rasterization costs are not 0, the + lower we can get them, the faster the test will run. The test runs in + ASAP mode to maximize framerate. The result is how quickly the browser + is able to render 600 frames of the animation. + + Improvements (or regressions) to general painting performance or SVG are + likely to affect this benchmark. + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 2 + * linux_counters: None + * mac_counters: None + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/gfx/rasterflood_svg.manifest + * tpmozafterpaint: False + * tppagecycles: 10 + * unit: ms + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a rasterflood_svg + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g4-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g4-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g4** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: sessionrestore + :class-container: anchor-id-sessionrestore + + * contact: :dale, :dao, :farre, session restore module owners/peers, and DOM team + * source: `talos/sessionrestore <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/sessionrestore>`__ + * bug: `bug 936630 <https://bugzilla.mozilla.org/show_bug.cgi?id=936630>`__, + `bug 1331937 <https://bugzilla.mozilla.org/show_bug.cgi?id=1331937>`__, + `bug 1531520 <https://bugzilla.mozilla.org/show_bug.cgi?id=1531520>`__ + * type: Startup_ + * measuring: time spent reading and restoring the session. + * reporting: interval in ms (lower is better). + * data: we load the session restore index page 10 times to collect 1 set of 10 data points. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 9 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305>`__ + * suite: identical to subtest + * description: + | Three tests measure the time spent reading and restoring the session + from a valid sessionstore.js. Time is counted from the *process start* + until the *sessionRestored* event. + + In *sessionrestore*, this is tested with a configuration that requires + the session to be restored. In *sessionrestore_no_auto_restore*, this is + tested with a configuration that requires the session to not be + restored. Both of the above tests use a sessionstore.js file that + contains one window and roughly 89 tabs. In + *sessionrestore_many_windows*, this is tested with a sessionstore.js + that contains 3 windows and 130 tabs. The first window contains 50 tabs, + 80 remaning tabs are divided equally between the second and the third + window. + * **Example Data** + + .. code-block:: + + [2362.0, 2147.0, 2171.0, 2134.0, 2116.0, 2145.0, 2141.0, 2141.0, 2136.0, 2080.0] + + * cycles: 10 + * extensions: ['${talos}/startup_test/sessionrestore/addon'] + * gecko_profile_entries: 10000000 + * gecko_profile_startup: True + * pine: False + * preferences: {'browser.startup.page': 3} + * profile_path: ${talos}/startup_test/sessionrestore/profile + * reinstall: ['sessionstore.jsonlz4', 'sessionstore.js', 'sessionCheckpoints.json'] + * timeout: 900 + * unit: ms + * url: about:home + * Command + + .. code-block:: + + ./mach talos-test -a sessionrestore + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: sessionrestore_many_windows + :class-container: anchor-id-sessionrestore_many_windows + + * See `sessionrestore <#sessionrestore>`_. + * profile_path: ${talos}/startup_test/sessionrestore/profile-manywindows + * Command + + .. code-block:: + + ./mach talos-test -a sessionrestore_many_windows + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-sessionrestore-many-windows-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-sessionrestore-many-windows-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-sessionrestore-many-windows** + - ✅ + - ✅ + - ❌ + - ❌ + + + +.. dropdown:: sessionrestore_no_auto_restore + :class-container: anchor-id-sessionrestore_no_auto_restore + + * See `sessionrestore <#sessionrestore>`_. + * preferences: {'browser.startup.page': 1, 'talos.sessionrestore.norestore': True} + * timeout: 300 + * Command + + .. code-block:: + + ./mach talos-test -a sessionrestore_no_auto_restore + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: startup_about_home_paint + :class-container: anchor-id-startup_about_home_paint + + * contact: :mconley, Firefox Desktop Front-end team, :gijs, :fqueze, and :dthayer + * source: `addon <https://hg.mozilla.org/mozilla-central/file/tip/testing/talos/talos/startup_test/startup_about_home_paint/addon/>`__ + * type: Startup_ + * measuring: The time from process start to the point where the about:home page reports that it has painted the Top Sites. + * data: we load restart the browser 20 times, and collect a timestamp for each run. + * reporting: test time in ms (lower is better) + * **Example Data** + + .. code-block:: + + [1503.0, 1497.0, 1523.0, 1536.0, 1511.0, 1485.0, 1594.0, 1580.0, 1531.0, 1471.0, 1502.0, 1520.0, 1488.0, 1533.0, 1531.0, 1502.0, 1486.0, 1489.0, 1487.0, 1475.0] + + * cycles: 20 + * extensions: ['${talos}/startup_test/startup_about_home_paint/addon'] + * pine: False + * preferences: {'browser.startup.homepage.abouthome_cache.enabled': False} + * timeout: 600 + * tpmanifest: ${talos}/startup_test/startup_about_home_paint/startup_about_home_paint.manifest + * url: None + * Command + + .. code-block:: + + ./mach talos-test -a startup_about_home_paint + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: startup_about_home_paint_cached + :class-container: anchor-id-startup_about_home_paint_cached + + * contact: :mconley, Firefox Desktop Front-end team, :gijs, :fqueze, and :dthayer + * See `startup_about_home_paint <#startup_about_home_paint>`_. + * description: + | Tests loading about:home on startup with the about:home startup cache enabled. + * cycles: 20 + * extensions: ['${talos}/startup_test/startup_about_home_paint/addon'] + * pine: False + * preferences: {'browser.startup.homepage.abouthome_cache.enabled': True} + * tpmanifest: ${talos}/startup_test/startup_about_home_paint/startup_about_home_paint.manifest + * url: None + * Command + + .. code-block:: + + ./mach talos-test -a startup_about_home_paint_cached + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: startup_about_home_paint_realworld_webextensions + :class-container: anchor-id-startup_about_home_paint_realworld_webextensions + + * contact: :mconley, Firefox Desktop Front-end team, :gijs, :fqueze, and :dthayer + * source: `addon <https://hg.mozilla.org/mozilla-central/file/tip/testing/talos/talos/startup_test/startup_about_home_paint/addon/>`__ + * type: Startup_, `Real-world WebExtensions`_ + * measuring: The time from process start to the point where the about:home page reports that it has painted the Top Sites when 5 popular, real-world WebExtensions are installed and enabled. + * data: we install the 5 real-world WebExtensions, then load and restart the browser 20 times, and collect a timestamp for each run. + * reporting: test time in ms (lower is better) + * **Example Data** + + .. code-block:: + + [1503.0, 1497.0, 1523.0, 1536.0, 1511.0, 1485.0, 1594.0, 1580.0, 1531.0, 1471.0, 1502.0, 1520.0, 1488.0, 1533.0, 1531.0, 1502.0, 1486.0, 1489.0, 1487.0, 1475.0] + + * cycles: 20 + * extensions: ['${talos}/startup_test/startup_about_home_paint/addon', '${talos}/getinfooffline'] + * pine: False + * preferences: {'browser.startup.homepage.abouthome_cache.enabled': False} + * tpmanifest: ${talos}/startup_test/startup_about_home_paint/startup_about_home_paint.manifest + * url: None + * webextensions_folder: ${talos}/webextensions + * Command + + .. code-block:: + + ./mach talos-test -a startup_about_home_paint_realworld_webextensions + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-realworld-webextensions-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-realworld-webextensions-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-realworld-webextensions** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: stylebench + :class-container: anchor-id-stylebench + + * contact: :emilio and Layout team + * source: `stylebench.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/stylebench/stylebench.manifest>`__ + * type: `Page load`_ + * measuring: speed of dynamic style recalculation + * reporting: runs/minute score + * tpmanifest: ${talos}/tests/stylebench/stylebench.manifest + * Command + + .. code-block:: + + ./mach talos-test -a stylebench + + +.. dropdown:: tabpaint + :class-container: anchor-id-tabpaint + + * contact: :mconley, Firefox Desktop Front-end team, :gijs, :fqueze, and :dthayer + * source: `tabpaint <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tabpaint>`__ + * bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1253382 + * type: `Page load`_ + * measuring: + * The time it takes to paint the content of a newly opened tab when + the tab is opened from the parent (ex: by hitting Ctrl-T) + * The time it takes to paint the content of a newly opened tab when + the tab is opened from content (ex: by clicking on a target="_blank" link) + * **NOT** measuring: + * The time it takes to animate the tabs. That's the responsibility + of the TART test. tabpaint is strictly concerned with the painting of the web content. + * data: we load the tabpaint trigger page 20 times, each run produces + two values (the time it takes to paint content when opened from the + parent, and the time it takes to paint content when opened from + content), resulting in 2 sets of 20 data points. + * **Example Data** + + .. code-block:: + + 0;tabpaint-from-parent;105;76;66;64;64;69;65;63;70;68;64;60;65;63;54;61;64;67;61;64 + 1;tabpaint-from-content;129;68;72;72;70;78;86;85;82;79;120;92;76;80;74;82;76;89;77;85 + + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 19 data points + * suite: geometric_mean(subtests) + * extensions: ['${talos}/tests/tabpaint', '${talos}/pageloader'] + * gecko_profile_entries: 1000000 + * preferences: {'browser.link.open_newwindow': 3, 'browser.link.open_newwindow.restriction': 2, 'browser.newtab.preload': False} + * timeout: 600 + * tploadnocache: True + * tpmanifest: ${talos}/tests/tabpaint/tabpaint.manifest + * tppagecycles: 20 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tabpaint + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tabswitch + :class-container: anchor-id-tabswitch + + * contact: :mconley, Firefox Desktop Front-end team, :gijs, :fqueze, and :dthayer + * source: `tabswitch <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tabswitch>`__ + * bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1166132 + * type: `Page load`_ + * measuring: + * The time between switching a tab and the first paint to the content area + * reporting: + * data: we load 50 web pages 5 times each, resulting in 50 sets of 5 data points. + * **To run it locally**, you'd need `tp5n.zip <#Page_sets>`__. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 4 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305>`__ + * suite: `geometric mean`_ of the 50 subtest results. + * **Example Data** + + .. code-block:: + + 0;amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/507846.html;66.34;54.15;53.08;55.79;49.12 + 1;cgi.ebay.com/cgi.ebay.com/ALL-NEW-KINDLE-3-eBOOK-WIRELESS-READING-DEVICE-W-WIFI-/130496077314@pt=LH_DefaultDomain_0&hash=item1e622c1e02.html;50.85;46.57;39.51;36.71;36.47 + 2;163.com/www.163.com/index.html;95.05;80.80;76.09;69.29;68.96 + 3;mail.ru/mail.ru/index.html;66.21;52.04;56.33;55.11;45.61 + 4;bbc.co.uk/www.bbc.co.uk/news/index.html;35.80;44.59;48.02;45.71;42.58 + 5;store.apple.com/store.apple.com/us@mco=Nzc1MjMwNA.html;52.98;49.45;58.47;56.79;61.29 + 6;imdb.com/www.imdb.com/title/tt1099212/index.html;46.51;55.12;46.22;50.60;47.63 + 7;cnn.com/www.cnn.com/index.html;43.02;50.77;43.88;49.70;50.02 + 8;sohu.com/www.sohu.com/index.html;74.03;62.89;63.30;67.71;89.35 + 9;youku.com/www.youku.com/index.html;43.98;52.69;45.80;63.00;57.02 + 10;ifeng.com/ifeng.com/index.html;88.01;87.54;104.47;94.93;113.91 + 11;tudou.com/www.tudou.com/index.html;45.32;48.10;46.03;39.26;58.38 + 12;chemistry.about.com/chemistry.about.com/index.html;38.24;37.07;39.59;39.48;39.60 + 13;beatonna.livejournal.com/beatonna.livejournal.com/index.html;35.59;50.75;36.17;48.49;52.61 + 14;rakuten.co.jp/www.rakuten.co.jp/index.html;90.28;71.95;62.66;60.33;67.76 + 15;uol.com.br/www.uol.com.br/index.html;42.89;48.05;53.77;40.02;42.41 + 16;thepiratebay.org/thepiratebay.org/top/201.html;40.46;42.46;47.63;57.66;45.49 + 17;page.renren.com/page.renren.com/index.html;47.61;66.78;47.91;62.78;47.19 + 18;chinaz.com/chinaz.com/index.html;50.34;58.17;118.43;55.47;63.80 + 19;globo.com/www.globo.com/index.html;41.34;38.52;42.82;53.14;45.20 + 20;spiegel.de/www.spiegel.de/index.html;33.60;34.34;36.25;36.18;47.04 + 21;dailymotion.com/www.dailymotion.com/us.html;37.68;36.13;39.52;37.15;42.79 + 22;goo.ne.jp/goo.ne.jp/index.html;50.74;47.30;63.04;58.41;58.96 + 23;stackoverflow.com/stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered.html;44.66;44.40;43.39;47.38;57.65 + 24;ezinearticles.com/ezinearticles.com/index.html@Migraine-Ocular---The-Eye-Migraines&id=4684133.html;37.38;45.01;40.29;36.26;39.28 + 25;huffingtonpost.com/www.huffingtonpost.com/index.html;39.57;43.35;55.01;44.13;58.28 + 26;media.photobucket.com/media.photobucket.com/image/funny%20gif/findstuff22/Best%20Images/Funny/funny-gif1.jpg@o=1.html;39.77;42.46;75.54;42.38;47.72 + 27;imgur.com/imgur.com/gallery/index.html;34.72;37.37;46.74;40.93;37.08 + 28;reddit.com/www.reddit.com/index.html;42.47;39.89;51.54;51.51;41.68 + 29;noimpactman.typepad.com/noimpactman.typepad.com/index.html;54.28;47.40;52.38;52.15;50.97 + 30;myspace.com/www.myspace.com/albumart.html;48.97;64.12;61.66;48.32;68.53 + 31;mashable.com/mashable.com/index.html;36.76;40.95;35.30;53.86;42.76 + 32;dailymail.co.uk/www.dailymail.co.uk/ushome/index.html;42.06;40.64;44.24;37.32;61.35 + 33;whois.domaintools.com/whois.domaintools.com/mozilla.com.html;34.73;35.23;39.25;48.24;35.72 + 34;indiatimes.com/www.indiatimes.com/index.html;52.67;55.51;46.29;52.69;58.82 + 35;reuters.com/www.reuters.com/index.html;32.92;33.08;36.95;39.23;39.27 + 36;xinhuanet.com/xinhuanet.com/index.html;125.85;102.56;138.89;130.34;147.45 + 37;56.com/www.56.com/index.html;63.89;75.00;61.45;62.20;58.67 + 38;bild.de/www.bild.de/index.html;35.61;43.74;34.79;33.45;31.83 + 39;guardian.co.uk/www.guardian.co.uk/index.html;38.91;55.93;62.34;42.63;45.99 + 40;naver.com/www.naver.com/index.html;78.10;89.07;127.67;75.18;109.32 + 41;yelp.com/www.yelp.com/biz/alexanders-steakhouse-cupertino.html;42.54;46.92;39.19;49.82;50.43 + 42;wsj.com/online.wsj.com/home-page.html;46.43;55.51;44.16;81.79;48.78 + 43;google.com/www.google.com/search@q=mozilla.html;35.62;36.71;44.47;45.00;40.22 + 44;xunlei.com/xunlei.com/index.html;67.57;60.69;83.83;85.53;85.08 + 45;aljazeera.net/aljazeera.net/portal.html;65.03;51.84;73.29;64.77;69.70 + 46;w3.org/www.w3.org/standards/webdesign/htmlcss.html;53.57;58.50;72.98;66.95;55.62 + 47;homeway.com.cn/www.hexun.com/index.html;105.59;117.32;108.95;116.10;70.32 + 48;youtube.com/www.youtube.com/music.html;40.53;41.48;59.67;40.81;40.07 + 49;people.com.cn/people.com.cn/index.html;96.49;103.64;115.12;66.05;117.84 + + * extensions: ['${talos}/tests/tabswitch', '${talos}/pageloader'] + * gecko_profile_entries: 5000000 + * preferences: {'addon.test.tabswitch.urlfile': '${talos}/tests/tp5o.html', 'addon.test.tabswitch.webserver': '${webserver}', 'addon.test.tabswitch.maxurls': -1, 'browser.toolbars.bookmarks.visibility': 'never'} + * timeout: 900 + * tploadnocache: True + * tpmanifest: ${talos}/tests/tabswitch/tabswitch.manifest + * tppagecycles: 5 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tabswitch + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-tabswitch-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tabswitch** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tart + :class-container: anchor-id-tart + + * contact: :mconley, Firefox Desktop Front-end team, :gijs, :fqueze, and :dthayer + * source: `tart <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tart>`__ + * type: `Page load`_ + * measuring: Desktop Firefox UI animation speed and smoothness + * reporting: intervals in ms (lower is better) + * see below for details + * data: there are 30 reported subtests from TART which we load 25 times, resulting in 30 sets of 25 data points. + * summarization: + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 24 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305>`__ + * suite: `geometric mean`_ of the 30 subtest results. + * TART is the **Tab Animation Regression Test**. + * TART tests tab animation on these cases: + * Simple: single new tab of about:blank open/close without affecting (shrinking/expanding) other tabs. + * icon: same as above with favicons and long title instead of about:blank. + * Newtab: newtab open with thumbnails preview + * without affecting other tabs, with and without preload. + * Fade: opens a tab, then measures fadeout/fadein (tab animation without the overhead of opening/closing a tab). + * Case 1 is tested with DPI scaling of 1. + * Case 2 is tested with DPI scaling of 1.0 and 2.0. + * Case 3 is tested with the default scaling of the test system. + * Case 4 is tested with DPI scaling of 2.0 with the "icon" tab (favicon and long title). + * Each animation produces 3 test results: + * error: difference between the designated duration and the actual completion duration from the trigger. + * half: average frame interval over the 2nd half of the animation. + * all: average frame interval over all recorded intervals. + * And the run logs also include the explicit intervals from which these 3 values were derived. + * **Example Data** + + .. code-block:: + + 0;simple-open-DPI1.half.TART;2.35;2.42;2.63;2.47;2.71;2.38;2.37;2.41;2.48;2.70;2.44;2.41;2.51;2.43;2.41;2.56;2.76;2.49;2.36;2.40;2.70;2.53;2.35;2.46;2.47 + 1;simple-open-DPI1.all.TART;2.80;2.95;3.12;2.92;3.46;2.87;2.92;2.99;2.89;3.24;2.94;2.95;3.25;2.92;3.02;3.00;3.21;3.31;2.84;2.87;3.10;3.13;3.10;2.94;2.95 + 2;simple-open-DPI1.error.TART;33.60;36.33;35.93;35.97;38.17;34.77;36.00;35.01;36.25;36.22;35.24;35.76;36.64;36.31;34.74;34.40;34.34;41.48;35.04;34.83;34.27;34.04;34.37;35.22;36.52 + 3;simple-close-DPI1.half.TART;1.95;1.88;1.91;1.94;2.00;1.97;1.88;1.76;1.84;2.18;1.99;1.83;2.04;1.93;1.81;1.77;1.79;1.90;1.82;1.84;1.78;1.75;1.76;1.89;1.81 + 4;simple-close-DPI1.all.TART;2.19;2.08;2.07;2.32;2.65;2.32;2.26;1.96;2.02;2.26;2.05;2.16;2.19;2.11;2.04;1.98;2.05;2.02;2.01;2.11;1.97;1.97;2.05;2.01;2.12 + 5;simple-close-DPI1.error.TART;21.35;23.87;22.60;22.02;22.97;22.35;22.15;22.79;21.81;21.90;22.26;22.58;23.15;22.43;22.76;23.36;21.86;22.70;22.96;22.70;22.28;22.03;21.78;22.33;22.23 + 6;icon-open-DPI1.half.TART;2.42;2.33;2.50;2.58;2.36;2.51;2.60;2.35;2.52;2.51;2.59;2.34;3.29;2.63;2.46;2.57;2.53;2.50;2.39;2.51;2.44;2.66;2.72;2.36;2.52 + 7;icon-open-DPI1.all.TART;3.12;2.94;3.42;3.23;3.10;3.21;3.33;3.14;3.24;3.32;3.46;2.90;3.65;3.19;3.27;3.47;3.32;3.13;2.95;3.23;3.21;3.33;3.47;3.15;3.32 + 8;icon-open-DPI1.error.TART;38.39;37.96;37.03;38.85;37.03;37.17;37.19;37.56;36.67;36.33;36.89;36.85;37.54;38.46;35.38;37.52;36.68;36.48;36.03;35.71;37.12;37.08;37.74;38.09;35.85 + 9;icon-close-DPI1.half.TART;1.94;1.93;1.79;1.89;1.83;1.83;1.90;1.89;1.75;1.76;1.77;1.74;1.81;1.86;1.95;1.99;1.73;1.83;1.97;1.80;1.94;1.84;2.01;1.88;2.03 + 10;icon-close-DPI1.all.TART;2.14;2.14;1.98;2.03;2.02;2.25;2.29;2.13;1.97;2.01;1.94;2.01;1.99;2.05;2.11;2.09;2.02;2.02;2.12;2.02;2.20;2.11;2.19;2.07;2.27 + 11;icon-close-DPI1.error.TART;24.51;25.03;25.17;24.54;23.86;23.70;24.02;23.61;24.10;24.53;23.92;23.75;23.73;23.78;23.42;25.40;24.21;24.55;23.96;24.96;24.41;24.96;24.16;24.20;23.65 + 12;icon-open-DPI2.half.TART;2.60;2.60;2.60;2.53;2.51;2.53;2.59;2.43;2.66;2.60;2.47;2.61;2.64;2.43;2.49;2.63;2.61;2.60;2.52;3.06;2.65;2.74;2.69;2.44;2.43 + 13;icon-open-DPI2.all.TART;3.45;3.22;3.38;3.47;3.10;3.31;3.47;3.13;3.37;3.14;3.28;3.20;3.40;3.15;3.14;3.23;3.41;3.16;3.26;3.55;3.29;3.49;3.44;3.11;3.22 + 14;icon-open-DPI2.error.TART;40.20;37.86;37.53;41.46;37.03;38.77;37.48;36.97;37.28;37.72;36.09;36.71;38.89;38.21;37.37;38.91;36.79;36.10;37.60;36.99;37.56;35.76;38.92;37.46;37.52 + 15;icon-close-DPI2.half.TART;2.27;1.97;1.83;1.86;2.08;1.88;1.80;1.90;1.77;1.89;2.06;1.89;1.89;1.86;2.01;1.79;1.78;1.83;1.89;1.85;1.74;1.82;1.84;1.81;1.79 + 16;icon-close-DPI2.all.TART;2.33;2.13;2.18;2.03;2.33;2.03;1.95;2.06;1.96;2.13;2.25;2.10;2.13;2.03;2.18;2.00;2.05;2.01;2.08;2.05;1.96;2.04;2.10;2.04;2.08 + 17;icon-close-DPI2.error.TART;22.99;23.02;24.32;23.58;23.05;23.34;22.92;23.22;22.90;23.33;23.33;23.05;22.80;23.45;24.05;22.39;22.14;22.97;22.85;22.13;22.89;22.98;23.69;22.99;23.08 + 18;iconFade-close-DPI2.half.TART;2.14;1.84;1.78;1.84;1.66;2.07;1.81;3.82;1.68;1.85;1.62;2.54;2.06;1.85;2.17;1.80;1.71;1.67;2.11;1.73;2.94;2.14;1.93;1.72;2.05 + 19;iconFade-close-DPI2.all.TART;2.17;1.76;1.80;1.89;1.70;1.93;1.80;3.38;1.78;1.90;1.70;2.50;1.94;1.81;2.29;1.82;1.79;1.76;2.23;1.80;2.85;2.06;1.84;1.83;2.09 + 20;iconFade-close-DPI2.error.TART;4.67;4.11;3.69;4.51;4.46;3.88;4.54;3.68;4.56;3.82;4.32;4.87;4.42;3.72;3.72;4.54;4.93;4.46;4.64;3.39;4.09;3.28;3.58;4.11;3.80 + 21;iconFade-open-DPI2.half.TART;2.37;2.61;2.37;2.62;2.54;2.84;2.57;2.44;4.33;2.57;2.59;2.67;2.58;2.48;2.38;2.39;2.50;2.39;2.50;2.57;2.52;2.55;2.47;2.69;2.41 + 22;iconFade-open-DPI2.all.TART;2.45;2.64;2.39;2.60;2.57;2.60;2.61;2.59;3.14;2.55;2.54;2.66;2.57;2.48;2.47;2.46;2.55;2.45;2.51;2.61;2.54;2.58;2.50;2.54;2.40 + 23;iconFade-open-DPI2.error.TART;3.64;4.67;4.31;5.79;6.43;3.64;4.82;8.68;5.78;4.38;3.80;3.98;4.64;653.63;4.63;3.76;4.23;5.01;5.48;4.99;3.48;5.10;5.02;6.14;5.58 + 24;newtab-open-preload-no.half.TART;5.02;2.90;3.06;3.03;2.94;2.94;3.08;3.12;3.60;3.19;2.82;2.96;3.67;7.85;2.79;3.12;3.18;2.92;2.86;2.96;2.96;3.00;2.90;2.97;2.94 + 25;newtab-open-preload-no.all.TART;7.11;4.66;5.03;4.68;4.50;4.58;4.76;4.76;5.67;4.96;4.36;4.51;5.21;11.16;4.38;4.69;4.77;4.45;4.45;4.70;4.51;4.61;4.54;4.69;4.60 + 26;newtab-open-preload-no.error.TART;40.82;40.85;37.38;37.40;36.30;36.47;36.89;37.63;37.12;38.65;36.73;36.95;36.11;38.59;37.39;37.77;37.93;37.54;37.46;38.29;36.58;38.25;38.32;37.92;36.93 + 27;newtab-open-preload-yes.half.TART;3.14;2.96;2.97;8.37;2.98;3.00;2.96;3.05;3.12;3.48;3.07;3.23;3.05;2.88;2.92;3.06;2.90;3.01;3.19;2.90;3.18;3.11;3.04;3.16;3.21 + 28;newtab-open-preload-yes.all.TART;5.10;4.60;4.63;8.94;5.01;4.69;4.63;4.67;4.93;5.43;4.78;5.12;4.77;4.65;4.50;4.78;4.75;4.63;4.76;4.45;4.86;4.88;4.69;4.86;4.92 + 29;newtab-open-preload-yes.error.TART;35.90;37.24;38.57;40.60;36.04;38.12;38.78;36.73;36.91;36.69;38.12;36.69;37.79;35.80;36.11;38.01;36.59;38.85;37.14;37.30;38.02;38.95;37.64;37.86;36.43 + + * extensions: ['${talos}/pageloader', '${talos}/tests/tart/addon'] + * gecko_profile_entries: 1000000 + * gecko_profile_interval: 10 + * linux_counters: None + * mac_counters: None + * pine: False + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * tpcycles: 1 + * tploadnocache: True + * tpmanifest: ${talos}/tests/tart/tart.manifest + * tpmozafterpaint: False + * tppagecycles: 25 + * unit: ms + * w7_counters: None + * win_counters: None + * Command + + .. code-block:: + + ./mach talos-test -a tart + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tp5 + :class-container: anchor-id-tp5 + + * description: + | Note that the tp5 test no longer exists (only talos-tp5o) though many + tests still make use of this pageset. Here, we provide an overview of + the tp5 pageset and some information about how data using the tp5 + pageset might be used in various suites. + * Command + + .. code-block:: + + ./mach talos-test -a tp5 + + +.. dropdown:: tp5n + :class-container: anchor-id-tp5n + + * contact: fx-perf@mozilla.com + * description: + | The tp5 is an updated web page test set to 100 pages from April 8th, 2011. Effort was made for the pages to no longer be splash screens/login pages/home pages but to be pages that better reflect the actual content of the site in question. + * cleanup: ${talos}/xtalos/parse_xperf.py -c ${talos}/bcontroller.json + * cycles: 1 + * linux_counters: [] + * mac_counters: [] + * mainthread: True + * multidomain: True + * preferences: {'extensions.enabledScopes': '', 'talos.logfile': 'browser_output.txt'} + * resolution: 20 + * setup: ${talos}/xtalos/start_xperf.py -c ${talos}/bcontroller.json + * timeout: 1800 + * tpcycles: 1 + * tpmanifest: ${talos}/fis/tp5n/tp5n.manifest + * tpmozafterpaint: True + * tppagecycles: 1 + * tptimeout: 10000 + * unit: ms + * w7_counters: [] + * win_counters: [] + * xperf_counters: ['main_startup_fileio', 'main_startup_netio', 'main_normal_fileio', 'main_normal_netio', 'nonmain_startup_fileio', 'nonmain_normal_fileio', 'nonmain_normal_netio', 'mainthread_readcount', 'mainthread_readbytes', 'mainthread_writecount', 'mainthread_writebytes', 'time_to_session_store_window_restored_ms'] + * xperf_providers: ['PROC_THREAD', 'LOADER', 'HARD_FAULTS', 'FILENAME', 'FILE_IO', 'FILE_IO_INIT'] + * xperf_stackwalk: ['FileCreate', 'FileRead', 'FileWrite', 'FileFlush', 'FileClose'] + * xperf_user_providers: ['Mozilla Generic Provider', 'Microsoft-Windows-TCPIP'] + * Command + + .. code-block:: + + ./mach talos-test -a tp5n + + * **Test Task**: + + .. list-table:: **test-windows11-64-2009-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-xperf** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-xperf** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tp5o + :class-container: anchor-id-tp5o + + * contact: :davehunt, and perftest team + * source: `tp5n.zip <#page-sets>`__ + * type: `Page load`_ + * data: we load each of the 51 tp5o pages 25 times, resulting in 51 sets of 25 data points. + * **To run it locally**, you'd need `tp5n.zip <#page-sets>`__. + * summarization: tp5 with limited pageset (48 pages as others have too much noise) + * subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 20; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l449>`__ + * suite: `geometric mean`_ of the 51 subtest results. + * description: + | Tests the time it takes Firefox to load the `tp5 web page test + set <#page-sets>`__. The web set was culled from the Alexa top 500 April + 8th, 2011 and consists of 100 pages in tp5n and 51 in tp5o. Some suites + use a subset of these, i.e. 48/51 tests to reduce noise + * check with the + owner of the test suite which uses the pageset to check if this + difference exists there. + + Here are the broad steps we use to create the test set: + + #. Take the Alexa top 500 sites list + #. Remove all sites with questionable or explicit content + #. Remove duplicate site (for ex. many Google search front pages) + #. Manually select to keep interesting pages (such as pages in different locales) + #. Select a more representative page from any site presenting a simple search/login/etc. page + #. Deal with Windows 255 char limit for cached pages + #. Limit test set to top 100 pages + + Note that the above steps did not eliminate all outside network access + so we had to take further action to scrub all the pages so that there + are 0 outside network accesses (this is done so that the tp test is as + deterministic measurement of our rendering/layout/paint process as + possible). + * **Example Data** + + .. code-block:: + + 0;163.com/www.163.com/index.html;1035;512;542;519;505;514;551;513;554;793;487;528;528;498;503;530;527;490;521;535;521;496;498;564;520 + 1;56.com/www.56.com/index.html;1081;583;580;577;597;580;623;558;572;592;598;580;564;583;596;600;579;580;566;573;566;581;571;600;586 + 2;aljazeera.net/aljazeera.net/portal.html;688;347;401;382;346;362;347;372;337;345;365;337;380;338;355;360;356;366;367;352;350;366;346;375;382 + 3;amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/507846.html;1392;878;901;852;886;867;877;864;862;877;866;888;3308;870;863;869;873;850;851;850;857;873;869;860;855 + 4;bbc.co.uk/www.bbc.co.uk/news/index.html;455;271;272;271;279;289;276;285;277;291;281;286;278;286;274;285;276;285;287;286;276;288;279;272;278 + 5;beatonna.livejournal.com/beatonna.livejournal.com/index.html;290;123;123;129;120;121;124;125;119;125;120;150;121;147;121;121;113;121;119;122;117;112;127;117;139 + 6;bild.de/www.bild.de/index.html;1314;937;946;931;922;918;920;937;934;930;947;928;936;933;933;928;930;941;951;946;947;938;925;939;938 + 7;cgi.ebay.com/cgi.ebay.com/ALL-NEW-KINDLE-3-eBOOK-WIRELESS-READING-DEVICE-W-WIFI-/130496077314@pt=LH_DefaultDomain_0&hash=item1e622c1e02.html;495;324;330;328;321;308;315;308;321;313;327;330;317;339;333;322;312;370;336;327;310;312;312;355;330 + 8;chemistry.about.com/chemistry.about.com/index.html;238;156;156;154;158;161;152;151;152;167;179;152;154;156;161;161;157;167;151;167;154;149;178;153;160 + 9;chinaz.com/chinaz.com/index.html;347;223;255;234;245;233;264;234;244;228;260;224;258;223;280;220;243;225;251;226;258;232;258;232;247 + 10;cnn.com/www.cnn.com/index.html;551;384;436;394;391;375;371;407;371;374;398;372;368;388;376;380;386;377;363;383;384;370;388;381;374 + 11;dailymail.co.uk/www.dailymail.co.uk/ushome/index.html;984;606;551;561;545;542;576;564;543;560;566;557;561;544;545;576;548;539;568;567;557;560;545;544;578 + 12;dailymotion.com/www.dailymotion.com/us.html;473;271;286;272;285;288;290;290;280;268;286;269;287;275;289;282;293;287;304;261;289;284;281;277;286 + 13;digg.com/digg.com/news/story/New_logo_for_Mozilla_Firefox_browser.html;410;321;304;303;322;300;319;321;320;306;323;313;312;305;312;338;317;338;301;325;297;302;309;305;300 + 14;ezinearticles.com/ezinearticles.com/index.html@Migraine-Ocular---The-Eye-Migraines&id=4684133.html;234;177;163;163;186;176;185;175;167;156;162;199;163;190;173;181;175;178;165;159;182;170;183;169;158 + 15;globo.com/www.globo.com/index.html;684;468;466;485;482;445;433;467;467;450;487;466;440;484;444;451;511;443;429;469;468;430;485;459;447 + 16;google.com/www.google.com/search@q=mozilla.html;150;100;102;101;97;104;99;116;107;100;98;137;102;102;99;106;98;112;100;102;105;104;107;96;100 + 17;goo.ne.jp/goo.ne.jp/index.html;328;125;132;132;143;121;122;120;132;145;166;139;144;125;128;152;128;145;130;132;154;126;142;133;139 + 18;guardian.co.uk/www.guardian.co.uk/index.html;462;311;296;322;309;305;303;288;301;308;301;304;319;309;295;305;294;308;304;322;310;314;302;303;292 + 19;homeway.com.cn/www.hexun.com/index.html;584;456;396;357;417;374;376;406;363;392;400;378;378;402;390;373;398;393;366;385;383;361;418;386;351 + 20;huffingtonpost.com/www.huffingtonpost.com/index.html;811;609;575;596;568;585;589;571;568;600;571;588;585;570;574;616;576;564;598;594;589;590;572;572;612 + 21;ifeng.com/ifeng.com/index.html;829;438;478;462;448;465;469;470;429;463;465;432;482;444;476;453;460;476;461;484;467;510;447;477;443 + 22;imdb.com/www.imdb.com/title/tt1099212/index.html;476;337;358;332;414;379;344;420;354;363;387;345;358;371;341;385;359;379;353;349;392;349;358;378;347 + 23;imgur.com/imgur.com/gallery/index.html;419;205;224;231;207;222;206;231;204;219;209;210;210;208;202;215;203;210;203;212;218;219;202;224;230 + 24;indiatimes.com/www.indiatimes.com/index.html;530;339;348;384;376;381;353;350;403;333;356;393;350;328;393;329;389;346;394;349;382;332;409;327;354 + 25;mail.ru/mail.ru/index.html;500;256;308;251;271;270;270;246;273;252;279;249;301;252;251;256;271;246;267;254;265;248;277;247;275 + 26;mashable.com/mashable.com/index.html;699;497;439;508;440;448;512;446;431;500;445;427;495;455;458;499;440;432;522;443;447;488;445;461;489 + 27;media.photobucket.com/media.photobucket.com/image/funny%20gif/findstuff22/Best%20Images/Funny/funny-gif1.jpg@o=1.html;294;203;195;189;213;186;195;186;204;188;190;220;204;202;195;204;192;204;191;187;204;199;191;192;211 + 28;myspace.com/www.myspace.com/albumart.html;595;446;455;420;433;425;416;429;452;411;435;439;389;434;418;402;422;426;396;438;423;391;434;438;395 + 29;naver.com/www.naver.com/index.html;626;368;338;386;342;377;371;352;379;348;362;357;359;354;386;338;394;330;326;372;345;392;336;336;368 + 30;noimpactman.typepad.com/noimpactman.typepad.com/index.html;431;333;288;292;285;313;277;289;282;292;276;293;270;294;289;281;275;302;285;290;280;285;278;284;283 + 31;page.renren.com/page.renren.com/index.html;373;232;228;228;213;227;224;227;226;216;234;226;230;212;213;221;224;230;212;218;217;221;212;222;230 + 32;people.com.cn/people.com.cn/index.html;579;318;305;339;307;341;325;326;307;309;315;314;318;317;321;309;307;299;312;313;305;326;318;384;310 + 33;rakuten.co.jp/www.rakuten.co.jp/index.html;717;385;371;388;381;348;394;358;396;368;343;386;348;388;393;388;360;339;398;357;392;378;395;386;367 + 34;reddit.com/www.reddit.com/index.html;340;254;248;255;241;241;248;275;251;250;250;252;243;274;240;269;254;249;242;257;271;253;243;278;252 + 35;reuters.com/www.reuters.com/index.html;513;404;355;358;379;343;354;385;379;354;418;363;342;412;355;351;402;375;354;400;362;355;380;373;336 + 36;slideshare.net/www.slideshare.net/jameswillamor/lolcats-in-popular-culture-a-historical-perspective.html;397;279;270;283;285;283;291;286;289;284;275;281;288;284;280;279;290;301;290;270;292;282;289;267;278 + 37;sohu.com/www.sohu.com/index.html;727;414;479;414;431;485;418;440;488;431;432;464;442;407;488;435;416;465;445;414;480;416;403;463;429 + 38;spiegel.de/www.spiegel.de/index.html;543;430;391;380;440;387;375;430;380;397;415;383;434;420;384;399;421;392;384;418;388;380;427;403;392 + 39;stackoverflow.com/stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered.html;503;377;356;438;370;388;409;367;366;407;375;363;393;360;363;396;376;391;426;363;378;408;400;359;408 + 40;store.apple.com/store.apple.com/us@mco=Nzc1MjMwNA.html;488;327;344;343;333;329;328;348;361;342;362;332;389;333;382;331;382;343;405;343;326;325;329;323;340 + 41;thepiratebay.org/thepiratebay.org/top/201.html;412;274;317;260;256;269;266;261;258;289;245;284;256;277;251;302;276;307;268;268;247;285;260;271;257 + 42;tudou.com/www.tudou.com/index.html;522;304;281;283;287;285;288;307;279;288;282;303;292;288;290;287;311;271;279;274;294;272;290;269;290 + 43;uol.com.br/www.uol.com.br/index.html;668;387;450;411;395;452;386;431;452;394;385;436;413;414;440;401;412;439;408;430;426;415;382;433;387 + 44;w3.org/www.w3.org/standards/webdesign/htmlcss.html;225;143;129;151;181;141;147;137;159;179;142;153;136;139;191;140;151;143;141;181;140;154;142;143;183 + 45;wsj.com/online.wsj.com/home-page.html;634;466;512;463;467;507;461;432;492;494;491;507;466;477;495;455;451;495;461;463;494;468;444;497;442 + 46;xinhuanet.com/xinhuanet.com/index.html;991;663;727;659;647;639;644;656;666;658;670;648;676;653;652;654;641;636;664;668;655;657;646;674;633 + 47;xunlei.com/xunlei.com/index.html;802;625;624;636;641;652;659;642;623;635;628;606;667;688;683;694;672;640;628;620;653;626;633;654;643 + 48;yelp.com/www.yelp.com/biz/alexanders-steakhouse-cupertino.html;752;475;502;472;477;512;489;478;501;472;454;517;487;474;521;467;450;513;491;464;536;507;455;511;481 + 49;youku.com/www.youku.com/index.html;844;448;498;441;417;497;478;439;467;436;447;465;438;461;466;446;452;496;457;446;486;449;467;499;442 + 50;youtube.com/www.youtube.com/music.html;443;338;253;289;238;296;254;290;242;302;237;290;253;305;253;293;251;311;244;293;255;291;246;316;249 + + * cycles: 1 + * gecko_profile_entries: 4000000 + * gecko_profile_interval: 2 + * linux_counters: ['XRes'] + * mac_counters: [] + * mainthread: False + * multidomain: True + * responsiveness: True + * timeout: 1800 + * tpcycles: 1 + * tpmanifest: ${talos}/fis/tp5n/tp5o.manifest + * tpmozafterpaint: True + * tppagecycles: 25 + * tptimeout: 5000 + * unit: ms + * w7_counters: ['% Processor Time'] + * win_counters: ['% Processor Time'] + * Command + + .. code-block:: + + ./mach talos-test -a tp5o + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-tp5o-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-tp5o-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-tp5o** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tp5o_scroll + :class-container: anchor-id-tp5o_scroll + + * contact: :botond, :tnikkel, :hiro, and layout team + * source: `tp5n.zip <#page-sets>`__ + * type: `Page load`_ + * data: we load each of the 51 tp5o pages 12 times, resulting in 51 sets of 12 data points. + * **To run it locally**, you'd need `tp5n.zip <#page-sets>`__. + * summarization: Measures average frames interval while scrolling the pages of the tp5o set + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 11; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l470>`__ + * suite: `geometric mean`_ of the 51 subtest results. + * description: + | This test is identical to tscrollx, but it scrolls the 50 pages of the + tp5o set (rather than 6 synthetic pages which tscrollx scrolls). There + are two variants for each test page. The "regular" variant waits 500ms + after the page load event fires, then iterates 100 scroll steps of 10px + each (or until the bottom of the page is reached + * whichever comes + first), then reports the average frame interval. The "CSSOM" variant is + similar, but uses APZ's smooth scrolling mechanism to do compositor + scrolling instead of main-thread scrolling. So it just requests the + final scroll destination and the compositor handles the scrolling and + reports frame intervals. + * **Example Data** + + .. code-block:: + + 0;163.com/www.163.com/index.html;9.73;8.61;7.37;8.17;7.58;7.29;6.88;7.45;6.91;6.61;8.47;7.12 + 1;56.com/www.56.com/index.html;10.85;10.24;10.75;10.30;10.23;10.10;10.31;10.06;11.10;10.06;9.56;10.30 + 2;aljazeera.net/aljazeera.net/portal.html;9.23;7.15;7.50;7.26;7.73;7.05;7.14;7.66;7.23;7.93;7.26;7.18 + 3;amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/507846.html;7.14;6.87;7.18;6.31;6.93;6.71;6.37;7.00;6.59;5.37;7.31;6.13 + 4;bbc.co.uk/www.bbc.co.uk/news/index.html;7.39;6.33;6.22;7.66;6.67;7.77;6.91;7.74;7.08;6.36;6.03;7.12 + 5;beatonna.livejournal.com/beatonna.livejournal.com/index.html;5.79;5.79;5.68;5.46;5.55;5.48;5.69;5.83;5.88;5.97;5.93;5.88 + 6;bild.de/www.bild.de/index.html;8.65;7.63;7.17;6.36;7.44;7.68;8.63;6.71;8.34;7.15;7.82;7.70 + 7;cgi.ebay.com/cgi.ebay.com/ALL-NEW-KINDLE-3-eBOOK-WIRELESS-READING-DEVICE-W-WIFI-/130496077314@pt=LH_DefaultDomain_0&hash=item1e622c1e02.html;7.12;6.81;7.22;6.98;7.05;5.68;7.15;6.54;7.31;7.18;7.82;7.77 + 8;chemistry.about.com/chemistry.about.com/index.html;6.76;6.17;6.41;6.88;5.67;5.47;6.83;6.28;6.16;6.81;6.21;6.75 + 9;chinaz.com/chinaz.com/index.html;10.72;7.99;7.33;7.10;7.85;8.62;8.39;6.72;6.26;6.65;8.14;7.78 + 10;cnn.com/www.cnn.com/index.html;7.73;6.80;6.08;8.27;9.24;7.81;7.69;7.05;8.17;7.70;7.90;6.81 + 11;dailymail.co.uk/www.dailymail.co.uk/ushome/index.html;6.37;8.28;7.19;8.00;8.09;7.43;6.90;7.24;7.77;7.29;7.38;6.14 + 12;dailymotion.com/www.dailymotion.com/us.html;9.53;9.80;9.29;9.03;9.10;8.64;8.62;8.71;8.77;9.81;9.64;8.96 + 13;digg.com/digg.com/news/story/New_logo_for_Mozilla_Firefox_browser.html;7.72;7.06;7.60;5.67;6.85;7.32;7.80;5.98;8.27;6.68;7.52;8.39 + 14;ezinearticles.com/ezinearticles.com/index.html@Migraine-Ocular---The-Eye-Migraines&id=4684133.html;7.14;7.11;8.09;7.17;6.87;7.12;7.65;7.74;7.26;7.36;6.91;6.95 + 15;globo.com/www.globo.com/index.html;6.71;7.91;5.83;7.34;7.75;8.00;7.73;7.85;7.03;6.42;8.43;8.11 + 16;google.com/www.google.com/search@q=mozilla.html;6.49;6.23;7.96;6.39;7.23;8.19;7.35;7.39;6.94;7.24;7.55;7.62 + 17;goo.ne.jp/goo.ne.jp/index.html;8.56;7.18;7.15;7.03;6.85;7.62;7.66;6.99;7.84;7.51;7.23;7.18 + 18;guardian.co.uk/www.guardian.co.uk/index.html;7.32;7.62;8.18;7.62;7.83;8.08;7.60;8.17;8.47;7.54;7.92;8.09 + 19;homeway.com.cn/www.hexun.com/index.html;10.18;8.75;8.83;8.64;8.98;8.07;7.76;9.29;8.05;7.55;8.91;7.78 + 20;huffingtonpost.com/www.huffingtonpost.com/index.html;8.38;7.17;7.03;6.83;6.49;6.47;6.69;7.08;6.81;7.29;7.13;7.70 + 21;ifeng.com/ifeng.com/index.html;12.45;8.65;8.75;7.56;8.26;7.71;8.04;7.45;7.83;7.14;8.38;7.68 + 22;imdb.com/www.imdb.com/title/tt1099212/index.html;8.53;5.65;6.94;7.18;6.10;7.57;6.26;8.34;8.16;7.29;7.62;8.51 + 23;imgur.com/imgur.com/gallery/index.html;8.10;7.20;7.50;7.88;7.27;6.97;8.13;7.14;7.59;7.39;8.01;8.82 + 24;indiatimes.com/www.indiatimes.com/index.html;8.00;6.74;7.37;8.52;7.03;8.45;7.08;8.47;9.26;7.89;7.17;6.74 + 25;mail.ru/mail.ru/index.html;7.64;9.50;9.47;7.03;6.45;6.24;8.03;6.72;7.18;6.39;6.25;6.25 + 26;mashable.com/mashable.com/index.html;7.97;8.03;6.10;7.80;7.91;7.26;7.49;7.45;7.60;7.08;7.63;7.36 + 27;media.photobucket.com/media.photobucket.com/image/funny%20gif/findstuff22/Best%20Images/Funny/funny-gif1.jpg@o=1.html;290.00;195.00;217.00;199.00;204.00;196.00;198.00;206.00;209.00;208.00;192.00;196.00 + 28;myspace.com/www.myspace.com/albumart.html;14.40;13.45;13.29;13.62;13.42;14.15;13.86;14.34;14.69;14.10;13.82;14.13 + 29;naver.com/www.naver.com/index.html;9.15;8.31;9.40;9.89;7.29;8.43;8.87;8.77;8.96;8.24;8.16;8.21 + 30;noimpactman.typepad.com/noimpactman.typepad.com/index.html;7.27;7.14;7.70;7.86;7.43;6.95;7.30;7.58;7.51;7.95;7.43;7.05 + 31;page.renren.com/page.renren.com/index.html;7.94;8.13;6.76;7.77;6.93;6.60;7.62;7.61;6.88;7.56;7.55;7.48 + 32;people.com.cn/people.com.cn/index.html;11.92;9.22;8.49;8.55;8.34;8.49;6.91;9.92;8.69;8.63;7.69;9.34 + 33;rakuten.co.jp/www.rakuten.co.jp/index.html;11.10;7.13;8.68;7.85;8.37;7.91;6.74;8.27;8.55;8.93;7.15;9.02 + 34;reddit.com/www.reddit.com/index.html;6.38;7.95;6.84;7.04;6.96;7.15;8.05;7.71;8.13;7.13;6.60;7.53 + 35;reuters.com/www.reuters.com/index.html;7.51;7.25;6.60;6.98;7.41;6.45;7.61;7.46;6.11;7.15;7.05;6.94 + 36;slideshare.net/www.slideshare.net/jameswillamor/lolcats-in-popular-culture-a-historical-perspective.html;7.20;6.32;6.80;6.87;6.29;6.45;7.18;6.92;6.57;7.41;7.08;6.51 + 37;sohu.com/www.sohu.com/index.html;11.72;9.64;8.85;7.12;7.96;9.14;7.76;8.19;7.14;7.68;8.08;7.24 + 38;spiegel.de/www.spiegel.de/index.html;7.24;7.30;6.64;7.01;6.74;6.70;6.36;6.84;7.86;7.08;7.12;7.40 + 39;stackoverflow.com/stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered.html;7.39;5.88;7.22;6.51;7.12;6.51;6.46;6.53;6.63;6.54;6.48;6.80 + 40;store.apple.com/store.apple.com/us@mco=Nzc1MjMwNA.html;6.23;7.17;7.39;8.98;7.99;8.03;9.12;8.37;8.56;7.61;8.06;7.55 + 41;thepiratebay.org/thepiratebay.org/top/201.html;9.08;8.93;8.09;7.49;7.30;7.80;7.54;7.65;7.91;7.53;8.37;8.04 + 42;tudou.com/www.tudou.com/index.html;10.06;9.38;8.68;7.37;8.57;9.11;8.20;7.91;8.78;9.64;8.11;8.47 + 43;uol.com.br/www.uol.com.br/index.html;9.04;9.49;9.48;9.31;8.68;8.41;9.16;8.91;9.49;8.37;9.77;8.73 + 44;w3.org/www.w3.org/standards/webdesign/htmlcss.html;6.62;5.98;6.87;6.47;7.22;6.05;6.42;6.50;7.47;7.18;5.82;7.11 + 45;wsj.com/online.wsj.com/home-page.html;7.49;8.57;6.84;8.12;7.60;7.24;8.16;8.22;6.81;8.28;8.11;8.58 + 46;xinhuanet.com/xinhuanet.com/index.html;13.66;9.21;10.09;9.56;8.99;10.29;10.24;8.91;11.23;10.82;9.64;10.11 + 47;xunlei.com/xunlei.com/index.html;8.99;8.16;8.82;8.37;7.01;8.48;7.98;8.69;8.10;8.10;7.10;6.41 + 48;yelp.com/www.yelp.com/biz/alexanders-steakhouse-cupertino.html;8.18;7.45;7.01;8.14;7.12;7.82;8.24;7.13;7.00;6.41;6.85;7.31 + 49;youku.com/www.youku.com/index.html;12.21;10.29;10.37;10.34;8.40;9.82;9.23;9.91;9.64;9.91;8.90;10.23 + 50;youtube.com/www.youtube.com/music.html;9.90;9.06;9.29;9.17;8.85;8.77;9.83;9.21;9.29;10.09;9.69;8.64 + + * **Possible regression causes**: Some examples of things that cause regressions in this test are + * Increased displayport size (which causes a larger display list to be built) + * Slowdown in the building of display list + * Slowdown in rasterization of content + * Slowdown in composite times + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 2 + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': True, 'apz.paint_skipping.enabled': False, 'layout.css.scroll-behavior.spring-constant': "'10'", 'toolkit.framesRecording.bufferSize': 10000} + * tpcycles: 1 + * tpmanifest: ${talos}/tests/tp5n/tp5o.manifest + * tpmozafterpaint: False + * tppagecycles: 12 + * tpscrolltest: True + * unit: 1/FPS + * Command + + .. code-block:: + + ./mach talos-test -a tp5o_scroll + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g1-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g1-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g1** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tp5o_webext + :class-container: anchor-id-tp5o_webext + + * contact: :mixedpuppy and webextension team + * preferences: {'xpinstall.signatures.required': False} + * webextensions: ${talos}/webextensions/dummy/dummy.xpi + * Command + + .. code-block:: + + ./mach talos-test -a tp5o_webext + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g5-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g5-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tresize + :class-container: anchor-id-tresize + + * contact: :gcp and platform integration + * source: `tresize-test.html <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tresize/addon/content/tresize-test.html>`__ + * type: Startup_ + * measuring: Time to do XUL resize, in ms (lower is better). + * data: we run the tresize test page 20 times, resulting in 1 set of 20 data points. + * summarization: + * subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 15 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l205>`__ + * suite: same as subtest result + * description: + | A purer form of paint measurement than tpaint. This test opens a single + window positioned at 10,10 and sized to 300,300, then resizes the window + outward \|max\| times measuring the amount of time it takes to repaint + each resize. Dumps the resulting dataset and average to stdout or + logfile. + + In `bug + 1102479 <https://bugzilla.mozilla.org/show_bug.cgi?id=1102479>`__ + tresize was rewritten to work in e10s mode which involved a full rewrite + of the test. + * **Example Data** + + .. code-block:: + + [23.2565333333333, 23.763383333333362, 22.58369999999999, 22.802766666666653, 22.304050000000025, 23.010383333333326, 22.865466666666677, 24.233716666666705, 24.110983333333365, 22.21390000000004, 23.910333333333316, 23.409816666666647, 19.873049999999992, 21.103966666666686, 20.389749999999978, 20.777349999999984, 20.326283333333365, 22.341616666666667, 20.29813333333336, 20.769600000000104] + + * **Possible regression causes** + * slowdown in the paint pipeline + * resizes also trigger a rendering flush so bugs in the flushing code can manifest as regressions + * introduction of more spurious MozAfterPaint events + * see `bug 1471961 <https://bugzilla.mozilla.org/show_bug.cgi?id=1471961>`__ + * extensions: ['${talos}/pageloader', '${talos}/tests/tresize/addon'] + * gecko_profile_entries: 1000000 + * gecko_profile_interval: 2 + * timeout: 900 + * tpmanifest: ${talos}/tests/tresize/tresize.manifest + * tpmozafterpaint: True + * tppagecycles: 20 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tresize + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-chrome-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-chrome-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-chrome** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: ts_paint + :class-container: anchor-id-ts_paint + + * contact: :mconley, Firefox Desktop Front-end team, + * source: `tspaint_test.html <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tspaint_test.html>`__ + * Perfomatic: "Ts, Paint" + * type: Startup_ + * data: 20 times we start the browser and time how long it takes to + paint the startup test page, resulting in 1 set of 20 data points. + * summarization: + * subtest: identical to suite + * suite: `ignore first`_ data point, then take the `median`_ of the remaining 19 data points; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l135>`__ + * description: + | Starts the browser to display tspaint_test.html with the start time in + the url, waits for `MozAfterPaint and onLoad <#paint>`__ to fire, then + records the end time and calculates the time to startup. + * **Example Data** + + .. code-block:: + + [1666.0, 1195.0, 1139.0, 1198.0, 1248.0, 1224.0, 1213.0, 1194.0, 1229.0, 1196.0, 1191.0, 1230.0, 1247.0, 1169.0, 1217.0, 1184.0, 1196.0, 1192.0, 1224.0, 1192.0] + + * **Possible regression causes** + * (and/or maybe tpaint?) will regress if a new element is added to the + browser window (e.g. browser.xul) and it's frame gets created. Fix + this by ensuring it's display:none by default. + * cycles: 20 + * gecko_profile_entries: 10000000 + * gecko_profile_startup: True + * mainthread: False + * responsiveness: False + * timeout: 150 + * tpmozafterpaint: True + * unit: ms + * url: startup_test/tspaint_test.html + * win7_counters: [] + * xperf_counters: [] + * Command + + .. code-block:: + + ./mach talos-test -a ts_paint + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: ts_paint_heavy + :class-container: anchor-id-ts_paint_heavy + + * `ts_paint <#ts_paint>`_ test run against a heavy user profile. + * contact: :mconley, Firefox Desktop Front-end team, + * profile: simple + * Command + + .. code-block:: + + ./mach talos-test -a ts_paint_heavy + + +.. dropdown:: ts_paint_webext + :class-container: anchor-id-ts_paint_webext + + * contact: :mconley, Firefox Desktop Front-end team, + * preferences: {'xpinstall.signatures.required': False} + * webextensions: ${talos}/webextensions/dummy/dummy.xpi + * Command + + .. code-block:: + + ./mach talos-test -a ts_paint_webext + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g5-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-g5-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-g5** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tscrollx + :class-container: anchor-id-tscrollx + + * contact: :jrmuizel and gfx + * source: `scroll.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/scroll>`__ + * type: `Page load`_ + * measuring: Scroll performance + * reporting: Average frame interval (1 ÷ fps). Lower is better. + * data: we load 6 pages 25 times each, collecting 6 sets of 25 data points + * summarization: `Replacing tscroll,tsvg with tscrollx,tsvgx <https://groups.google.com/d/topic/mozilla.dev.platform/RICw5SJhNMo/discussion>`__ + * subtest: `ignore first`_ data point, then take the `median`_ of the remaining 24; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623>`__ + * suite: `geometric mean`_ of the 6 subtest results. + * description: + | This test scrolls several pages where each represent a different known + "hard" case to scroll (\* needinfo), and measures the average frames + interval (1/FPS) on each. The ASAP test (tscrollx) iterates in unlimited + frame-rate mode thus reflecting the maximum scroll throughput per page. + To turn on ASAP mode, we set these preferences: + + ``preferences = {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1}`` + + See also `tp5o_scroll <#tp5o_scroll>`_ which has relevant information for this test. + * **Example Data** + + .. code-block:: + + 0;tiled.html;5.41;5.57;5.34;5.64;5.53;5.48;5.44;5.49;5.50;5.50;5.49;5.66;5.50;5.37;5.57;5.54;5.46;5.31;5.41;5.57;5.50;5.52;5.71;5.31;5.44 + fixed.html;10.404609053497941;10.47;10.66;10.45;10.73;10.79;10.64;10.64;10.82;10.43;10.92;10.47;10.47;10.64;10.74;10.67;10.40;10.83;10.77;10.54;10.38;10.70;10.44;10.38;10.56 + downscale.html;5.493209876543211;5.27;5.50;5.50;5.51;5.46;5.58;5.58;5.51;5.49;5.49;5.47;9.09;5.56;5.61;5.50;5.47;5.59;5.47;5.49;5.60;5.61;5.58;5.40;5.43 + downscale.html;10.676522633744854;10.82;10.79;10.41;10.75;10.91;10.52;10.61;10.50;10.55;10.80;10.17;10.68;10.41;10.42;10.41;10.58;10.28;10.56;10.66;10.68;10.47;10.60;10.61;10.26 + 4;iframe.svg;13.82;14.87;14.78;14.35;14.73;14.50;14.15;14.46;14.80;14.48;15.10;14.93;14.77;14.52;14.08;15.01;14.58;14.52;15.23;14.35;14.72;14.28;14.30;14.27;14.96 + 5;reader.htm;10.72;10.62;10.23;10.48;10.42;10.64;10.40;10.40;10.14;10.60;10.51;10.36;10.57;10.41;10.52;10.75;10.19;10.72;10.44;9.75;10.49;10.07;10.54;10.46;10.44 + + * gecko_profile_entries: 1000000 + * gecko_profile_interval: 1 + * pine: False + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': True, 'apz.paint_skipping.enabled': False, 'layout.css.scroll-behavior.spring-constant': "'10'", 'toolkit.framesRecording.bufferSize': 10000} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/scroll/scroll.manifest + * tpmozafterpaint: False + * tppagecycles: 25 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tscrollx + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tsvg_static + :class-container: anchor-id-tsvg_static + + * contact: :jwatt, :dholbert + * source: `svg_static <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svg_static/>`__ + * type: `Page load`_ + * data: we load the 5 svg pages 25 times, resulting in 5 sets of 25 data points + * summarization: An svg-only number that measures SVG rendering performance of some complex (but static) SVG content. + * subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 20; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623>`__ + * suite: `geometric mean`_ of the 5 subtest results. + * **Example Data** + + .. code-block:: + + 0;gearflowers.svg;262;184;184;198;197;187;181;186;177;192;196;194;194;186;195;190;237;193;188;182;188;196;191;194;184 + 1;composite-scale.svg;69;52;48;49;57;51;52;87;52;49;49;51;58;53;64;57;49;65;67;58;53;59;56;68;50 + 2;composite-scale-opacity.svg;72;53;91;54;51;58;60;46;51;57;59;58;66;70;57;61;47;51;76;65;52;65;64;64;63 + 3;composite-scale-rotate.svg;70;76;89;62;62;78;57;77;79;82;74;56;61;79;73;64;75;74;81;82;76;58;77;61;62 + 4;composite-scale-rotate-opacity.svg;91;60;67;84;62;66;78;69;65;68;62;73;68;63;64;71;79;77;63;80;85;65;82;76;81 + + * gecko_profile_entries: 10000000 + * gecko_profile_interval: 1 + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/svg_static/svg_static.manifest + * tpmozafterpaint: True + * tppagecycles: 25 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tsvg_static + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tsvgm + :class-container: anchor-id-tsvgm + + * An svg-only number that measures SVG rendering performance for dynamic content only. + * contact: :jwatt, :dholbert + * add test details + * gecko_profile_entries: 1000000 + * gecko_profile_interval: 10 + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/svgx/svgm.manifest + * tpmozafterpaint: False + * tppagecycles: 7 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tsvgm + + +.. dropdown:: tsvgr_opacity + :class-container: anchor-id-tsvgr_opacity + + * contact: :jwatt, :dholbert + * source: `svg_opacity.manifest <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svg_opacity/svg_opacity.manifest>`__ + * type: `Page load`_ + * data: we load the 2 svg opacity pages 25 times, resulting in 2 sets of 25 data points + * summarization: `Row Major <#row-major-vs-column-major>`__ and 25 cycles/page. + * subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 20; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623>`__ + * suite: `geometric mean`_ of the 2 subtest results. + * description: + | Renders many semi-transparent, partially overlapping SVG rectangles, and + measures time to completion of this rendering. + + Note that this test also tends to reflect changes in network efficiency + and navigation bar rendering issues. + * Most of the page load tests measure from before the location is + changed, until onload + mozafterpaint, therefore any changes in + chrome performance from the location change, or network performance + (the pages load from a local web server) would affect page load + times. SVG opacity is rather quick by itself, so any such + chrome/network/etc performance changes would affect this test more + than other page load tests (relatively, in percentages). + * **Example Data** + + .. code-block:: + + 0;big-optimizable-group-opacity-2500.svg;170;171;205;249;249;244;192;252;192;431;182;250;189;249;151;168;209;194;247;250;193;250;255;247;247 + 1;small-group-opacity-2500.svg;585;436;387;441;512;438;440;380;443;391;450;386;459;383;445;388;450;436;485;443;383;438;528;444;441 + + * gecko_profile_entries: 10000000 + * gecko_profile_interval: 1 + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/svg_opacity/svg_opacity.manifest + * tpmozafterpaint: True + * tppagecycles: 25 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tsvgr_opacity + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: tsvgx + :class-container: anchor-id-tsvgx + + * contact: :jwatt, :dholbert + * source: `svgx <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svgx>`__ + * type: `Page load`_ + * data: we load the 7 svg pages 25 times, resulting in 7 sets of 25 data points + * summarization: `Replacing tscroll,tsvg with tscrollx,tsvgx <https://groups.google.com/d/topic/mozilla.dev.platform/RICw5SJhNMo/discussion>`__ + * subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 20; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623>`__ + * suite: `geometric mean`_ of the 7 subtest results. + * description: + | An svg-only number that measures SVG rendering performance, with + animations or iterations of rendering. This is an ASAP test --i.e. it + iterates in unlimited frame-rate mode thus reflecting the maximum + rendering throughput of each test. The reported value is the overall + duration the sequence/animation took to complete. To turn on ASAP mode, + we set these preferences: + + ``preferences = {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1}`` + * **Example Data** + + .. code-block:: + + 0;hixie-001.xml;562;555;508;521;522;520;499;510;492;514;502;504;500;521;510;506;511;505;495;517;520;512;503;504;502 + 1;hixie-002.xml;510;613;536;530;536;522;498;505;500;504;498;529;498;509;493;512;501;506;504;499;496;505;508;511;503 + 2;hixie-003.xml;339;248;242;261;250;241;240;248;258;244;235;240;247;248;239;247;241;245;242;245;251;239;241;240;237 + 3;hixie-004.xml;709;540;538;536;540;536;552;539;535;535;543;533;536;535;545;537;537;537;537;539;538;535;536;538;536 + 4;hixie-005.xml;3096;3086;3003;3809;3213;3323;3143;3313;3192;3203;3225;3048;3069;3101;3189;3251;3172;3122;3266;3183;3159;3076;3014;3237;3100 + 5;hixie-006.xml;5586;5668;5565;5666;5668;5728;5886;5534;5484;5607;5678;5577;5745;5753;5532;5585;5506;5516;5648;5778;5894;5994;5794;5852;5810 + 6;hixie-007.xml;1823;1743;1739;1743;1744;1787;1802;1788;1782;1766;1787;1750;1748;1788;1766;1779;1767;1794;1758;1768;1781;1773;1765;1798;1805 + + * **Possible regression causes** + * Did you change the dimensions of the content area? Even a little? The + tsvgx test seems to be sensitive to changes like this. See `bug + 1375479 <https://bugzilla.mozilla.org/show_bug.cgi?id=1375479>`__, + for example. Usually, these sorts of "regressions" aren't real + regressions + * they just mean that we need to re-baseline our + expectations from the test. + * gecko_profile_entries: 1000000 + * gecko_profile_interval: 10 + * preferences: {'layout.frame_rate': 0, 'docshell.event_starvation_delay_hint': 1, 'dom.send_after_paint_to_content': False} + * timeout: 600 + * tpchrome: False + * tpcycles: 1 + * tpmanifest: ${talos}/tests/svgx/svgx.manifest + * tpmozafterpaint: False + * tppagecycles: 25 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a tsvgx + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-svgr-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-svgr** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: twinopen + :class-container: anchor-id-twinopen + + * contact: :gcp and platform integration + * source: `twinopen <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/twinopen>`__ + * type: Startup_ + * data: we open a new browser window 20 times, resulting in 1 set of 20 data points. + * summarization: Time from calling OpenBrowserWindow until the chrome of the new window has `painted <https://developer.mozilla.org/en-US/docs/Web/Events#MozAfterPaint>`__. + * subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 15; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190>`__ + * suite: identical to subtest + * description: + | Tests the amount of time it takes the open a new window from a currently + open browser. This test does not include startup time. Multiple test + windows are opened in succession, results reported are the average + amount of time required to create and display a window in the running + instance of the browser. (Measures ctrl-n performance.) + * **Example Data** + + .. code-block:: + + [209.219, 222.180, 225.299, 225.970, 228.090, 229.450, 230.625, 236.315, 239.804, 242.795, 244.5, 244.770, 250.524, 251.785, 253.074, 255.349, 264.729, 266.014, 269.399, 326.190] + + * extensions: ['${talos}/pageloader', '${talos}/tests/twinopen'] + * gecko_profile_entries: 2000000 + * gecko_profile_interval: 1 + * preferences: {'browser.startup.homepage': 'about:blank'} + * timeout: 300 + * tpmanifest: ${talos}/tests/twinopen/twinopen.manifest + * tpmozafterpaint: True + * tppagecycles: 20 + * unit: ms + * Command + + .. code-block:: + + ./mach talos-test -a twinopen + + * **Test Task**: + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + * - **talos-other-profiling** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-other** + - ✅ + - ✅ + - ❌ + - ✅ + + + +.. dropdown:: v8_7 + :class-container: anchor-id-v8_7 + + * description: + | This is the V8 (version 7) javascript benchmark taken verbatim and slightly modified + to fit into our pageloader extension and talos harness. The previous version of this + test is V8 version 5 which was run on selective branches and operating systems. + * contact: No longer being maintained by any team/individual + * gecko_profile_entries: 1000000 + * gecko_profile_interval: 1 + * lower_is_better: False + * preferences: {'dom.send_after_paint_to_content': False} + * resolution: 20 + * tpcycles: 1 + * tpmanifest: ${talos}/tests/v8_7/v8.manifest + * tpmozafterpaint: False + * unit: score + * Command + + .. code-block:: + + ./mach talos-test -a v8_7 + + + + +Extra Talos Tests +***************** + +.. contents:: + :depth: 1 + :local: + +about_newtab_with_snippets +========================== + +.. note:: + + add test details + +File IO +------- + +File IO is tested using the tp5 test set in the `xperf`_ +test. + +Possible regression causes +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- **nonmain_startup_fileio opt (with or without e10s) windows7-32** – + `bug + 1274018 <https://bugzilla.mozilla.org/show_bug.cgi?id=1274018>`__ + This test seems to consistently report a higher result for + mozilla-central compared to Try even for an identical revision due to + extension signing checks. In other words, if you are comparing Try + and Mozilla-Central you may see a false-positive regression on + perfherder. Graphs: + `non-e10s <https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-central,e5f5eaa174ef22fdd6b6e150e8c450aa827c2ff6,1,1%5D&series=%5Btry,e5f5eaa174ef22fdd6b6e150e8c450aa827c2ff6,1,1%5D>`__ + `e10s <https://treeherder.mozilla.org/perf.html#/graphs?series=%5B%22mozilla-central%22,%222f3af3833d55ff371ecf01c41aeee1939ef3a782%22,1,1%5D&series=%5B%22try%22,%222f3af3833d55ff371ecf01c41aeee1939ef3a782%22,1,1%5D&timerange=604800>`__ + +Xres (X Resource Monitoring) +---------------------------- + +A memory metric tracked during tp5 test runs. This metric is sampled +every 20 seconds. This metric is collected on linux only. + +`xres man page <https://linux.die.net/man/3/xres>`__. + +% CPU +----- + +Cpu usage tracked during tp5 test runs. This metric is sampled every 20 +seconds. This metric is collected on windows only. + +Responsiveness +-------------- + +contact: :jimm, :overholt + +Measures the delay for the event loop to process a `tracer +event <https://wiki.mozilla.org/Performance/Snappy#Current_Infrastructure>`__. +For more details, see `bug +631571 <https://bugzilla.mozilla.org/show_bug.cgi?id=631571>`__. + +The score on this benchmark is proportional to the sum of squares of all +event delays that exceed a 20ms threshold. Lower is better. + +We collect 8000+ data points from the browser during the test and apply +`this +formula <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/output.py#l95>`__ +to the results: + +.. code-block:: python + + return sum([float(x)*float(x) / 1000000.0 for x in val_list]) + +tpaint +====== + +.. warning:: + + This test no longer exists + +- contact: :davidb +- source: + `tpaint-window.html <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tpaint.html>`__ +- type: Startup_ +- data: we load the tpaint test window 20 times, resulting in 1 set of + 20 data points. +- summarization: + + - subtest: `ignore first`_ **5** data points, then take the `median`_ of the remaining 15; `source: + test.py <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190>`__ + - suite: identical to subtest + ++-----------------+---------------------------------------------------+ +| Talos test name | Description | ++-----------------+---------------------------------------------------+ +| tpaint | twinopen but measuring the time after we receive | +| | the `MozAfterPaint and OnLoad event <#paint>`__. | ++-----------------+---------------------------------------------------+ + +Tests the amount of time it takes the open a new window. This test does +not include startup time. Multiple test windows are opened in +succession, results reported are the average amount of time required to +create and display a window in the running instance of the browser. +(Measures ctrl-n performance.) + +**Example Data** + +.. code-block:: none + + [209.219, 222.180, 225.299, 225.970, 228.090, 229.450, 230.625, 236.315, 239.804, 242.795, 244.5, 244.770, 250.524, 251.785, 253.074, 255.349, 264.729, 266.014, 269.399, 326.190] + +Possible regression causes +-------------------------- + +- None listed yet. If you fix a regression for this test and have some + tips to share, this is a good place for them. + +xperf +===== + +- contact: fx-perf@mozilla.com +- source: `xperf + instrumentation <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xtalos>`__ +- type: `Page load`_ (tp5n) / Startup_ +- measuring: IO counters from windows (currently, only startup IO is in + scope) +- reporting: Summary of read/write counters for disk, network (lower is + better) + +These tests only run on windows builds. See `this active-data +query <https://activedata.allizom.org/tools/query.html#query_id=zqlX+2Jn>`__ +for an updated set of platforms that xperf can be found on. If the query +is not found, use the following on the query page: + +.. code-block:: javascript + + { + "from":"task", + "groupby":["run.name","build.platform"], + "limit":2000, + "where":{"regex":{"run.name":".*xperf.*"}} + } + +Talos will turn orange for 'x' jobs on windows 7 if your changeset +accesses files which are not predefined in the +`allowlist <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xtalos/xperf_allowlist.json>`__ +during startup; specifically, before the +"`sessionstore-windows-restored <https://hg.mozilla.org/mozilla-central/file/0eebc33d8593/toolkit/components/startup/nsAppStartup.cpp#l631>`__" +Firefox event. If your job turns orange, you will see a list of files in +Treeherder (or in the log file) which have been accessed unexpectedly +(similar to this): + +.. code-block:: none + + TEST-UNEXPECTED-FAIL : xperf: File '{profile}\secmod.db' was accessed and we were not expecting it. DiskReadCount: 6, DiskWriteCount: 0, DiskReadBytes: 16904, DiskWriteBytes: 0 + TEST-UNEXPECTED-FAIL : xperf: File '{profile}\cert8.db' was accessed and we were not expecting it. DiskReadCount: 4, DiskWriteCount: 0, DiskReadBytes: 33288, DiskWriteBytes: 0 + TEST-UNEXPECTED-FAIL : xperf: File 'c:\$logfile' was accessed and we were not expecting it. DiskReadCount: 0, DiskWriteCount: 2, DiskReadBytes: 0, DiskWriteBytes: 32768 + TEST-UNEXPECTED-FAIL : xperf: File '{profile}\secmod.db' was accessed and we were not expecting it. DiskReadCount: 6, DiskWriteCount: 0, DiskReadBytes: 16904, DiskWriteBytes: 0 + TEST-UNEXPECTED-FAIL : xperf: File '{profile}\cert8.db' was accessed and we were not expecting it. DiskReadCount: 4, DiskWriteCount: 0, DiskReadBytes: 33288, DiskWriteBytes: 0 + TEST-UNEXPECTED-FAIL : xperf: File 'c:\$logfile' was accessed and we were not expecting it. DiskReadCount: 0, DiskWriteCount: 2, DiskReadBytes: 0, DiskWriteBytes: 32768 + +In the case that these files are expected to be accessed during startup +by your changeset, then we can add them to the +`allowlist <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos>`__. + +Xperf runs tp5 while collecting xperf metrics for disk IO and network +IO. The providers we listen for are: + +- `'PROC_THREAD', 'LOADER', 'HARD_FAULTS', 'FILENAME', 'FILE_IO', + 'FILE_IO_INIT' <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xperf.config#l10>`__ + +The values we collect during stackwalk are: + +- `'FileRead', 'FileWrite', + 'FileFlush' <https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xperf.config#l11>`__ + +Notes: + +- Currently some runs may `return all-zeros and skew the + results <https://bugzilla.mozilla.org/show_bug.cgi?id=1614805>`__ +- Additionally, these runs don't have dedicated hardware and have a + large variability. At least 30 runs are likely to be needed to get + stable statistics (xref `bug + 1616236 <https://bugzilla.mozilla.org/show_bug.cgi?id=1616236>`__) + +Build metrics +************* + +These are not part of the Talos code, but like Talos they are benchmarks +that record data using the graphserver and are analyzed by the same +scripts for regressions. + +Number of constructors (num_ctors) +================================== + +This test runs at build time and measures the number of static +initializers in the compiled code. Reducing this number is helpful for +`startup +optimizations <https://blog.mozilla.org/tglek/2010/05/27/startup-backward-constructors/>`__. + +- https://hg.mozilla.org/build/tools/file/348853aee492/buildfarm/utils/count_ctors.py + + - these are run for linux 32+64 opt and pgo builds. + +Platform microbenchmark +*********************** + +IsASCII and IsUTF8 gtest microbenchmarks +======================================== + +- contact: :hsivonen +- source: + `TestStrings.cpp <https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp>`__ +- type: Microbench_ +- reporting: intervals in ms (lower is better) +- data: each test is run and measured 5 times +- summarization: take the `median`_ of the 5 data points; `source: + MozGTestBench.cpp <https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46>`__ + +Test whose name starts with PerfIsASCII test the performance of the +XPCOM string IsASCII function with ASCII inputs if different lengths. + +Test whose name starts with PerfIsUTF8 test the performance of the XPCOM +string IsUTF8 function with ASCII inputs if different lengths. + +Possible regression causes +-------------------------- + +- The --enable-rust-simd accidentally getting turned off in automation. +- Changes to encoding_rs internals. +- LLVM optimizations regressing between updates to the copy of LLVM + included in the Rust compiler. + +Microbench +========== + +- contact: :bholley +- source: + `MozGTestBench.cpp <https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp>`__ +- type: Custom GTest micro-benchmarking +- data: Time taken for a GTest function to execute +- summarization: Not a Talos test. This suite is provides a way to add + low level platform performance regression tests for things that are + not suited to be tested by Talos. + +PerfStrip Tests +=============== + +- contact: :davidb +- source: + https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp +- type: Microbench_ +- reporting: execution time in ms (lower is better) for 100k function + calls +- data: each test run and measured 5 times +- summarization: + +PerfStripWhitespace - call StripWhitespace() on 5 different test cases +20k times (each) + +PerfStripCharsWhitespace - call StripChars("\f\t\r\n") on 5 different +test cases 20k times (each) + +PerfStripCRLF - call StripCRLF() on 5 different test cases 20k times +(each) + +PerfStripCharsCRLF() - call StripChars("\r\n") on 5 different test cases +20k times (each) + +Stylo gtest microbenchmarks +=========================== + +- contact: :bholley, :SimonSapin +- source: + `gtest <https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest>`__ +- type: Microbench_ +- reporting: intervals in ms (lower is better) +- data: each test is run and measured 5 times +- summarization: take the `median`_ of the 5 data points; `source: + MozGTestBench.cpp <https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46>`__ + +Servo_StyleSheet_FromUTF8Bytes_Bench parses a sample stylesheet 20 times +with Stylo’s CSS parser that is written in Rust. It starts from an +in-memory UTF-8 string, so that I/O or UTF-16-to-UTF-8 conversion is not +measured. + +Gecko_nsCSSParser_ParseSheet_Bench does the same with Gecko’s previous +CSS parser that is written in C++, for comparison. + +Servo_DeclarationBlock_SetPropertyById_Bench parses the string "10px" +with Stylo’s CSS parser and sets it as the value of a property in a +declaration block, a million times. This is similar to animations that +are based on JavaScript code modifying Element.style instead of using +CSS @keyframes. + +Servo_DeclarationBlock_SetPropertyById_WithInitialSpace_Bench is the +same, but with the string " 10px" with an initial space. That initial +space is less typical of JS animations, but is almost always there in +stylesheets or full declarations like "width: 10px". This microbenchmark +was used to test the effect of some specific code changes. Regressions +here may be acceptable if Servo_StyleSheet_FromUTF8Bytes_Bench is not +affected. + +History of tp tests +******************* + +tp +== + +The original tp test created by Mozilla to test browser page load time. +Cycled through 40 pages. The pages were copied from the live web during +November, 2000. Pages were cycled by loading them within the main +browser window from a script that lived in content. + +tp2/tp_js +========= + +The same tp test but loading the individual pages into a frame instead +of the main browser window. Still used the old 40 page, year 2000 web +page test set. + +tp3 +=== + +An update to both the page set and the method by which pages are cycled. +The page set is now 393 pages from December, 2006. The pageloader is +re-built as an extension that is pre-loaded into the browser +chrome/components directories. + +tp4 +=== + +Updated web page test set to 100 pages from February 2009. + +tp4m +==== + +This is a smaller pageset (21 pages) designed for mobile Firefox. This +is a blend of regular and mobile friendly pages. + +We landed on this on April 18th, 2011 in `bug +648307 <https://bugzilla.mozilla.org/show_bug.cgi?id=648307>`__. This +runs for Android and Maemo mobile builds only. + +tp5 +=== + +Updated web page test set to 100 pages from April 8th, 2011. Effort was +made for the pages to no longer be splash screens/login pages/home pages +but to be pages that better reflect the actual content of the site in +question. There are two test page data sets for tp5 which are used in +multiple tests (i.e. awsy, xperf, etc.): (i) an optimized data set +called tp5o, and (ii) the standard data set called tp5n. + +tp6 +=== + +Created June 2017 with recorded pages via mitmproxy using modern google, +amazon, youtube, and facebook. Ideally this will contain more realistic +user accounts that have full content, in addition we would have more +than 4 sites- up to top 10 or maybe top 20. + +These were migrated to Raptor between 2018 and 2019. + +.. _geometric mean: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Data#geometric_mean +.. _ignore first: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Data#ignore_first +.. _median: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Data#median diff --git a/testing/perfdocs/generated/test-list.rst b/testing/perfdocs/generated/test-list.rst new file mode 100644 index 0000000000..a88e4971cf --- /dev/null +++ b/testing/perfdocs/generated/test-list.rst @@ -0,0 +1,73 @@ +################ +Raptor Test List +################ + +Currently the following Raptor tests are available. Note: Check the test details below to see which browser (i.e. Firefox, Google Chrome, Android) each test is supported on. + +Page-Load Tests +--------------- + +Raptor page-load test documentation is generated by `PerfDocs <https://firefox-source-docs.mozilla.org/code-quality/lint/linters/perfdocs.html>`_ and available in the `Firefox Source Docs <https://firefox-source-docs.mozilla.org/testing/perfdocs/raptor.html>`_. + +Benchmark Tests +--------------- + +motionmark-animometer, motionmark-htmlsuite +=========================================== + +* summarization: + + * subtest: FPS from the subtest, each subtest is run for 15 seconds, repeat this 5 times and report the median value + * suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite) + +speedometer +=========== + +* measuring: responsiveness of web applications +* data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks. +* summarization: + + * subtest: For all of the 16 subtests, we collect `a summed of all their internal benchmark results <https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/Speedometer/resources/benchmark-report.js#66-67>`_ for each of them. To obtain a single score per subtest, we take `a median of the replicates <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/output.py#427-470>`_. + * score: `geometric mean of the 16 subtest metrics (along with some special corrections) <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/output.py#317-330>`_. + +This is the `Speedometer v1.0 <http://browserbench.org/Speedometer/>`_ JavaScript benchmark taken verbatim and slightly modified to work with the Raptor harness. + +youtube-playback +================ + +* details: `YouTube playback performance <https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Youtube_playback_performance>`_ +* measuring: media streaming playback performance (dropped video frames) +* reporting: For each video the number of dropped and decoded frames, as well as its percentage value is getting recorded. The overall reported result is the mean value of dropped video frames across all tested video files. +* data: Given the size of the used media files those tests are currently run as live site tests, and are kept up-to-date via the `perf-youtube-playback <https://github.com/mozilla/perf-youtube-playback/>`_ repository on Github. + +This are the `Playback Performance Tests <https://ytlr-cert.appspot.com/2019/main.html?test_type=playbackperf-test>`_ benchmark taken verbatim and slightly modified to work with the Raptor harness. + +webaudio +======== + +* measuring: Rendering speed of various synthetic Web Audio API workloads +* reporting: The time time it took to render the audio of each test case, and a geometric mean of the full test suite. Lower is better +* data: Upstream is https://github.com/padenot/webaudio-benchmark/. Those benchmarks are run by other projects. Upstream is vendored in mozilla-central via an simple update script, at `third_party/webkit/PerformanceTests/webaudio` + +Scenario Tests +-------------- + +This test type runs browser tests that use idle pages for a specified amount of time to gather resource usage information such as power usage. The pages used for testing do not need to be recorded with mitmproxy. + +When creating a new scenario test, ensure that the `page-timeout` is greater than the `scenario-time` to make sure raptor doesn't exit the test before the scenario timer ends. + +This test type can also be used for specialized tests that require communication with the control-server to do things like sending the browser to the background for X minutes. + +Power-Usage Measurement Tests +============================= +These Android power measurement tests output 3 different PERFHERDER_DATA entries. The first contains the power usage of the test itself, the second contains the power usage of the android OS (named os-baseline) over the course of 1 minute, and the third (the name is the test name with '%change-power' appended to it) is a combination of these two measures which shows the percentage increase in power consumption when the test is run, in comparison to when it is not running. In these perfherder data blobs, we provide power consumption attributed to the cpu, wifi, and screen in Milli-ampere-hours (mAh). + +raptor-scn-power-idle +^^^^^^^^^^^^^^^^^^^^^ + +* measuring: Power consumption for idle Android browsers, with about:blank loaded and app foregrounded, over a 20-minute duration + +raptor-scn-power-idle-bg +^^^^^^^^^^^^^^^^^^^^^^^^ + +* measuring: Power consumption for idle Android browsers, with about:blank loaded and app backgrounded, over a 10-minute duration diff --git a/testing/perfdocs/generated/tools.rst b/testing/perfdocs/generated/tools.rst new file mode 100644 index 0000000000..4bb975e9f9 --- /dev/null +++ b/testing/perfdocs/generated/tools.rst @@ -0,0 +1,21 @@ +Running a performance tool +========================== + +You can run `perftest-tools` locally. + +Running locally +--------------- + +You can run `mach perftest-tools` in a mozilla-central source +checkout:: + + $ ./mach perftest-tools side-by-side --help + +The `mach` command will bootstrap the installation of all required dependencies for the +side-by-side tool to run. + +The following arguments are required: `-t/--test-name`, `--base-revision`, `--new-revision`, +`--base-platform` + +The `--help` argument will explain more about what arguments you need to +run in order to use the tool. diff --git a/testing/perfdocs/generated/variants-try-perf.png b/testing/perfdocs/generated/variants-try-perf.png Binary files differnew file mode 100644 index 0000000000..54050db962 --- /dev/null +++ b/testing/perfdocs/generated/variants-try-perf.png diff --git a/testing/perfdocs/generated/vision.rst b/testing/perfdocs/generated/vision.rst new file mode 100644 index 0000000000..7f176cf6a7 --- /dev/null +++ b/testing/perfdocs/generated/vision.rst @@ -0,0 +1,66 @@ +Vision +====== + +The `mozperftest` project was created with the intention to replace all +existing performance testing frameworks that exist in the mozilla central +source tree with a single one, and make performance tests a standardized, first-class +citizen, alongside mochitests and xpcshell tests. + +We want to give the ability to any developer to write performance tests in +their component, both locally and in the CI, exactly like how they would do with +`xpcshell` tests and `mochitests`. + +Historically, we have `Talos`, that provided a lot of different tests, from +micro-benchmarks to page load tests. From there we had `Raptor`, that was a +fork of Talos, focusing on page loads only. Then `mach browsertime` was added, +which was a wrapper around the `browsertime` tool. + +All those frameworks besides `mach browsertime` were mainly focusing on working +well in the CI, and were hard to use locally. `mach browsertime` worked locally but +not on all platforms and was specific to the browsertime framework. + +`mozperftest` currently provides the `mach perftest` command, that will scan +for all tests that are declared in ini files such as +https://searchfox.org/mozilla-central/source/netwerk/test/perf/perftest.ini and +registered under **PERFTESTS_MANIFESTS** in `moz.build` files such as +https://searchfox.org/mozilla-central/source/netwerk/test/moz.build#17 + +If you launch `./mach perftest` without any parameters, you will get a full list +of available tests, and you can pick and run one. Adding `--push-to-try` will +run it on try. + +The framework loads perf tests and read its metadata, that can be declared +within the test. We have a parser that is currently able to recognize and load +**xpcshell** tests and **browsertime** tests, and a runner for each one of those. + +But the framework can be extended to support more formats. We would like to add +support for **jsshell** and any other format we have in m-c. + +A performance test is a script that perftest runs, and that returns metrics we +can use. Right now we consume those metrics directly in the console, and +also in perfherder, but other formats could be added. For instance, there's +a new **influxdb** output that has been added, to push the data in an **influxdb** +time series database. + +What is important is to make sure performance tests belong to the component it's +testing in the source tree. We've learned with Talos that grouping all performance +tests in a single place is problematic because there's no sense of ownership from +developers once it's added there. It becomes the perf team problem. If the tests +stay in each component alongside mochitests and xpcshell tests, the component +maintainers will own and maintain it. + + +Next steps +---------- + +We want to rewrite all Talos and Raptor tests into perftest. For Raptor, we need +to have the ability to use proxy records, which is a work in progress. From there, +running a **raptor** test will be a simple, one liner browsertime script. + +For Talos, we'll need to refactor the existing micro-benchmarks into xpchsell tests, +and if that does not suffice, create a new runner. + +For JS benchmarks, once the **jsshell** runner is added into perftest, it will be +straightforward. + + diff --git a/testing/perfdocs/generated/webextension.rst b/testing/perfdocs/generated/webextension.rst new file mode 100644 index 0000000000..94e770b672 --- /dev/null +++ b/testing/perfdocs/generated/webextension.rst @@ -0,0 +1,525 @@ +################### +Raptor WebExtension +################### + + **Note: WebExtension is being deprecated, new tests should be written using Browsertime** + +.. contents:: + :depth: 2 + :local: + +WebExtension Page-Load Tests +---------------------------- + +Page-load tests involve loading a specific web page and measuring the load performance (i.e. `time-to-first-non-blank-paint <https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Non-Blank_Paint_.28fnbpaint.29>`_, first-contentful-paint, `dom-content-flushed <https://wiki.mozilla.org/TestEngineering/Performance/Glossary#DOM_Content_Flushed_.28dcf.29>`_). + +For page-load tests by default, instead of using live web pages for performance testing, Raptor uses a tool called `Mitmproxy <https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Mitmproxy>`_. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on `tooltool <https://github.com/mozilla/build-tooltool>`_ and are automatically downloaded by Raptor when they are required for a test. Raptor uses mitmproxy via the `mozproxy <https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy>`_ package. + +There are two different types of Raptor page-load tests: warm page-load and cold page-load. + +Warm Page-Load +============== +For warm page-load tests, the browser is just started up once; so the browser is warm on each page-load. + +**Raptor warm page-load test process when running on Firefox/Chrome/Chromium desktop:** + +* A new browser profile is created +* The desktop browser is started up +* Post-startup browser settle pause of 30 seconds +* A new tab is opened +* The test URL is loaded; measurements taken +* The tab is reloaded 24 more times; measurements taken each time +* The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts + +**Raptor warm page-load test process when running on Firefox android browser apps:** + +* The android app data is cleared (via ``adb shell pm clear firefox.app.binary.name``) +* The new browser profile is copied onto the android device SD card +* The Firefox android app is started up +* Post-startup browser settle pause of 30 seconds +* The test URL is loaded; measurements taken +* The tab is reloaded 14 more times; measurements taken each time +* The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts + +Cold Page-Load +============== +For cold page-load tests, the browser is shut down and restarted between page load cycles, so the browser is cold on each page-load. This is what happens for Raptor cold page-load tests: + +**Raptor cold page-load test process when running on Firefox/Chrome/Chromium desktop:** + +* A new browser profile is created +* The desktop browser is started up +* Post-startup browser settle pause of 30 seconds +* A new tab is opened +* The test URL is loaded; measurements taken +* The tab is closed +* The desktop browser is shut down +* Entire process is repeated for the remaining browser cycles (25 cycles total) +* The measurements from all browser cycles are used to calculate overall results + +**Raptor cold page-load test process when running on Firefox Android browser apps:** + +* The Android app data is cleared (via ``adb shell pm clear firefox.app.binary.name``) +* A new browser profile is created +* The new browser profile is copied onto the Android device SD card +* The Firefox Android app is started up +* Post-startup browser settle pause of 30 seconds +* The test URL is loaded; measurements taken +* The Android app is shut down +* Entire process is repeated for the remaining browser cycles (15 cycles total) +* Note that the SSL cert DB is only created once (browser cycle 1) and copied into the profile for each additional browser cycle, thus not having to use the 'certutil' tool and re-created the db on each cycle +* The measurements from all browser cycles are used to calculate overall results + +Using Live Sites +================ +It is possible to use live web pages for the page-load tests instead of using the mitmproxy recordings. To do this, add ``--live`` to your command line or select one of the 'live' variants when running ``./mach try fuzzy --full``. + +Disabling Alerts +================ +It is possible to disable alerting for all our performance tests. Open the target test manifest such as the raptor-tp6*.ini file (`Raptor tests folder <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests>`_), and make sure there are no ``alert_on`` specifications. + +When it's removed there will no longer be a ``shouldAlert`` field in the output Perfherder data (you can find the `schema here <https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json#68,165>`_). As long as ``shouldAlert`` is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3. + +High value tests +================ + +We have a notion of **high-value** tests in performance testing. These are chosen based on how many alerts they can catch using `this script <https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/generate_high_value_tests.py>`_ along with `a redash query <https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/sql_query.txt>`_. The lists below are the minimum set of test pages we should run to catch as many alerts as we can. + +**Desktop** + +Last updated: June 2021 + + - amazon + - bing + - cnn + - fandom + - gslides + - instagram + - twitter + - wikipedia + - yahoo-mail + +**Mobile** + +Last updated: November 2020 + + - allrecipes + - amazon-search + - espn + - facebook + - google-search + - microsoft-support + - youtube-watch + +WebExtension Benchmark Tests +---------------------------- + +Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI. + +Scenario Tests +-------------- + +Currently, there are three subtypes of Raptor-run "scenario" tests, all on (and only on) Android: + +#. **power-usage tests** +#. **memory-usage tests** +#. **CPU-usage tests** + +For a combined-measurement run with distinct Perfherder output for each measurement type, you can do: + +:: + + ./mach raptor --test raptor-scn-power-idle-bg-fenix --app fenix --binary org.mozilla.fenix.performancetest --host 10.0.0.16 --power-test --memory-test --cpu-test + +Each measurement subtype (power-, memory-, and cpu-usage) will have a corresponding PERFHERDER_DATA blob: + +:: + + 22:31:05 INFO - raptor-output Info: PERFHERDER_DATA: {"framework": {"name": "raptor"}, "suites": [{"name": "raptor-scn-power-idle-bg-fenix-cpu", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 0, "subtests": [{"lowerIsBetter": true, "unit": "%", "name": "cpu-browser_cpu_usage", "value": 0, "alertThreshold": 2.0}], "type": "cpu", "unit": "%"}]} + 22:31:05 INFO - raptor-output Info: cpu results can also be found locally at: /Users/sdonner/moz_src/mozilla-unified/testing/mozharness/build/raptor-cpu.json + +(repeat for power, memory snippets) + +Power-Use Tests (Android) +========================= +Prerequisites +^^^^^^^^^^^^^ + + +#. rooted (i.e. superuser-capable), bootloader-unlocked Moto G5 or Google Pixel 2: internal (for now) `test-device setup doc. <https://docs.google.com/document/d/1XQLtvVM2U3h1jzzzpcGEDVOp4jMECsgLYJkhCfAwAnc/edit>`_ +#. set up to run Raptor from a Firefox source tree (see `Running Locally <https://wiki.mozilla.org/Performance_sheriffing/Raptor#Running_Locally>`_) +#. `GeckoView-bootstrapped <https://wiki.mozilla.org/Performance_sheriffing/Raptor#Running_on_the_Android_GeckoView_Example_App>`_ environment + +**Raptor power-use measurement test process when running on Firefox Android browser apps:** + +* The Android app data is cleared, via: + +:: + + adb shell pm clear firefox.app.binary.name + + +* The new browser profile is copied onto the Android device's SD card +* We set ``scenario_time`` to **20 minutes** (1200000 milliseconds), and ``page_timeout`` to '22 minutes' (1320000 milliseconds) + +**It's crucial that ``page_timeout`` exceed ``scenario_time``; if not, measurement tests will fail/bail early** + +* We launch the {Fenix, GeckoView, Reference Browser} on-Android app +* Post-startup browser settle pause of 30 seconds +* On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab) +* Power-use/battery-level measurements (app-specific measurements) are taken, via: + +:: + + adb shell dumpsys batterystats + + +* Raw power-use measurement data is listed in the perfherder-data.json/raptor.json artifacts + +In the Perfherder dashboards for these power usage tests, all data points have milli-Ampere-hour units, with a lower value being better. +Proportional power usage is the total power usage of hidden battery sippers that is proportionally "smeared"/distributed across all open applications. + +Running Scenario Tests Locally +============================== + +To run on a tethered phone via USB from a macOS host, on: + +Fenix +^^^^^ + + +:: + + ./mach raptor --test raptor-scn-power-idle-fenix --app fenix --binary org.mozilla.fenix.performancetest --power-test --host 10.252.27.96 + +GeckoView +^^^^^^^^^ + + +:: + + ./mach raptor --test raptor-scn-power-idle-geckoview --app geckoview --binary org.mozilla.geckoview_example --power-test --host 10.252.27.96 + +Reference Browser +^^^^^^^^^^^^^^^^^ + + +:: + + ./mach raptor --test raptor-scn-power-idle-refbrow --app refbrow --binary org.mozilla.reference.browser.raptor --power-test --host 10.252.27.96 + +**NOTE:** +*it is important that you include* ``--power-test``, *when running power-usage measurement tests, as that will help ensure that local test-measurement data doesn't accidentally get submitted to Perfherder* + +Writing New Tests +================= + +Pushing to Try server +===================== +As an example, a relatively good cross-sampling of builds can be seen in https://hg.mozilla.org/try/rev/6c07631a0c2bf56b51bb82fd5543d1b34d7f6c69. + +* Include both G5 Android 7 (hw-g5-7-0-arm7-api-16/) *and* Pixel 2 Android 8 (p2-8-0-android-aarch64/) target platforms +* pgo builds tend to be -- from my limited empirical evidence -- about 10 - 15 minutes longer to complete than their opt counterparts + +Dashboards +========== + +See `performance results <https://wiki.mozilla.org/TestEngineering/Performance/Results>`_ for our various dashboards. + +Running WebExtension Locally +---------------------------- + +WebExtension Prerequisites +========================== + +In order to run Raptor on a local machine, you need: + +* A local mozilla repository clone with a `successful Firefox build <https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions>`_ completed +* Git needs to be in the path in the terminal/window in which you build Firefox / run Raptor, as Raptor uses Git to check-out a local copy for some of the performance benchmarks' sources. +* If you plan on running Raptor tests on Google Chrome, you need a local install of Google Chrome and know the path to the chrome binary +* If you plan on running Raptor on Android, your Android device must already be set up (see more below in the Android section) + +Getting a List of Raptor Tests +============================== + +To see which Raptor performance tests are currently available on all platforms, use the 'print-tests' option, e.g.: + +:: + + $ ./mach raptor --print-tests + +That will output all available tests on each supported app, as well as each subtest available in each suite (i.e. all the pages in a specific page-load tp6* suite). + +Running on Firefox +================== + +To run Raptor locally, just build Firefox and then run: + +:: + + $ ./mach raptor --test <raptor-test-name> + +For example, to run the raptor-tp6 pageload test locally, just use: + +:: + + $ ./mach raptor --test raptor-tp6-1 + +You can run individual subtests too (i.e. a single page in one of the tp6* suites). For example, to run the amazon page-load test on Firefox: + +:: + + $ ./mach raptor --test raptor-tp6-amazon-firefox + +Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json. + +Running on the Android GeckoView Example App +============================================ + +When running Raptor tests on a local Android device, Raptor is expecting the device to already be set up and ready to go. + +First, ensure your local host machine has the Android SDK/Tools (i.e. ADB) installed. Check if it is already installed by attaching your Android device to USB and running: + +:: + + $ adb devices + +If your device serial number is listed, then you're all set. If ADB is not found, you can install it by running (in your local mozilla-development repo): + +:: + + $ ./mach bootstrap + +Then, in bootstrap, select the option for "Firefox for Android Artifact Mode," which will install the required tools (no need to do an actual build). + +Next, make sure your Android device is ready to go. Local Android-device prerequisites are: + +* Device is `rooted <https://docs.google.com/document/d/1XQLtvVM2U3h1jzzzpcGEDVOp4jMECsgLYJkhCfAwAnc/edit>`_ + +Note: If you are using Magisk to root your device, use `version 17.3 <https://github.com/topjohnwu/Magisk/releases/tag/v17.3>`_ + +* Device is in 'superuser' mode + +* The geckoview example app is already installed on the device (from ``./mach bootstrap``, above). Download the geckoview_example.apk from the appropriate `android build on treeherder <https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2Cbuild>`_, then install it on your device, i.e.: + +:: + + $ adb install -g ../Downloads/geckoview_example.apk + +The '-g' flag will automatically set all application permissions ON, which is required. + +Note, when the Gecko profiler should be run, or a build with build symbols is needed, then use a `Nightly build of geckoview_example.apk <https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=nightly%2Candroid>`_. + +When updating the geckoview example app, you MUST uninstall the existing one first, i.e.: + +:: + + $ adb uninstall org.mozilla.geckoview_example + +Once your Android device is ready, and attached to local USB, from within your local mozilla repo use the following command line to run speedometer: + +:: + + $ ./mach raptor --test raptor-speedometer --app=geckoview --binary="org.mozilla.geckoview_example" + +Note: Speedometer on Android GeckoView is currently running on two devices in production - the Google Pixel 2 and the Moto G5 - therefore it is not guaranteed that it will run successfully on all/other untested android devices. There is an intermittent failure on the Moto G5 where speedometer just stalls (`Bug 1492222 <https://bugzilla.mozilla.org/show_bug.cgi?id=1492222>`_). + +To run a Raptor page-load test (i.e. tp6m-1) on the GeckoView Example app, use this command line: + +:: + + $ ./mach raptor --test raptor-tp6m-1 --app=geckoview --binary="org.mozilla.geckoview_example" + +A couple notes about debugging: + +* Raptor browser-extension console messages *do* appear in adb logcat via the GeckoConsole - so this is handy: + +:: + + $ adb logcat | grep GeckoConsole + +* You can also debug Raptor on Android using the Firefox WebIDE; click on the Android device listed under "USB Devices" and then "Main Process" or the 'localhost: Speedometer.." tab process + +Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json. + +Running on Google Chrome +======================== + +To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run: + +:: + + $ ./mach raptor --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>" + +For example, to run the raptor-speedometer benchmark on Google Chrome use: + +:: + + $ ./mach raptor --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome + +Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json. + +Page-Timeouts +============= + +On different machines the Raptor tests will run at different speeds. The default page-timeout is defined in each Raptor test INI file. On some machines you may see a test failure with a 'raptor page-timeout' which means the page-load timed out, or the benchmark test iteration didn't complete, within the page-timeout limit. + +You can override the default page-timeout by using the --page-timeout command-line arg. In this example, each test page in tp6-1 will be given two minutes to load during each page-cycle: + +:: + + ./mach raptor --test raptor-tp6-1 --page-timeout 120000 + +If an iteration of a benchmark test is not finishing within the allocated time, increase it by: + +:: + + ./mach raptor --test raptor-speedometer --page-timeout 600000 + +Page-Cycles +=========== + +Page-cycles is the number of times a test page is loaded (for page-load tests); for benchmark tests, this is the total number of iterations that the entire benchmark test will be run. The default page-cycles is defined in each Raptor test INI file. + +You can override the default page-cycles by using the --page-cycles command-line arg. In this example, the test page will only be loaded twice: + +:: + + ./mach raptor --test raptor-tp6-google-firefox --page-cycles 2 + +Running Page-Load Tests on Live Sites +===================================== +To use live pages instead of page recordings, just edit the `Raptor tp6* test INI <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests>`_ file and add the following attribute either at the top (for all pages in the suite) or under an individual page/subtest heading: + + use_live_pages = true + +With that setting, Raptor will not start the playback tool (i.e. Mitmproxy) and will not turn on the corresponding browser proxy, therefore forcing the test page to load live. + +Running Raptor on Try +--------------------- + +Raptor tests can be run on `try <https://treeherder.mozilla.org/#/jobs?repo=try>`_ on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above). + +**Note:** Raptor is currently 'tier 2' on `Treeherder <https://treeherder.mozilla.org/#/jobs?repo=try>`_, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected / turned on in the Treeherder 'Tiers' menu. + +The easiest way to run Raptor tests on try is to use mach try fuzzy: + +:: + + $ ./mach try fuzzy --full + +Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run. + +To see the Raptor test results on your try run: + +#. In treeherder select one of the Raptor test jobs (i.e. 'sp' in 'Rap-e10s', or 'Rap-C-e10s') +#. Below the jobs, click on the "Performance" tab; you'll see the aggregated results listed +#. If you wish to see the raw replicates, click on the "Job Details" tab, and select the "perfherder-data.json" artifact + +Raptor Hardware in Production +============================= + +The Raptor performance tests run on dedicated hardware (the same hardware that the Talos performance tests use). See the `performance platforms <https://wiki.mozilla.org/TestEngineering/Performance/Platforms>`_ for more details. + +Profiling Raptor Jobs +--------------------- + +Raptor tests are able to create Gecko profiles which can be viewed in `profiler.firefox.com. <https://profiler.firefox.com/>`__ This is currently only supported when running Raptor on Firefox desktop. + +Nightly Profiling Jobs in Production +==================================== +We have Firefox desktop Raptor jobs with Gecko-profiling enabled running Nightly in production on Mozilla Central (on Linux64, Win10, and OSX). This provides a steady cache of Gecko profiles for the Raptor tests. Search for the `"Rap-Prof" treeherder group on Mozilla Central <https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=Rap-Prof>`_. + +Profiling Locally +================= + +To tell Raptor to create Gecko profiles during a performance test, just add the '--gecko-profile' flag to the command line, i.e.: + +:: + + $ ./mach raptor --test raptor-sunspider --gecko-profile + +When the Raptor test is finished, you will be able to find the resulting gecko profiles (ZIP) located locally in: + + mozilla-central/testing/mozharness/build/blobber_upload_dir/ + +Note: While profiling is turned on, Raptor will automatically reduce the number of pagecycles to 3. If you wish to override this, add the --page-cycles argument to the raptor command line. + +Raptor will automatically launch Firefox and load the latest Gecko profile in `profiler.firefox.com <https://profiler.firefox.com>`__. To turn this feature off, just set the DISABLE_PROFILE_LAUNCH=1 env var. + +If auto-launch doesn't work for some reason, just start Firefox manually and browse to `profiler.firefox.com <https://profiler.firefox.com>`__, click on "Browse" and select the Raptor profile ZIP file noted above. + +If you're on Windows and want to profile a Firefox build that you compiled yourself, make sure it contains profiling information and you have a symbols zip for it, by following the `directions on MDN <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows#Profiling_local_talos_runs>`_. + +Profiling on Try Server +======================= + +To turn on Gecko profiling for Raptor test jobs on try pushes, just add the '--gecko-profile' flag to your try push i.e.: + +:: + + $ ./mach try fuzzy --gecko-profile + +Then select the Raptor test jobs that you wish to run. The Raptor jobs will be run on try with profiling included. While profiling is turned on, Raptor will automatically reduce the number of pagecycles to 2. + +See below for how to view the gecko profiles from within treeherder. + +Customizing the profiler +======================== +If the default profiling options are not enough, and further information is needed the gecko profiler can be customized. + +Enable profiling of additional threads +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +In some cases it will be helpful to also measure threads which are not part of the default set. Like the **MediaPlayback** thread. This can be accomplished by using: + +#. the **gecko_profile_threads** manifest entry, and specifying the thread names as comma separated list +#. the **--gecko-profile-thread** argument for **mach** for each extra thread to profile + +Add Profiling to Previously Completed Jobs +========================================== + +Note: You might need treeherder 'admin' access for the following. + +Gecko profiles can now be created for Raptor performance test jobs that have already completed in production (i.e. mozilla-central) and on try. To repeat a completed Raptor performance test job on production or try, but add gecko profiling, do the following: + +#. In treeherder, select the symbol for the completed Raptor test job (i.e. 'ss' in 'Rap-e10s') +#. Below, and to the left of the 'Job Details' tab, select the '...' to show the menu +#. On the pop-up menu, select 'Create Gecko Profile' + +The same Raptor test job will be repeated but this time with gecko profiling turned on. A new Raptor test job symbol will be added beside the completed one, with a '-p' added to the symbol name. Wait for that new Raptor profiling job to finish. See below for how to view the gecko profiles from within treeherder. + +Viewing Profiles on Treeherder +============================== +When the Raptor jobs are finished, to view the gecko profiles: + +#. In treeherder, select the symbol for the completed Raptor test job (i.e. 'ss' in 'Rap-e10s') +#. Click on the 'Job Details' tab below +#. The Raptor profile ZIP files will be listed as job artifacts; +#. Select a Raptor profile ZIP artifact, and click the 'view in Firefox Profiler' link to the right + +Recording Pages for Raptor Pageload Tests +----------------------------------------- + +Raptor pageload tests ('tp6' and 'tp6m' suites) use the `Mitmproxy <https://mitmproxy.org/>`__ tool to record and play back page archives. For more information on creating new page playback archives, please see `Raptor and Mitmproxy <https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Mitmproxy>`__. + +Performance Tuning for Android devices +-------------------------------------- + +When the test is run against Android, Raptor executes a series of performance tuning commands over the ADB connection. + +Device agnostic: + +* memory bus +* device remain on when on USB power +* virtual memory (swappiness) +* services (thermal throttling, cpu throttling) +* i/o scheduler + +Device specific: + +* cpu governor +* cpu minimum frequency +* gpu governor +* gpu minimum frequency + +For a detailed list of current tweaks, please refer to `this <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#676>`_ Searchfox page. diff --git a/testing/perfdocs/generated/writing.rst b/testing/perfdocs/generated/writing.rst new file mode 100644 index 0000000000..14764b4a1f --- /dev/null +++ b/testing/perfdocs/generated/writing.rst @@ -0,0 +1,176 @@ +Performance scripts +=================== + +Performance scripts are programs that drive the browser to run a specific +benchmark (like a page load or a lower level call) and produce metrics. + +We support two flavors right now in `perftest` (but it's easy to add +new ones): + +- **xpcshell** a classical xpcshell test, turned into a performance test +- **browsertime** a browsertime script, which runs a full browser and controls + it via a Selenium client. + +In order to qualify as performance tests, both flavors require metadata. + +For our supported flavors that are both Javascript modules, those are +provided in a `perfMetadata` mapping variable in the module, or in +the `module.exports` variable when using Node. + +This is the list of fields: + +- **owner**: name of the owner (person or team) [mandatory] +- **author**: author of the test +- **name**: name of the test [mandatory] +- **description**: short description [mandatory] +- **longDescription**: longer description +- **options**: options used to run the test +- **supportedBrowsers**: list of supported browsers (or "Any") +- **supportedPlatforms**: list of supported platforms (or "Any") +- **tags** a list of tags that describe the test + +Tests are registered using tests manifests and the **PERFTESTS_MANIFESTS** +variable in `moz.build` files - it's good practice to name this file +`perftest.ini`. + +Example of such a file: https://searchfox.org/mozilla-central/source/testing/performance/perftest.ini + + +xpcshell +-------- + +`xpcshell` tests are plain xpcshell tests, with two more things: + +- the `perfMetadata` variable, as described in the previous section +- calls to `info("perfMetrics", ...)` to send metrics to the `perftest` framework. + +Here's an example of such a metrics call:: + + # compute some speed metrics + let speed = 12345; + info("perfMetrics", { speed }); + + +Browsertime +----------- + +With the browsertime layer, performance scenarios are Node modules that +implement at least one async function that will be called by the framework once +the browser has started. The function gets a webdriver session and can interact +with the browser. + +You can write complex, interactive scenarios to simulate a user journey, +and collect various metrics. + +Full documentation is available `here <https://www.sitespeed.io/documentation/sitespeed.io/scripting/>`_ + +The mozilla-central repository has a few performance tests script in +`testing/performance` and more should be added in components in the future. + +By convention, a performance test is prefixed with **perftest_** to be +recognized by the `perftest` command. + +A performance test implements at least one async function published in node's +`module.exports` as `test`. The function receives two objects: + +- **context**, which contains: + + - **options** - All the options sent from the CLI to Browsertime + - **log** - an instance to the log system so you can log from your navigation script + - **index** - the index of the runs, so you can keep track of which run you are currently on + - **storageManager** - The Browsertime storage manager that can help you read/store files to disk + - **selenium.webdriver** - The Selenium WebDriver public API object + - **selenium.driver** - The instantiated version of the WebDriver driving the current version of the browser + +- **command** provides API to interact with the browser. It's a wrapper + around the selenium client `Full documentation here <https://www.sitespeed.io/documentation/sitespeed.io/scripting/#commands>`_ + + +Below is an example of a test that visits the BBC homepage and clicks on a link. + +.. sourcecode:: javascript + + "use strict"; + + async function setUp(context) { + context.log.info("setUp example!"); + } + + async function test(context, commands) { + await commands.navigate("https://www.bbc.com/"); + + // Wait for browser to settle + await commands.wait.byTime(10000); + + // Start the measurement + await commands.measure.start("pageload"); + + // Click on the link and wait for page complete check to finish. + await commands.click.byClassNameAndWait("block-link__overlay-link"); + + // Stop and collect the measurement + await commands.measure.stop(); + } + + async function tearDown(context) { + context.log.info("tearDown example!"); + } + + module.exports = { + setUp, + test, + tearDown, + owner: "Performance Team", + test_name: "BBC", + description: "Measures pageload performance when clicking on a link from the bbc.com", + supportedBrowsers: "Any", + supportePlatforms: "Any", + }; + + +Besides the `test` function, scripts can implement a `setUp` and a `tearDown` function to run +some code before and after the test. Those functions will be called just once, whereas +the `test` function might be called several times (through the `iterations` option) + + +Hooks +----- + +A Python module can be used to run functions during a run lifecycle. Available hooks are: + +- **before_iterations(args)** runs before everything is started. Gets the args, which + can be changed. The **args** argument also contains a **virtualenv** variable that + can be used for installing Python packages (e.g. through `install_package <https://searchfox.org/mozilla-central/source/python/mozperftest/mozperftest/utils.py#115-144>`_). +- **before_runs(env)** runs before the test is launched. Can be used to + change the running environment. +- **after_runs(env)** runs after the test is done. +- **on_exception(env, layer, exception)** called on any exception. Provides the + layer in which the exception occurred, and the exception. If the hook returns `True` + the exception is ignored and the test resumes. If the hook returns `False`, the + exception is ignored and the test ends immediately. The hook can also re-raise the + exception or raise its own exception. + +In the example below, the `before_runs` hook is setting the options on the fly, +so users don't have to provide them in the command line:: + + from mozperftest.browser.browsertime import add_options + + url = "'https://www.example.com'" + + common_options = [("processStartTime", "true"), + ("firefox.disableBrowsertimeExtension", "true"), + ("firefox.android.intentArgument", "'-a'"), + ("firefox.android.intentArgument", "'android.intent.action.VIEW'"), + ("firefox.android.intentArgument", "'-d'"), + ("firefox.android.intentArgument", url)] + + + def before_runs(env, **kw): + add_options(env, common_options) + + +To use this hook module, it can be passed to the `--hooks` option:: + + $ ./mach perftest --hooks hooks.py perftest_example.js + + diff --git a/testing/perfdocs/moz.build b/testing/perfdocs/moz.build new file mode 100644 index 0000000000..9905709937 --- /dev/null +++ b/testing/perfdocs/moz.build @@ -0,0 +1 @@ +SPHINX_TREES["/testing/perfdocs"] = "generated" |
