diff options
Diffstat (limited to 'testing/perfdocs')
| -rw-r--r-- | testing/perfdocs/generated/browsertime.rst | 54 | ||||
| -rw-r--r-- | testing/perfdocs/generated/index.rst | 2 | ||||
| -rw-r--r-- | testing/perfdocs/generated/performance-infrastructure.rst | 43 | ||||
| -rw-r--r-- | testing/perfdocs/generated/raptor.rst | 1712 | ||||
| -rw-r--r-- | testing/perfdocs/generated/talos.rst | 832 |
5 files changed, 2481 insertions, 162 deletions
diff --git a/testing/perfdocs/generated/browsertime.rst b/testing/perfdocs/generated/browsertime.rst index 4d1975224f..beff3f0ad8 100644 --- a/testing/perfdocs/generated/browsertime.rst +++ b/testing/perfdocs/generated/browsertime.rst @@ -34,7 +34,7 @@ For example, here's a test on ``https://www.sitespeed.io`` using this custom tes :: - ./mach raptor --browsertime -t browsertime --browsertime-arg test_script=pageload --browsertime-arg browsertime.url=https://www.sitespeed.io --browsertime-arg iterations=3 + ./mach raptor -t browsertime --browsertime-arg test_script=pageload --browsertime-arg browsertime.url=https://www.sitespeed.io --browsertime-arg iterations=3 That test will perform 3 iterations of the given url. Note also that we can use simplified names to make use of test scripts that are built into raptor. You can use ``pageload``, ``interactive``, or provide a path to another test script. @@ -50,7 +50,7 @@ There are two ways to run performance tests through browsertime listed below. :: - ./mach raptor --browsertime -t google-search + ./mach raptor -t google-search * Browsertime-"native": @@ -64,23 +64,25 @@ Benchmark tests :: - ./mach raptor -t raptor-speedometer --browsertime + ./mach raptor -t speedometer Running on Android ------------------ +To run on android, the device needs to have Geckoview or Fenix installed on it. Our tests will only work with physical devices, and `bug 1881570 <https://bugzilla.mozilla.org/show_bug.cgi?id=1881570>`__ tracks progress for enabling virtual devices (emulators). Running either of the commands below will attempt to install locally built APKs to the device **while uninstalling/removing any existing APKs of the package on the device**, but this can be skipped by setting ``MOZ_DISABLE_ADB_INSTALL=1`` in your environment. When that environment variable exists, we expect the APK to be pre-installed on the device. + Running on Raptor-Browsertime (recommended): * Running on Fenix :: - ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix + ./mach raptor -t amazon --app fenix --binary org.mozilla.fenix * Running on Geckoview :: - ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example + ./mach raptor -t amazon --app geckoview --binary org.mozilla.geckoview_example Running on vanilla Browsertime: @@ -122,7 +124,7 @@ Or for Raptor-Browsertime (use ``chrome`` for desktop, and ``chrome-m`` for mobi :: - ./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER> + ./mach raptor -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER> Running Page-load tests with third party WebExtensions ------------------------------------------------------ @@ -137,14 +139,14 @@ Launch amazon tp6 page-load test on Firefox Desktop: :: - ./mach raptor --browsertime -t amazon --conditioned-profile settled-webext + ./mach raptor -t amazon --conditioned-profile settled-webext Launch amazon tp6 mobile page-load test on Firefox for Android (the apk has to be pre-installed, mach raptor does detect if already installed but it does not install it): :: - ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix --conditioned-profile settled-webext + ./mach raptor -t amazon --app fenix --binary org.mozilla.fenix --conditioned-profile settled-webext To run these jobs on try, make sure to select the tp6 jobs that include the string `webextensions`, as an example (add ``--no-push`` to force try fuzzy to only list the jobs selected by the try fuzzy query) to run all tp6 page-load webextensons jobs currently defined: @@ -153,12 +155,14 @@ list the jobs selected by the try fuzzy query) to run all tp6 page-load webexten ./mach try fuzzy -q "'tp6 'webextensions" -Similarly for running tp6m on Firefox for Android builds: +Similarly for running tp6m (equivalent to tp6 but for mobile) on Firefox for Android builds: :: ./mach try fuzzy --full -q "'tp6m 'webextensions" +Note that this can also be done using ``./mach try perf --show-all -q "'tp6m 'webextensions"`` to produce a compare view link of the changes before/after the patch being tested. + The set of extensions installed are the ones listed in the ``"addons"`` property of the condprof customization file `webext.json`_ from the ``testing/condprofile/condprof/customization/`` directory. @@ -196,20 +200,27 @@ Other methods for adding additional arguments are: Running Browsertime on Try -------------------------- -You can run all of our browsertime pageload tests through ``./mach try fuzzy --full``. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time. -For example: +You can run all of our browsertime pageload tests through ``./mach try perf`` by selecting the ``Pageload`` category. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time. There are a lot of other tests/categories available as well. Documentation about this tool can be found in :ref:`Mach Try Perf`. + +For example, the following will select all ``Pageload`` categories to run on desktop: + +:: + + ./mach try perf -q "'Pageload" + +If you need to target android tasks, include the ``--android`` flag like so (remove the ``'android`` from the query string to target desktop tests at the same time): :: - ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable" + ./mach try perf --android -q "'Pageload 'android" -Retriggering Browsertime Visual Metrics Tasks ---------------------------------------------- +If you exclude the ``-q "..."`` option, an interface similar to the fuzzy interface will open, and show all available categories. -You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.). +Visual Metrics +-------------- -The following metrics are collected each time: ``fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange`` +The following visual metrics are collected in all page load tests: ``ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange`` Further information regarding these metrics can be viewed at `visual-metrics <https://www.sitespeed.io/documentation/sitespeed.io/metrics/#visual-metrics>`_ @@ -220,7 +231,7 @@ To run gecko profiling using Raptor-Browsertime you can add the ``--gecko-profil :: - ./mach raptor --browsertime -t amazon --gecko-profile + ./mach raptor -t amazon --gecko-profile Note that vanilla Browsertime does support Gecko Profiling but **it does not symbolicate the profiles** so it is **not recommended** to use for debugging performance regressions/improvements. @@ -248,6 +259,7 @@ Likewise, for chrome trace you will want to be aware of `these defaults. <https: Upgrading Browsertime In-Tree ----------------------------- + To upgrade the browsertime version used in-tree you can run, then commit the changes made to ``package.json`` and ``package-lock.json``: :: @@ -260,6 +272,7 @@ To test the upgrade, run a raptor test locally (with and without visual-metrics Updating Benchmark Tests ------------------------ + To upgrade any of our benchmark tests, you will need to change the revision used in the test manifest. There are three fields that you have available to use there: ``repository_revision`` to denote the revision, ``repository_branch`` to denote the branch name, and ``repository`` to provide the link of the Github repo that contains the benchmark. For instance, with Speedometer 3 (sp3), we can update the revision `by changing the repository_revision field found here <https://searchfox.org/mozilla-central/rev/aa3ccd258b64abfd4c5ce56c1f512bc7f65b844c/testing/raptor/raptor/tests/benchmarks/speedometer-desktop.ini#29>`_. If the change isn't found on the default branch (master/main branch), then you will need to add an entry for ``repository_branch`` to specify this. @@ -268,6 +281,7 @@ If the path to the test file changes (the file that is invoked to run the test), Finding the Geckodriver Being Used ---------------------------------- + If you're looking for the latest geckodriver being used there are two ways: * Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver * Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the ``Task`` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in. @@ -287,7 +301,7 @@ Mach Browsertime Setup ---------------------- **WARNING** - Raptor-Browsertime (i.e. ``./mach raptor --browsertime -t <TEST>``) is currently required to be ran first in order to acquire the Node-16 binary. In general, it is also not recommended to use ``./mach browsertime`` for testing as it will be deprecated soon. + Raptor-Browsertime (i.e. ``./mach raptor -t <TEST>``) is currently required to be ran first in order to acquire the Node-16 binary. In general, it is also not recommended to use ``./mach browsertime`` for testing as it will be deprecated soon. Note that if you are running Raptor-Browsertime then it will get installed automatically and also update itself. Otherwise, you can run: @@ -311,8 +325,8 @@ With the replacement of ImageMagick, former cross platform installation issues h -- For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command or a Raptor-Browsertime test +- For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command or a Raptor-Browsertime test. Alternatively, you may need to delete the ``tools/browsertime/node_modules`` folder. -- If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the :ref: `Running on Android` section) +- If you plan on running Browsertime on Android, your Android device must already be set up (see more above in the :ref: `Running on Android` section) - **If you encounter any issues not mentioned here, please** `file a bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Raptor>`_ **in the** ``Testing::Raptor`` **component.** diff --git a/testing/perfdocs/generated/index.rst b/testing/perfdocs/generated/index.rst index 2fe8e16de6..082affa202 100644 --- a/testing/perfdocs/generated/index.rst +++ b/testing/perfdocs/generated/index.rst @@ -14,6 +14,7 @@ Performance Testing mach-try-perf mozperftest perf-sheriffing + performance-infrastructure raptor talos @@ -30,6 +31,7 @@ For more detailed information about each test suite and project, see their docum * :doc:`mach-try-perf` * :doc:`mozperftest` * :doc:`perf-sheriffing` + * :doc:`performance-infrastructure` * :doc:`raptor` * :doc:`talos` diff --git a/testing/perfdocs/generated/performance-infrastructure.rst b/testing/perfdocs/generated/performance-infrastructure.rst new file mode 100644 index 0000000000..b8f52dfe60 --- /dev/null +++ b/testing/perfdocs/generated/performance-infrastructure.rst @@ -0,0 +1,43 @@ +========================== +Performance Infrastructure +========================== + +.. contents:: + :depth: 3 + + +Performance tests all run on bare matel, or real hardware devices instead of virtual machines. This gives us more realistic performance metrics, and helps with decreasing the variability of our data. See below for information on what hardware is used for each of the platforms we test on, try run wait times, as well as guidelines for requesting new devices. + + +Platforms, and Hardware Used +---------------------------- + +At the moment, these are listed in the Mozilla Wiki which is `found here <https://wiki.mozilla.org/Performance/Platforms>`_ (see `bug 1883400 <https://bugzilla.mozilla.org/show_bug.cgi?id=1883400>`__ for progress on this). + + +Try Runs, and Wait Times +------------------------ + +Given that our tests run on hardware, there's a limited amount of devices that can be used to run them. This means that it's very likely that a try run (e.g. tests scheduled by :ref:`Mach Try Perf`) will be delayed waiting for capacity to free up. This limited capacity is also why there is a limit of 600 tasks that can be scheduled with ``mach try perf``. + +Something to keep in mind is that try runs have a low priority, and our production branches (e.g. autoland/mozilla-central) have a higher priority. On days when there are more pushes to those branches, try runs will hit more delays. The platforms also have different capacities available to them which will change how long you have to wait for tests to start on them. **To find out how many tasks are currently pending, or running across all platforms** `see this redash query <https://sql.telemetry.mozilla.org/queries/98004#241985>`_ **or consult the graph below.** + +.. raw:: html + + <iframe src="https://sql.telemetry.mozilla.org/embed/query/98004/visualization/241985?api_key=MNTnV4fwt3oblbx1uXDK9njIvDUa6rp1sla9RENT&" width="720" height="550"></iframe> + +`See this dashboard for average wait times across all the platforms <https://sql.telemetry.mozilla.org/dashboard/average-performance-test-wait-times>`_. + +In some cases, the tasks may hit a "task timeout" where they expire before they can run. This can be expected if there are a lot of other higher priority tasks being scheduled, so it's good to check the query mentioned above to see what the load looks like on the platform being targeted. However, there are situations where the test pools go offline or other issues occur. In general, for these task timeouts, reach out in `#perftest on Matrix <https://matrix.to/#/#perftest:mozilla.org>`__ to notify us about the issue as we might not be aware of it. + + +Requesting New Devices for Testing +---------------------------------- + +At times, it can be useful to test changes on a different device to due a lack of configuration coverage with our existing set of devices. It's simple to request/expense a device for local testing if you need one. However, for testing in continuous integration (CI) and/or in try runs, it can be quite expensive, and time-consuming to get a device ready for it. In general it takes a **few months** to get the device(s) running, and given contractual limitations, it may also reduce the availability of other devices. The work spans multiple teams as well so the time it takes depends on the current/future tasks that those teams have at the moment. + +If a device is only required for a single issue, then it's recommended to order the device for local testing. If it's a long term project with multiple developers/teams that needs continuous monitoring, then it could be useful to have the devices in CI so they can be run on mozilla-central/autoland. This can also help ensure that your changes don't regress over time. Note that this should be planned ahead of time so that there's time to setup the devices before work on the project starts. + +For long term projects, it's also good to determine how long these devices would be needed for, and if the device setup in CI needs to be adjusted to encompass these additional configurations. If the devices are no longer needed after the long term project completes, it might not be useful to have these devices running in CI, and using local devices would be quicker, and cheaper to get. + +If you need to get a device for local testing, reach out to your manager to ask about expensing it. If you believe that you need to start testing on a new device in CI, please reach out to us in `#perftest on Matrix <https://matrix.to/#/#perftest:mozilla.org>`__. Alternatively, you can `file a bug here <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Performance&status_whiteboard=[fxp]>`_ for support on this. diff --git a/testing/perfdocs/generated/raptor.rst b/testing/perfdocs/generated/raptor.rst index 7914587277..789fbb6883 100644 --- a/testing/perfdocs/generated/raptor.rst +++ b/testing/perfdocs/generated/raptor.rst @@ -994,6 +994,691 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr +.. dropdown:: motionmark-1-3 + :class-container: anchor-id-motionmark-1-3-b + + **Owner**: Graphics Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 1 + * **cold**: true + * **expected**: pass + * **expose browser profiler**: true + * **expose chrome trace**: true + * **gecko profile features**: stackwalk,js,cpu,java,processcpu + * **gecko profile interval**: 1 + * **gecko profile threads**: GeckoMain,Compositor,Renderer,SwComposite,RenderBackend,SceneBuilder,WrWorker,CanvasWorkers,TextureUpdate + * **host from parent**: false + * **lower is better**: false + * **page cycles**: 1 + * **page timeout**: 600000 + * **repository**: https://github.com/webkit/motionmark + * **repository revision**: 5d9c88136d59c11daf78d539c73e4e3e88c091ab + * **suite name**: MotionMark + * **support class**: motionmark-1-3.py + * **test script**: motionmark-1-3.js + * **test url**: `<http://\<host\>:\<port\>/MotionMark/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=30&display=minimal&tiles=big&controller=ramp&system-frame-rate=60&frame-rate=60&time-measurement=performance>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-chrome-m-motionmark-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-cstm-car-m-motionmark-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1400-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + **Owner**: Graphics Team + + * **alert threshold**: 2.0 + * **apps**: fenix, geckoview, chrome-m, cstm-car-m + * **browser cycles**: 1 + * **cold**: true + * **expected**: pass + * **expose browser profiler**: true + * **gecko profile features**: stackwalk,js,cpu,java,processcpu + * **gecko profile interval**: 1 + * **gecko profile threads**: GeckoMain,Compositor,Renderer,SwComposite,RenderBackend,SceneBuilder,WrWorker,CanvasWorkers,TextureUpdate + * **host from parent**: false + * **lower is better**: false + * **page cycles**: 1 + * **page timeout**: 600000 + * **repository**: https://github.com/webkit/motionmark + * **repository revision**: 5d9c88136d59c11daf78d539c73e4e3e88c091ab + * **suite name**: MotionMark + * **support class**: motionmark-1-3.py + * **test script**: motionmark-1-3.js + * **test url**: `<http://\<host\>:\<port\>/MotionMark/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=30&display=minimal&tiles=big&controller=ramp&system-frame-rate=60&frame-rate=60&time-measurement=performance>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-chrome-m-motionmark-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-cstm-car-m-motionmark-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1400-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. dropdown:: motionmark-animometer :class-container: anchor-id-motionmark-animometer-b @@ -2714,6 +3399,691 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr +.. dropdown:: motionmark-htmlsuite-1-3 + :class-container: anchor-id-motionmark-htmlsuite-1-3-b + + **Owner**: Graphics Team + + * **alert threshold**: 2.0 + * **apps**: firefox, chrome, chromium, safari, custom-car + * **browser cycles**: 1 + * **cold**: true + * **expected**: pass + * **expose browser profiler**: true + * **expose chrome trace**: true + * **gecko profile features**: stackwalk,js,cpu,java,processcpu + * **gecko profile interval**: 1 + * **gecko profile threads**: GeckoMain,Compositor,Renderer,SwComposite,RenderBackend,SceneBuilder,WrWorker,CanvasWorkers,TextureUpdate + * **host from parent**: false + * **lower is better**: false + * **page cycles**: 1 + * **page timeout**: 600000 + * **repository**: https://github.com/webkit/motionmark + * **repository revision**: 5d9c88136d59c11daf78d539c73e4e3e88c091ab + * **suite name**: HTML suite + * **support class**: motionmark-1-3.py + * **test script**: motionmark-1-3.js + * **test url**: `<http://\<host\>:\<port\>/MotionMark/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=30&display=minimal&tiles=big&controller=ramp&system-frame-rate=60&frame-rate=60&time-measurement=performance>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-htmlsuite-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-htmlsuite-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-chrome-m-motionmark-htmlsuite-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-cstm-car-m-motionmark-htmlsuite-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-htmlsuite-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-htmlsuite-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1400-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + **Owner**: Graphics Team + + * **alert threshold**: 2.0 + * **apps**: fenix, geckoview, chrome-m, cstm-car-m + * **browser cycles**: 1 + * **cold**: true + * **expected**: pass + * **expose browser profiler**: true + * **gecko profile features**: stackwalk,js,cpu,java,processcpu + * **gecko profile interval**: 1 + * **gecko profile threads**: GeckoMain,Compositor,Renderer,SwComposite,RenderBackend,SceneBuilder,WrWorker,CanvasWorkers,TextureUpdate + * **host from parent**: false + * **lower is better**: false + * **page cycles**: 1 + * **page timeout**: 600000 + * **repository**: https://github.com/webkit/motionmark + * **repository revision**: 5d9c88136d59c11daf78d539c73e4e3e88c091ab + * **suite name**: HTML suite + * **support class**: motionmark-1-3.py + * **test script**: motionmark-1-3.js + * **test url**: `<http://\<host\>:\<port\>/MotionMark/developer.html?warmup-length=2000&warmup-frame-count=30&first-frame-minimum-length=0&test-interval=30&display=minimal&tiles=big&controller=ramp&system-frame-rate=60&frame-rate=60&time-measurement=performance>`__ + * **type**: benchmark + * **unit**: score + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-htmlsuite-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-htmlsuite-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-motionmark-1-3-chrome-m-motionmark-htmlsuite-1-3-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-cstm-car-m-motionmark-htmlsuite-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-fenix-motionmark-htmlsuite-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-motionmark-1-3-geckoview-motionmark-htmlsuite-1-3-nofis** + - ✅ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-clang-trunk-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-linux1804-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1015-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-macosx1400-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-benchmark-safari-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-32-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-nightlyasrelease-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows10-64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-benchmark-chrome-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-custom-car-motionmark-htmlsuite-1-3** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-benchmark-firefox-motionmark-htmlsuite-1-3** + - ✅ + - ✅ + - ❌ + - ❌ + + + .. dropdown:: motionmark-htmlsuite-ramp :class-container: anchor-id-motionmark-htmlsuite-ramp-b @@ -4111,7 +5481,7 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr * **page cycles**: 5 * **page timeout**: 600000 * **repository**: https://github.com/WebKit/Speedometer - * **repository revision**: 64a5e7c0ba0f668834c0adf01b933e54d2b0d1f0 + * **repository revision**: 8d67f28d0281ac4330f283495b7f48286654ad7d * **subtest lower is better**: true * **subtest unit**: ms * **support class**: speedometer3.py @@ -4183,12 +5553,12 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr - mozilla-release - mozilla-beta * - **browsertime-benchmark-speedometer3-mobile-chrome-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ * - **browsertime-benchmark-speedometer3-mobile-cstm-car-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ @@ -4214,12 +5584,12 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr - mozilla-release - mozilla-beta * - **browsertime-benchmark-speedometer3-mobile-chrome-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ * - **browsertime-benchmark-speedometer3-mobile-cstm-car-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ @@ -4507,7 +5877,7 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr * **page cycles**: 5 * **page timeout**: 420000 * **repository**: https://github.com/WebKit/Speedometer - * **repository revision**: 64a5e7c0ba0f668834c0adf01b933e54d2b0d1f0 + * **repository revision**: 8d67f28d0281ac4330f283495b7f48286654ad7d * **subtest lower is better**: true * **subtest unit**: ms * **support class**: speedometer3.py @@ -4579,12 +5949,12 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr - mozilla-release - mozilla-beta * - **browsertime-benchmark-speedometer3-mobile-chrome-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ * - **browsertime-benchmark-speedometer3-mobile-cstm-car-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ @@ -4610,12 +5980,12 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr - mozilla-release - mozilla-beta * - **browsertime-benchmark-speedometer3-mobile-chrome-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ * - **browsertime-benchmark-speedometer3-mobile-cstm-car-m-nofis** - - ❌ + - ✅ - ❌ - ❌ - ❌ @@ -10775,6 +12145,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send many small ArrayBuffers to IndexedDB in one transaction + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -11022,6 +12394,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send many small ArrayBuffers to IndexedDB independently + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -11269,6 +12643,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send many small Arrays to IndexedDB in one transaction + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -11516,6 +12892,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send many small Arrays to IndexedDB independently + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -11763,6 +13141,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send many small Blobs to IndexedDB in one transaction + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -12010,6 +13390,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send many small Blobs to IndexedDB independently + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -12257,6 +13639,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send a couple of large ArrayBuffers to IndexedDB in one transaction + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -12504,6 +13888,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send a couple of large ArrayBuffers to IndexedDB independently + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -12751,6 +14137,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send a couple of large JS Arrays to IndexedDB in one transaction + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -12998,6 +14386,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send a couple of large JS Arrays to IndexedDB independently + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -13245,6 +14635,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send a couple of large Blobs to IndexedDB in one transaction + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -13492,6 +14884,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Use add API to send a couple of large Blobs to IndexedDB independently + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -13739,6 +15133,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: Used to run vanilla browsertime tests through raptor. For example: `./mach raptor --browsertime -t browsertime --browsertime-arg url=https://www.sitespeed.io --browsertime-arg iterations=3` + * **alert on**: fcp, loadtime * **alert threshold**: 2.0 * **apps**: firefox @@ -13763,6 +15159,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: Network Team + **Description**: Verifies that connection time on a pageload is zero when a speculative connection has first been made + * **alert change type**: absolute * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange * **alert threshold**: 10.0 @@ -13964,6 +15362,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: Generates a constant value that can be changed to induce a regression. + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari, geckoview, fenix * **browser cycles**: 1 @@ -14025,6 +15425,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: DOM Lifecycle, Workers and Storage Team + **Description**: Measures the speed of key-range based reads of IndexedDB + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -14271,6 +15673,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: Runs the Grandprix benchmark + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -14296,6 +15700,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: Measures process switch time + * **accept zero vismet**: true * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 @@ -14500,6 +15906,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: A sample test that uses a python support file to modify the test command. + * **alert threshold**: 2.0 * **apps**: firefox * **browser cycles**: 1 @@ -14528,6 +15936,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: Network Team + **Description**: Pageload test using a throttled network environment + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange * **alert threshold**: 2.0 * **apps**: firefox @@ -14728,6 +16138,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: A pageload benchmark test that quickly runs through all of the pages available to produce a single overall metric. + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -14933,6 +16345,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: Network Team + **Description**: Measures http/2 file upload throughput + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -15161,6 +16575,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: Network Team + **Description**: Measures http/3 file upload throughput + * **alert threshold**: 2.0 * **apps**: firefox, chrome, chromium, safari * **browser cycles**: 1 @@ -15389,6 +16805,8 @@ Browsertime tests that use a custom pageload test script. These use the pageload **Owner**: PerfTest Team + **Description**: Measures pageload metrics for the first-install about:welcome page + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 * **apps**: firefox @@ -27060,6 +28478,8 @@ Browsertime tests that interact with the webpage. Includes responsiveness tests **Owner**: PerfTest Team + **Description**: Navigates to cnn main page, then to the world sub-page. + * **accept zero vismet**: true * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 @@ -27299,6 +28719,8 @@ Browsertime tests that interact with the webpage. Includes responsiveness tests **Owner**: PerfTest Team + **Description**: Navigates to facebook, then the sub-pages friends, marketplace, groups. + * **accept zero vismet**: true * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 @@ -27538,6 +28960,8 @@ Browsertime tests that interact with the webpage. Includes responsiveness tests **Owner**: PerfTest Team + **Description**: Navigates from the Bill Gates AMA to the Reddit members section. + * **accept zero vismet**: true * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 @@ -27777,6 +29201,8 @@ Browsertime tests that interact with the webpage. Includes responsiveness tests **Owner**: PerfTest Team + **Description**: Navigates the `thisisbillgates` user starting at the main user page, then to the posts, comments, hot, and top sections. + * **accept zero vismet**: true * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 @@ -28016,6 +29442,8 @@ Browsertime tests that interact with the webpage. Includes responsiveness tests **Owner**: PerfTest Team + **Description**: Navigates the `thisisbillgates` user starting at the main user page, then to the posts, comments, hot, and top sections. + * **accept zero vismet**: true * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange * **alert threshold**: 2.0 @@ -29839,6 +31267,137 @@ Page-load performance test suite on Android. The links direct to the actual webs +.. dropdown:: bild-de + :class-container: anchor-id-bild-de-m + + **Owner**: PerfTest Team + + * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange, largestContentfulPaint + * **alert threshold**: 2.0 + * **apps**: geckoview, fenix, refbrow, chrome-m + * **benchmark page**: true + * **browser cycles**: 15 + * **expected**: pass + * **lower is better**: true + * **page cycles**: 25 + * **page timeout**: 60000 + * **playback**: mitmproxy-android + * **playback pageset manifest**: mitm8-android-fenix-bild-de.manifest + * **playback version**: 8.1.1 + * **test url**: `<https://www.bild.de/sport/american-football/nfl/super-bowl-rekordquote-fuer-die-nfl-87139668.bild.html>`__ + * **type**: pageload + * **unit**: ms + * **use live sites**: false + * **Test Task**: + + .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-fenix-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-live-fenix-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-live-geckoview-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **browsertime-tp6m-chrome-m-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-chrome-m-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-fenix-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-geckoview-bild-de-nofis** + - ✅ + - ✅ + - ❌ + - ❌ + * - **browsertime-tp6m-live-chrome-m-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-live-fenix-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-live-geckoview-bild-de-nofis** + - ❌ + - ❌ + - ❌ + - ❌ + * - **browsertime-tp6m-refbrow-bild-de** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. dropdown:: bing :class-container: anchor-id-bing-m @@ -33863,7 +35422,7 @@ Page-load performance test suite on Android. The links direct to the actual webs * **page cycles**: 25 * **page timeout**: 60000 * **playback**: mitmproxy-android - * **playback pageset manifest**: mitm6-android-fenix-sina.manifest + * **playback pageset manifest**: mitm8-android-fenix-sina.manifest * **playback version**: 8.1.1 * **test url**: `<https://www.sina.com.cn/>`__ * **type**: pageload @@ -34085,137 +35644,6 @@ Page-load performance test suite on Android. The links direct to the actual webs -.. dropdown:: web-de - :class-container: anchor-id-web-de-m - - **Owner**: PerfTest Team - - * **alert on**: fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange, largestContentfulPaint - * **alert threshold**: 2.0 - * **apps**: geckoview, fenix, refbrow, chrome-m - * **benchmark page**: true - * **browser cycles**: 15 - * **expected**: pass - * **lower is better**: true - * **page cycles**: 25 - * **page timeout**: 60000 - * **playback**: mitmproxy-android - * **playback pageset manifest**: mitm7-android-gve-p2-web-de.manifest - * **playback version**: 8.1.1 - * **test url**: `<https://web.de/magazine/politik/politologe-glaubt-grossen-koalition-herbst-knallen-33563566>`__ - * **type**: pageload - * **unit**: ms - * **use live sites**: false - * **Test Task**: - - .. list-table:: **test-android-hw-a51-11-0-aarch64-qr/opt** - :widths: 30 15 15 15 15 - :header-rows: 1 - - * - **Test Name** - - mozilla-central - - autoland - - mozilla-release - - mozilla-beta - * - **browsertime-tp6m-fenix-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-fenix-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-geckoview-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-geckoview-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-live-fenix-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-live-geckoview-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-refbrow-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - - - .. list-table:: **test-android-hw-a51-11-0-aarch64-shippable-qr/opt** - :widths: 30 15 15 15 15 - :header-rows: 1 - - * - **Test Name** - - mozilla-central - - autoland - - mozilla-release - - mozilla-beta - * - **browsertime-tp6m-chrome-m-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-chrome-m-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-fenix-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-fenix-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-geckoview-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-geckoview-web-de-nofis** - - ✅ - - ✅ - - ❌ - - ❌ - * - **browsertime-tp6m-live-chrome-m-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-live-fenix-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-live-geckoview-web-de-nofis** - - ❌ - - ❌ - - ❌ - - ❌ - * - **browsertime-tp6m-refbrow-web-de** - - ❌ - - ❌ - - ❌ - - ❌ - - - .. dropdown:: wikipedia :class-container: anchor-id-wikipedia-m diff --git a/testing/perfdocs/generated/talos.rst b/testing/perfdocs/generated/talos.rst index 3916fd1345..fe95ab6207 100644 --- a/testing/perfdocs/generated/talos.rst +++ b/testing/perfdocs/generated/talos.rst @@ -1501,6 +1501,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: cpstartup :class-container: anchor-id-cpstartup @@ -3117,6 +3169,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: glvideo :class-container: anchor-id-glvideo @@ -3412,6 +3516,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: kraken :class-container: anchor-id-kraken @@ -4033,6 +4189,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_main_2d_av1 :class-container: anchor-id-offscreencanvas_webcodecs_main_2d_av1 @@ -4329,6 +4537,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_main_2d_h264 :class-container: anchor-id-offscreencanvas_webcodecs_main_2d_h264 @@ -4625,6 +4885,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_main_2d_vp9 :class-container: anchor-id-offscreencanvas_webcodecs_main_2d_vp9 @@ -4921,6 +5233,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_main_webgl_av1 :class-container: anchor-id-offscreencanvas_webcodecs_main_webgl_av1 @@ -5217,6 +5581,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_main_webgl_h264 :class-container: anchor-id-offscreencanvas_webcodecs_main_webgl_h264 @@ -5513,6 +5929,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_main_webgl_vp9 :class-container: anchor-id-offscreencanvas_webcodecs_main_webgl_vp9 @@ -5809,6 +6277,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_worker_2d_av1 :class-container: anchor-id-offscreencanvas_webcodecs_worker_2d_av1 @@ -6105,6 +6625,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_worker_2d_h264 :class-container: anchor-id-offscreencanvas_webcodecs_worker_2d_h264 @@ -6401,6 +6973,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_worker_2d_vp9 :class-container: anchor-id-offscreencanvas_webcodecs_worker_2d_vp9 @@ -6697,6 +7321,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_worker_webgl_av1 :class-container: anchor-id-offscreencanvas_webcodecs_worker_webgl_av1 @@ -6993,6 +7669,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_worker_webgl_h264 :class-container: anchor-id-offscreencanvas_webcodecs_worker_webgl_h264 @@ -7289,6 +8017,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: offscreencanvas_webcodecs_worker_webgl_vp9 :class-container: anchor-id-offscreencanvas_webcodecs_worker_webgl_vp9 @@ -7585,6 +8365,58 @@ For the sample commands found below, note that the capitalization used is import - ❌ + .. list-table:: **test-windows11-64-2009-hw-ref-shippable/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ✅ + - ✅ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + + .. list-table:: **test-windows11-64-2009-hw-ref/opt** + :widths: 30 15 15 15 15 + :header-rows: 1 + + * - **Test Name** + - mozilla-central + - autoland + - mozilla-release + - mozilla-beta + * - **talos-webgl** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-gli** + - ❌ + - ❌ + - ❌ + - ❌ + * - **talos-webgl-swr** + - ❌ + - ❌ + - ❌ + - ❌ + + .. dropdown:: pdfpaint :class-container: anchor-id-pdfpaint |