blob: ec54211038f9ed3b7425f530f0d681f3f94a1c97 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
=============================
IndexedDB Performance Testing
=============================
How to run tests on CI:
-----------------------
* Windows: ``mach try perf --show-all -q "test-windows10-64-shippable-qr/opt-browsertime-indexeddb"``
* Linux: ``mach try perf --show-all -q "test-linux1804-64-shippable-qr/opt-browsertime-indexeddb"``
* Mac: ``mach try perf --show-all -q "test-macosx1015-64-shippable-qr/opt-browsertime-indexeddb"``
* All but 32-bit jobs: ``mach try perf --chrome --safari --show-all -q 'shippable-browsertime-indexeddb !32'``
* In general:
* Open test selection interface with ``mach try perf --show-all``
* Filter out the preferred tests by typing letters which are expected to be part of the test job name string (as in the -q argument above)
* Note down the string used as a filter for rerunning the job (or rerun it with ``mach try again --list-tasks`` and ``mach try again --index``)
How to run tests locally with the profiler?
-------------------------------------------
* Build the browser with release or release with debug symbols flags (not in debug mode)
* Use ``mach raptor --browsertime -t $(test_name) --gecko-profile --post-startup-delay=1000`` where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini``
* After the test is complete, the generated profile is opened with the default browser.
* The generated profile file path is listed also in the command line output.
* For best symbolication results, it may help to
* run the same browser build that was used for the tests with ``./mach run``
* navigate to "profiler.firefox.com"
* use the "Load a profile from file" button
How to compare performance to a different browser?
--------------------------------------------------
* The test outputs a ``time_duration`` value for all supported browsers
* Using Chrome as an example,
* ``mach raptor --browsertime -t $(test_name) --post-startup-delay=1000 --app=chrome -b "/c/Program Files/Google/Chrome/Application/chrome.exe"``
* where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini``
* browser executable path after the ``-b`` argument varies locally
* in some cases, a test driver argument such as ``--browsertime-chromedriver`` may be required
How to add more tests?
----------------------
* For the test boilerplate, copy and rename an old test script such as ``testing/raptor/browsertime/indexeddb_write.js`` under the ``testing/raptor/browsertime/`` directory
* Modify the test case script argument of ``commands.js.run`` / Selenium's ``executeAsyncScript``
* Test parameters can be passed to such script with syntax ``${variable_name}`` where ``variable_name`` represents the parameter in the context of ``executeAsyncScript`` or ``commands.js.run``.
* Use quotes to capture a string value, for example ``"${variable_name}"``
* TIP: Debugging the test case could be simpler by serving it locally without the boilerplate
* Add ``[test_name]`` section to file ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`` where ``test_name`` **must be 10 characters or less** in order to be a valid ``Treeherder`` test name
* Under the ``[test_name]`` section, specify the test script name as a value of ``test_script =``
* Under the ``[test_name]`` section, specity the test parameters as a sequence of ``--browsertime.key=value`` arguments as a value of ``browsertime_args =``
* Under the ``[test_name]`` section, override any other values as needed
* Add test as a subtest to run for Desktop ``taskcluster/ci/test/browsertime-desktop.yml`` (maybe also for mobile)
* Add test documentation to ``testing/raptor/raptor/perfdocs/config.yml``
* Generated files:
* Run ``./mach lint --warnings --outgoing --fix`` to regenerate the documentation and task files, and warn about omissions
* Running ``./mach lint -l perfdocs --fix .`` may also be needed
* Testing:
* Test the new test by running it with the profiler
* Test the new test by running it with a different browser
* Test the new test by triggering it on CI
|
