1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
|
Performance scripts
===================
Performance scripts are programs that drive the browser to run a specific
benchmark (like a page load or a lower level call) and produce metrics.
We support two flavors right now in `perftest` (but it's easy to add
new ones):
- **xpcshell** a classical xpcshell test, turned into a performance test
- **browsertime** a browsertime script, which runs a full browser and controls
it via a Selenium client.
In order to qualify as performance tests, both flavors require metadata.
For our supported flavors that are both Javascript modules, those are
provided in a `perfMetadata` mapping variable in the module, or in
the `module.exports` variable when using Node.
This is the list of fields:
- **owner**: name of the owner (person or team) [mandatory]
- **author**: author of the test
- **name**: name of the test [mandatory]
- **description**: short description [mandatory]
- **longDescription**: longer description
- **options**: options used to run the test
- **supportedBrowsers**: list of supported browsers (or "Any")
- **supportedPlatforms**: list of supported platforms (or "Any")
- **tags** a list of tags that describe the test
Tests are registered using tests manifests and the **PERFTESTS_MANIFESTS**
variable in `moz.build` files - it's good practice to name this file
`perftest.ini`.
Example of such a file: https://searchfox.org/mozilla-central/source/testing/performance/perftest.ini
xpcshell
--------
`xpcshell` tests are plain xpcshell tests, with two more things:
- the `perfMetadata` variable, as described in the previous section
- calls to `info("perfMetrics", ...)` to send metrics to the `perftest` framework.
Here's an example of such a metrics call::
# compute some speed metrics
let speed = 12345;
info("perfMetrics", { speed });
Browsertime
-----------
With the browsertime layer, performance scenarios are Node modules that
implement at least one async function that will be called by the framework once
the browser has started. The function gets a webdriver session and can interact
with the browser.
You can write complex, interactive scenarios to simulate a user journey,
and collect various metrics.
Full documentation is available `here <https://www.sitespeed.io/documentation/sitespeed.io/scripting/>`_
The mozilla-central repository has a few performance tests script in
`testing/performance` and more should be added in components in the future.
By convention, a performance test is prefixed with **perftest_** to be
recognized by the `perftest` command.
A performance test implements at least one async function published in node's
`module.exports` as `test`. The function receives two objects:
- **context**, which contains:
- **options** - All the options sent from the CLI to Browsertime
- **log** - an instance to the log system so you can log from your navigation script
- **index** - the index of the runs, so you can keep track of which run you are currently on
- **storageManager** - The Browsertime storage manager that can help you read/store files to disk
- **selenium.webdriver** - The Selenium WebDriver public API object
- **selenium.driver** - The instantiated version of the WebDriver driving the current version of the browser
- **command** provides API to interact with the browser. It's a wrapper
around the selenium client `Full documentation here <https://www.sitespeed.io/documentation/sitespeed.io/scripting/#commands>`_
Below is an example of a test that visits the BBC homepage and clicks on a link.
.. sourcecode:: javascript
"use strict";
async function setUp(context) {
context.log.info("setUp example!");
}
async function test(context, commands) {
await commands.navigate("https://www.bbc.com/");
// Wait for browser to settle
await commands.wait.byTime(10000);
// Start the measurement
await commands.measure.start("pageload");
// Click on the link and wait for page complete check to finish.
await commands.click.byClassNameAndWait("block-link__overlay-link");
// Stop and collect the measurement
await commands.measure.stop();
}
async function tearDown(context) {
context.log.info("tearDown example!");
}
module.exports = {
setUp,
test,
tearDown,
owner: "Performance Team",
test_name: "BBC",
description: "Measures pageload performance when clicking on a link from the bbc.com",
supportedBrowsers: "Any",
supportePlatforms: "Any",
};
Besides the `test` function, scripts can implement a `setUp` and a `tearDown` function to run
some code before and after the test. Those functions will be called just once, whereas
the `test` function might be called several times (through the `iterations` option)
Hooks
-----
A Python module can be used to run functions during a run lifecycle. Available hooks are:
- **before_iterations(args)** runs before everything is started. Gets the args, which
can be changed. The **args** argument also contains a **virtualenv** variable that
can be used for installing Python packages (e.g. through `install_package <https://searchfox.org/mozilla-central/source/python/mozperftest/mozperftest/utils.py#115-144>`_).
- **before_runs(env)** runs before the test is launched. Can be used to
change the running environment.
- **after_runs(env)** runs after the test is done.
- **on_exception(env, layer, exception)** called on any exception. Provides the
layer in which the exception occured, and the exception. If the hook returns `True`
the exception is ignored and the test resumes. If the hook returns `False`, the
exception is ignored and the test ends immediatly. The hook can also re-raise the
exception or raise its own exception.
In the example below, the `before_runs` hook is setting the options on the fly,
so users don't have to provide them in the command line::
from mozperftest.browser.browsertime import add_options
url = "'https://www.example.com'"
common_options = [("processStartTime", "true"),
("firefox.disableBrowsertimeExtension", "true"),
("firefox.android.intentArgument", "'-a'"),
("firefox.android.intentArgument", "'android.intent.action.VIEW'"),
("firefox.android.intentArgument", "'-d'"),
("firefox.android.intentArgument", url)]
def before_runs(env, **kw):
add_options(env, common_options)
To use this hook module, it can be passed to the `--hooks` option::
$ ./mach perftest --hooks hooks.py perftest_example.js
|
