1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
|
Code coverage
=============
What is Code Coverage?
----------------------
**Code coverage** essentially measures how often certain lines are hit,
branches taken or conditions met in a program, given some test that you
run on it.
There are two very important things to keep in mind when talking about
code coverage:
- If a certain branch of code is not hit at all while running tests,
then those tests will never be able to find a bug in this particular
piece of the code.
- If a certain branch of code is executed (even very often), this still
is not a clear indication of the *quality of a test*. It could be
that a test exercises the code but does not actually check that the
code performs *correctly*.
As a conclusion, we can use code coverage to find areas that need (more)
tests, but we cannot use it to confirm that certain areas are well
tested.
Firefox Code Coverage reports
-----------------------------
We automatically run code coverage builds and tests on all
mozilla-central runs, for Linux and Windows. C/C++, Rust and JavaScript
are supported.
The generated reports can be found at https://coverage.moz.tools/. The
reports can be filtered by platform and/or test suite.
We also generate a report of all totally uncovered files, which can be
found at https://coverage.moz.tools/#view=zero. You can use this to find
areas of code that should be tested, or code that is no longer used
(dead code, which could be removed).
C/C++ Code Coverage on Firefox
------------------------------
There are several ways to get C/C++ coverage information for
mozilla-central, including creating your own coverage builds. The next
sections describe the available options.
Generate Code Coverage report from a try build (or any other CI build)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To spin a code coverage build, you need to select the linux64-ccov
platform (use --full when using the fuzzy selector to get the ccov
builds to show up).
E.g. for a try build:
.. code:: shell
./mach try fuzzy -q 'linux64-ccov'
There are two options now, you can either generate the report locally or
use a one-click loaner.
Generate report using a one-click loaner
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Select the B job on Treeherder and get a one-click loaner.
In the loaner, download and execute the script
https://github.com/mozilla/code-coverage/blob/master/report/firefox_code_coverage/codecoverage.py:
.. code:: shell
wget https://raw.githubusercontent.com/mozilla/code-coverage/master/report/firefox_code_coverage/codecoverage.py
python codecoverage.py
This command will automatically generate a HTML report of the code
coverage information in the **report** subdirectory in your current
working directory.
Generate report locally
^^^^^^^^^^^^^^^^^^^^^^^
Prerequisites:
- Create and activate a new `virtualenv`_, then run:
.. code:: shell
pip install firefox-code-coverage
Given a treeherder linux64-ccov build (with its branch, e.g.
\`mozilla-central\` or \`try`, and revision, the tip commit hash of your
push), run the following command:
.. code:: shell
firefox-code-coverage PATH/TO/MOZILLA/SRC/DIR/ BRANCH REVISION
This command will automatically download code coverage artifacts from
the treeherder build and generate an HTML report of the code coverage
information. The report will be stored in the **report** subdirectory in
your current working directory.
.. _virtualenv: https://docs.python.org/3/tutorial/venv.html
Creating your own Coverage Build
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Linux, Windows and Mac OS X it is straightforward to generate an
instrumented build using GCC or Clang. Adding the following lines to
your ``.mozconfig`` file should be sufficient:
.. code:: shell
# Enable code coverage
ac_add_options --enable-coverage
# Needed for e10s:
# With the sandbox, content processes can't write updated coverage counters in the gcda files.
ac_add_options --disable-sandbox
Some additional options might be needed, check the code-coverage
mozconfigs used on CI to be sure:
browser/config/mozconfigs/linux64/code-coverage,
browser/config/mozconfigs/win64/code-coverage,
browser/config/mozconfigs/macosx64/code-coverage.
Make sure you are not running with :ref:`artifact build <Understanding Artifact Builds>`
enabled, as it can prevent coverage artifacts from being created.
You can then create your build as usual. Once the build is complete, you
can run any tests/tools you would like to run and the coverage data gets
automatically written to special files. In order to view/process this
data, we recommend using the
`grcov <https://github.com/mozilla/grcov>`__ tool, a tool to manage and
visualize gcov results. You can also use the same process explained
earlier for CI builds.
Debugging Failing Tests on the Try Server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When code coverage is run through a push to try, all the data that is
created is ingested by ActiveData and processed into a different data
format for analysis. Anytime a code coverage run generates \*.gcda and
\*.gcno files, ActiveData starts working. Now, sometimes, a test will
permanently fail when it is running on a build that is instrumented with
GCOV. To debug these issues without overloading ActiveData with garbage
coverage data, open the file
`taskcluster/gecko_taskgraph/transforms/test/__init__.py <https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/transforms/test/__init__.py#516>`__
and add the following line,
.. code:: python
test['mozharness'].setdefault('extra-options', []).append('--disable-ccov-upload')
right after this line of code:
.. code:: python
test['mozharness'].setdefault('extra-options', []).append('--code-coverage')
Now when you push to try to debug some failing tests, or anything else,
there will not be any code coverage artifacts uploaded from the build
machines or from the test machines.
JS Debugger Per Test Code Coverage on Firefox
---------------------------------------------
There are two ways to get javascript per test code coverage information
for mozilla-central. The next sections describe these options.
Generate Per Test Code Coverage from a try build (or any other treeherder build)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To spin a code coverage build, you need to select the linux64-jsdcov
platform. E.g. for a try build:
.. code:: shell
./mach try fuzzy -q 'linux64-jsdcov'
This produces JavaScript Object Notation (JSON) files that can be
downloaded from the treeherder testing machines and processed or
analyzed locally.
Generate Per Test Code Coverage Locally
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To generate the JSON files containing coverage information locally, simply
add an extra argument called ``--jscov-dir-prefix`` which accepts a
directory as it's input and stores the resulting data in that directory.
For example, to collect code coverage for the entire Mochitest suite:
.. code:: shell
./mach mochitest --jscov-dir-prefix /PATH/TO/COVERAGE/DIR/
Currently, only the Mochitest and Xpcshell test suites have this
capability.
|
