blob: 4464fce1dc9e11a92905209cfb58cfdf037e49aa (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
|
Tests
=====
A high-level test strategy for Firefox Telemetry is defined in the
`Test Strategy document <https://docs.google.com/document/d/1Mi6va3gE4HSv5LjXNREvMa2V4q-LKIFDTwA2o4yeo_c/edit>`_.
Firefox Telemetry is a complicated and old component.
So too are the organization and expanse of its tests.
Let’s break them down by harness.
Unless otherwise mentioned the tests live in subdirectories of
``toolkit/components/telemetry/tests``.
Mochitest
---------
:Location: ``t/c/t/t/browser/``
:Language: Javascript
(`mochitest <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest>`_)
This test harness runs nearly the entire Firefox and gives access to multiple tabs and browser chrome APIs.
It requires window focus to complete correctly,
so it isn’t recommended to add new tests here.
The tests that are here maybe would be more at home as telemetry-tests-client tests as they tend to be integration tests.
Google Test
-----------
:Location: ``t/c/t/t/gtest/``
:Language: C++
(`googletest <https://github.com/google/googletest>`_)
This test harness runs a specially-built gtest shell around libxul which allows you to write unit tests against public C++ APIs.
It should be used to test the C++ API and core of Firefox Telemetry.
This is for tests like
“Do we correctly accumulate to bucket 0 if I pass -1 to ``Telemetry::Accumulate``?”
Integration Tests (telemetry-tests-client and telemetry-integration-tests)
--------------------------------------------------------------------------
:Location: ``t/c/t/t/marionette/tests/client`` and ``t/c/t/t/integration/``
:Language: Python
(`unittest <https://docs.python.org/3/library/unittest.html>`_,
`pytest <https://docs.pytest.org/en/latest/>`_)
The most modern of the test harnesses,
telemetry-integration-tests uses marionette to puppet the entire browser allowing us to write integration tests that include ping servers and multiple browser runs.
You should use this if you’re testing Big Picture things like
“Does Firefox resend its “deletion-request” ping if the network is down when Telemetry is first disabled?”.
At time of writing there are two “editions” of integration tests.
Prefer writing new tests in telemetry-tests-client
(the unittest-based one in ``t/c/t/t/marionette/tests/client``)
while we evaluate CI support for telemetry-integration-tests.
Definitions Files Tests
-----------------------
:Location: ``t/c/t/t/python``
:Language: Python
(`unittest <https://docs.python.org/3/library/unittest.html>`_)
This harness pulls in the parsers and scripts used to turn JSON and YAML probe definitions into code.
It should be used to test the build scripts and formats of the definitions files
Histograms.json, Scalars.yaml, and Events.yaml.
This is for tests like
“Does the build fail if someone forgot to put in a bug number for a new Histogram?”.
xpcshell
--------
:Location: ``t/c/t/t/unit``
:Language: Javascript
(`xpcshell <https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests>`_)
This test harness uses a stripped-down shell of the Firefox browser to run privileged Javascript.
It should be used to write unit tests for the Javascript API and app-level logic of Firefox Telemetry.
This is for tests like
“Do we correctly accumulate to bucket 0 if I pass -1 to ``Telemetry.getHistogramById(...).add``?”
and
“Do we reschedule pings that want to be sent near local midnight?”.
Since these tests are easy to write and quick to run we have in the past bent this harness in a few interesting shapes
(see PingServer)
to have it support integration tests as well.
New integration tests should use telemetry-tests-client instead.
Instrumentation Tests
---------------------
:Location: Various
:Language: Usually Javascript
(`xpcshell <https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests>`_ or
`mochitest <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest>`_)
In addition to the tests of Firefox Telemetry,
other code owners have written tests that ensure that their code records appropriate values to Telemetry.
They should use the
``toolkit/components/telemetry/tests/unit/TelemetryTestUtils.jsm``
module to make their lives easier.
This can be used for tests like
“If five bookmarks are read from the database,
does the bookmark count Histogram have a value of 5 in it?”.
|
