1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
=================
Integration Tests
=================
The aim of the telemetry-tests-client suite is to verify Firefox collects telemetry probes, aggregates that data, and submits telemetry
pings containing the data to a HTTP server. The integration tests try to make no assumptions about the internal workings of Firefox and
use automation to mimic user behavior.
The integration test suite for Firefox Client Telemetry runs on CI `tier 1 <https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy>`_
with treeherder symbol `tt(c)`
and is checked in to version control at mozilla-central under
`toolkit/components/telemetry/tests/marionette/tests/client <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/marionette/tests/client/>`_.
Test Main Tab Scalars
---------------------
- PATH: ``telemetry/tests/marionette/tests/client/test_main_tab_scalars.py``
- This test opens and closes a number of browser tabs,
restarts the browser in a new session
and then verifies the correctness of scalar data in the resulting `main` ping.
Test Search Counts
------------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_search_counts_across_subsessions.py``
- This test performs a search in a new tab,
restarts Firefox in a new session and verifies the correctness of client, session and subsession IDs,
as well as scalar and keyed histogram data in the `shutdown` ping,
installs an addon, verifies the `environment-change` ping, and performs three additional search actions
before restarting and verifying the new `main` ping.
Test Deletion Request Ping
--------------------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_deletion_request_ping.py``
- This test installs an addon and verifies a ping is received. The test takes note of the client ID.
It then disables telemetry and checks for a `deletion-request` ping.
After it receives the correct ping it makes sure that no other pings are sent.
Telemetry is then re-enabled and the `main` ping is checked to see if the client ID has changed.
The test asserts that the user has opted back in to telemetry.
Test Event Ping
---------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_event_ping.py``
- This test checks for a basic `event` ping. It opens firefox, performs a search and checks the `event`
ping for the correct number of searches performed (1) and the correct search engine.
Test Fog Custom Ping
--------------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_custom_ping.py``
- This test creates a custom ping using the Glean API and asserts this ping is sent correctly.
Test Fog Deletion Request Ping
------------------------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_deletion_request_ping.py``
- This test opens the browser, performs a search and disables telemetry after the search.
It asserts that the telemetry is disabled and no pings exist.
The browser is restarted and telemetry is then re-enabled.
Then we set a `debug tag <https://mozilla.github.io/glean/book/user/debugging/debug-ping-view.html>`_
which is attached to the ping.
Telemetry is then disabled again to trigger a `deletion-request` ping.
We verify that 1) The debug tag is present; and 2) that the client ID
in the second `deletion-request` ping is different from the first client ID.
Test Fog User Activity
----------------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_user_activity.py``
- This test checks that a `baseline` ping is sent when the user starts or stops using Firefox.
Test Background Update Ping
---------------------------
- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_user_activity.py``
- In this test we launch Firefox to prepare a profile and to disable the background update setting.
We exit Firefox,
leaving the (unlocked) profile to be used as the default profile for the background update task (and not having multiple instances running).
The task will not try to update, but it will send a ping.
Then we restart Firefox to unwind the background update setting and allow shutdown to proceed cleanly.
Running the tests locally
-------------------------
You can run the tests on your local machine using
`mach <https://firefox-source-docs.mozilla.org/mach/index.html>`__:
``./mach telemetry-tests-client``
Running the tests on try
------------------------
You can run the tests across all platforms on the try server using
`mach <https://firefox-source-docs.mozilla.org/mach/index.html>`__:
``./mach try fuzzy -q "'telemetry-tests-client"``
Disabling an individual failing test
------------------------------------
The telemetry-tests-client suite is implemented in Python and uses Marionette for browser automation and wptserve for the HTTP ping server.
The integration tests are based on Python's unittest testing library and can be disabled by calling
`self.skipTest("reason") <https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures>`_ in a test method.
The example below demonstrates how to disable test_main_ping2:
.. code-block:: python
import unittest
from telemetry_harness.testcase import TelemetryTestCase
class TestMainPingExample(TelemetryTestCase):
"""Example tests for the telemetry main ping."""
def test_main_ping1(self):
"""Example test that we want to run."""
self.search_in_new_tab("mozilla firefox")
def test_main_ping2(self):
"""Example test that we want to skip."""
self.skipTest("demonstrating skipping")
self.search_in_new_tab("firefox telemetry")
Who to contact for help
-----------------------
- The test cases are owned by Chris Hutten-Czapski (chutten on matrix) from the Firefox Telemetry team
(`#telemetry <https://chat.mozilla.org/#/room/#telemetry:mozilla.org>`__ on matrix)
- The test harness is owned by Raphael Pierzina (raphael on matrix) from the Ecosystem Test Engineering team
(`#telemetry <https://chat.mozilla.org/#/room/#telemetry:mozilla.org>`__ on matrix)
Bugzilla
--------
Bugs can be filed under the Toolkit product for the Telemetry component.
|
