summaryrefslogtreecommitdiffstats
path: root/toolkit/components/telemetry/docs/internals/integration_tests/index.rst
diff options
context:
space:
mode:
Diffstat (limited to 'toolkit/components/telemetry/docs/internals/integration_tests/index.rst')
-rw-r--r--toolkit/components/telemetry/docs/internals/integration_tests/index.rst143
1 files changed, 143 insertions, 0 deletions
diff --git a/toolkit/components/telemetry/docs/internals/integration_tests/index.rst b/toolkit/components/telemetry/docs/internals/integration_tests/index.rst
new file mode 100644
index 0000000000..389fa8b80e
--- /dev/null
+++ b/toolkit/components/telemetry/docs/internals/integration_tests/index.rst
@@ -0,0 +1,143 @@
+=================
+Integration Tests
+=================
+
+The aim of the telemetry-tests-client suite is to verify Firefox collects telemetry probes, aggregates that data, and submits telemetry
+pings containing the data to a HTTP server. The integration tests try to make no assumptions about the internal workings of Firefox and
+use automation to mimic user behavior.
+
+The integration test suite for Firefox Client Telemetry runs on CI `tier 1 <https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy>`_
+with treeherder symbol `tt(c)`
+and is checked in to version control at mozilla-central under
+`toolkit/components/telemetry/tests/marionette/tests/client <https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/marionette/tests/client/>`_.
+
+Test Main Tab Scalars
+---------------------
+
+- PATH: ``telemetry/tests/marionette/tests/client/test_main_tab_scalars.py``
+- This test opens and closes a number of browser tabs,
+ restarts the browser in a new session
+ and then verifies the correctness of scalar data in the resulting `main` ping.
+
+Test Search Counts
+------------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_search_counts_across_subsessions.py``
+- This test performs a search in a new tab,
+ restarts Firefox in a new session and verifies the correctness of client, session and subsession IDs,
+ as well as scalar and keyed histogram data in the `shutdown` ping,
+ installs an addon, verifies the `environment-change` ping, and performs three additional search actions
+ before restarting and verifying the new `main` ping.
+
+
+Test Deletion Request Ping
+--------------------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_deletion_request_ping.py``
+- This test installs an addon and verifies a ping is received. The test takes note of the client ID.
+ It then disables telemetry and checks for a `deletion-request` ping.
+ After it receives the correct ping it makes sure that no other pings are sent.
+ Telemetry is then re-enabled and the `main` ping is checked to see if the client ID has changed.
+ The test asserts that the user has opted back in to telemetry.
+
+Test Event Ping
+---------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_event_ping.py``
+- This test checks for a basic `event` ping. It opens firefox, performs a search and checks the `event`
+ ping for the correct number of searches performed (1) and the correct search engine.
+
+Test Fog Custom Ping
+--------------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_custom_ping.py``
+- This test creates a custom ping using the Glean API and asserts this ping is sent correctly.
+
+Test Fog Deletion Request Ping
+------------------------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_deletion_request_ping.py``
+- This test opens the browser, performs a search and disables telemetry after the search.
+ It asserts that the telemetry is disabled and no pings exist.
+ The browser is restarted and telemetry is then re-enabled.
+ Then we set a `debug tag <https://mozilla.github.io/glean/book/user/debugging/debug-ping-view.html>`_
+ which is attached to the ping.
+ Telemetry is then disabled again to trigger a `deletion-request` ping.
+ We verify that 1) The debug tag is present; and 2) that the client ID
+ in the second `deletion-request` ping is different from the first client ID.
+
+Test Fog User Activity
+----------------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_user_activity.py``
+- This test checks that a `baseline` ping is sent when the user starts or stops using Firefox.
+
+Test Background Update Ping
+---------------------------
+
+- PATH: ``toolkit/telemetry/tests/marionette/tests/client/test_fog_user_activity.py``
+- In this test we launch Firefox to prepare a profile and to disable the background update setting.
+ We exit Firefox,
+ leaving the (unlocked) profile to be used as the default profile for the background update task (and not having multiple instances running).
+ The task will not try to update, but it will send a ping.
+ Then we restart Firefox to unwind the background update setting and allow shutdown to proceed cleanly.
+
+Running the tests locally
+-------------------------
+
+You can run the tests on your local machine using
+`mach <https://firefox-source-docs.mozilla.org/mach/index.html>`__:
+
+``./mach telemetry-tests-client``
+
+Running the tests on try
+------------------------
+
+You can run the tests across all platforms on the try server using
+`mach <https://firefox-source-docs.mozilla.org/mach/index.html>`__:
+
+``./mach try fuzzy -q "'telemetry-tests-client"``
+
+Disabling an individual failing test
+------------------------------------
+
+The telemetry-tests-client suite is implemented in Python and uses Marionette for browser automation and wptserve for the HTTP ping server.
+The integration tests are based on Python's unittest testing library and can be disabled by calling
+`self.skipTest("reason") <https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures>`_ in a test method.
+
+The example below demonstrates how to disable test_main_ping2:
+
+.. code-block:: python
+
+ import unittest
+
+ from telemetry_harness.testcase import TelemetryTestCase
+
+ class TestMainPingExample(TelemetryTestCase):
+ """Example tests for the telemetry main ping."""
+
+ def test_main_ping1(self):
+ """Example test that we want to run."""
+
+ self.search_in_new_tab("mozilla firefox")
+
+ def test_main_ping2(self):
+ """Example test that we want to skip."""
+
+ self.skipTest("demonstrating skipping")
+
+ self.search_in_new_tab("firefox telemetry")
+
+
+Who to contact for help
+-----------------------
+
+- The test cases are owned by Chris Hutten-Czapski (chutten on matrix) from the Firefox Telemetry team
+ (`#telemetry <https://chat.mozilla.org/#/room/#telemetry:mozilla.org>`__ on matrix)
+- The test harness is owned by Raphael Pierzina (raphael on matrix) from the Ecosystem Test Engineering team
+ (`#telemetry <https://chat.mozilla.org/#/room/#telemetry:mozilla.org>`__ on matrix)
+
+Bugzilla
+--------
+
+Bugs can be filed under the Toolkit product for the Telemetry component.