diff options
Diffstat (limited to 'testing/web-platform/tests/tools/wave/docs')
58 files changed, 2728 insertions, 0 deletions
diff --git a/testing/web-platform/tests/tools/wave/docs/README.md b/testing/web-platform/tests/tools/wave/docs/README.md new file mode 100644 index 0000000000..88092b63e7 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/README.md @@ -0,0 +1,14 @@ +# WAVE Test Runner Documentation + +As part of the [WAVE project](https://cta.tech/Resources/Standards/WAVE-Project) +the WAVE Test Runner was implemented to run tests that confirm proper implementation +of specified features. The code base is used in different subprojects, each of which +may have different requirements and scopes, so some features and screenshots in +this documentation may use specific contexts. + +## Contents + +- [Configuration](./config.md): How to configure the test runner +- [REST API](./rest-api/README.md): Documentation of endpoints, parameters and payloads + - [Guides](./rest-api/guides/README.md): How to use certain API mechanisms +- [Usage Guide](./usage/usage.md): General usage guide diff --git a/testing/web-platform/tests/tools/wave/docs/config.md b/testing/web-platform/tests/tools/wave/docs/config.md new file mode 100644 index 0000000000..afc7c0b05f --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/config.md @@ -0,0 +1,326 @@ +# Configuration - [WAVE Test Runner](./README.md) + +Using a configuration file, the WAVE Test Runner can be configured to be more +functional in different use cases. This document lists all configuration +parameters and what they are used for. + +## Contents + +1. [Location and structure](#1-location-and-structure) +2. [Parameters](#2-parameters) + 1. [Results directory](#21-results-directory) + 2. [Test Timeouts](#22-test-timeouts) + 3. [Enable import of results](#23-enable-import-of-results) + 4. [Web namespace](#24-web-namespace) + 5. [Persisting interval](#25-persisting-interval) + 6. [API titles](#26-api-titles) + 7. [Enable listing all sessions](#27-enable-listing-all-sessions) + 8. [Event caching duration](#28-event-caching-duration) + 9. [Enable test type selection](#29-enable-test-type-selection) + +## 1. Location and structure + +Configuration parameters are defined in a JSON file called `config.json` in +the project root of the WPT runner. This configuration file is also used by +the WPT runner, so any WAVE Test Runner related configuration parameters are +wrapped inside a `wave` object. + +``` +<PRJ_ROOT>/config.json +``` + +```json +{ + "wave": { + "results": "./results" + } +} +``` + +All the default values are stored in a configuration file inside the wave +directory: + +``` +<PRJ_ROOT>/tools/wave/config.default.json +``` + +```json +{ + "wave": { + "results": "./results", + "timeouts": { + "automatic": 60000, + "manual": 300000 + }, + "enable_import_results": false, + "web_root": "/_wave", + "persisting_interval": 20, + "api_titles": [], + "enable_read_sessions": false, + "event_cache_duration": 60000 + } +} +``` +[🠑 top](#configuration---wave-test-runner) + +## 2. Parameters + +### 2.1 Results directory + +The results parameter sets where results and session information are stored. + +**Parameters**: + +```json +{ + "results": "<String>" +} +``` + +- **results**: Path to the results directory. Can be absolute, or relative to + the project root. + +**Default**: + +```json +{ + "results": "./results" +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.2 Test Timeouts + +The test timeouts set the default test timeout for different test types. + +**Parameters**: + +```json +{ + "timeouts": { + "automatic": "<Number>", + "manual": "<Number>" + } +} +``` + +- **timeouts**: Holds the key value pairs for different types of tests + - **automatic**: Default time to wait for automatic tests in milliseconds. + - **manual**: Default time to wait for manual tests in milliseconds. + +**Default**: + +```json +{ + "timeouts": { + "automatic": 600000, + "manual": 300000 + } +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.3 Enable import of results + +This parameter enables the capability to import session results from other +WAVE Test Runner instances into the current one. + +**Parameters**: + +```json +{ + "enable_import_results": "<Boolean>" +} +``` + +- **enable_import_results**: Sets whether or not to enable the [REST API endpoint to import results](./rest-api/results-api/import.md) + +**Default**: + +```json +{ + "enable_import_results": "false" +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.4 Web namespace + +All static resources and REST API endpoints are accessible under a +configurable namespace. This namespace can be set using the `web_root` +parameter. + +**Parameters**: + +```json +{ + "web_root": "<String>" +} +``` + +- **web_root**: The namespace to use + +**Default**: + +```json +{ + "web_root": "/_wave" +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.5 Persisting interval + +The persisting interval specifies how many tests have to be completed until +all session information is updated in the results directory. + +For example, if set to 5, then every 5 completed tests the `info.json` in the +results directory is updated with the current state of the session. When +restarting the server, this state is used to reconstruct all sessions testing +state. + +**Parameters**: + +```json +{ + "persisting_interval": "<Number>" +} +``` + +- **persisting_interval**: The number of tests to execute until the persisted + session information gets updated + +**Default**: + +```json +{ + "persisting_interval": 20 +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.6 API titles + +The API titles are used to display a more human readible representation of an +API that tests are available for. Using the parameter it is possible to assign +a name to an API subdirectory. + +**Parameters**: + +```json +{ + "api_titles": [ + { + "title": "<String>", + "path": "<String>" + }, + ... + ] +} +``` + +- **api_titles**: An array of titles assigned to paths + - **title**: The displayed title of the API in the UI + - **path**: The path relative to the project root of the tested API + +**Default**: + +```json +{ + "api_titles": [] +} +``` + +**Example**: + +```json +{ + "api_titles": [ + { + "title": "WebGL", + "path": "/webgl" + }, + { + "title": "WebRTC Extensions", + "path": "/webrtc-extensions" + } + ] +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.7 Enable listing all sessions + +This parameter enables the [REST API endpoint to list all available sessions](./rest-api/sessions-api/read_sessions.md). + +**Parameters**: + +```json +{ + "enable_read_sessions": "<Boolean>" +} +``` + +- **enable_import_results**: Sets whether or not to enable the REST API endpoint read all sessions + +**Default**: + +```json +{ + "enable_read_sessions": "false" +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.8 Event caching duration + +This parameters specifies how long events are hold in the cache. Depending on +how fast clients are able to evaluate events, this value may be changed +accordingly. + +**Parameters**: + +```json +{ + "event_cache_duration": "<Number>" +} +``` + +- **event_cache_duration**: The duration events are hold in the cache in milliseconds + +**Default**: + +```json +{ + "event_cache_duration": 60000 +} +``` + +[🠑 top](#configuration---wave-test-runner) + +### 2.9 Enable test type selection + +Sets display of test type configuration UI elements. + +**Parameters**: + +```json +{ + "enable_test_type_selection": "<Boolean>" +} +``` + +- **enable_test_type_selection**: Whether or not test type UI controls are displayed + +**Default**: + +False + +[🠑 top](#configuration---wave-test-runner) diff --git a/testing/web-platform/tests/tools/wave/docs/res/configuration_page_bottom.jpg b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_bottom.jpg Binary files differnew file mode 100644 index 0000000000..85d4bdc3fc --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_bottom.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_malfunctioning.jpg b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_malfunctioning.jpg Binary files differnew file mode 100644 index 0000000000..28f42230d5 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_malfunctioning.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_prev_excluded.jpg b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_prev_excluded.jpg Binary files differnew file mode 100644 index 0000000000..c71b34f668 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_prev_excluded.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_raw.jpg b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_raw.jpg Binary files differnew file mode 100644 index 0000000000..f1428f313d --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_exclude_add_raw.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/configuration_page_top.jpg b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_top.jpg Binary files differnew file mode 100644 index 0000000000..93b6522bc7 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/configuration_page_top.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/landing_page.jpg b/testing/web-platform/tests/tools/wave/docs/res/landing_page.jpg Binary files differnew file mode 100644 index 0000000000..c032a7b291 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/landing_page.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions.jpg b/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions.jpg Binary files differnew file mode 100644 index 0000000000..642a009d4a --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions_filtered.jpg b/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions_filtered.jpg Binary files differnew file mode 100644 index 0000000000..266a5c1159 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions_filtered.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions_pinned_recent.jpg b/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions_pinned_recent.jpg Binary files differnew file mode 100644 index 0000000000..fb92a47bf4 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/overview_page_sessions_pinned_recent.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/overview_page_top.jpg b/testing/web-platform/tests/tools/wave/docs/res/overview_page_top.jpg Binary files differnew file mode 100644 index 0000000000..860272796a --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/overview_page_top.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/results_page_api_results.jpg b/testing/web-platform/tests/tools/wave/docs/res/results_page_api_results.jpg Binary files differnew file mode 100644 index 0000000000..e0e2314a78 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/results_page_api_results.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/results_page_api_results_export.jpg b/testing/web-platform/tests/tools/wave/docs/res/results_page_api_results_export.jpg Binary files differnew file mode 100644 index 0000000000..c85ce980fe --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/results_page_api_results_export.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/results_page_bottom.jpg b/testing/web-platform/tests/tools/wave/docs/res/results_page_bottom.jpg Binary files differnew file mode 100644 index 0000000000..60244a8e46 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/results_page_bottom.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/results_page_last_timed_out.jpg b/testing/web-platform/tests/tools/wave/docs/res/results_page_last_timed_out.jpg Binary files differnew file mode 100644 index 0000000000..bb63bd4d51 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/results_page_last_timed_out.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/results_page_malfunctioning_list.jpg b/testing/web-platform/tests/tools/wave/docs/res/results_page_malfunctioning_list.jpg Binary files differnew file mode 100644 index 0000000000..fcd6b370fd --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/results_page_malfunctioning_list.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/res/results_page_top.jpg b/testing/web-platform/tests/tools/wave/docs/res/results_page_top.jpg Binary files differnew file mode 100644 index 0000000000..3d0f876c7b --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/res/results_page_top.jpg diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/README.md b/testing/web-platform/tests/tools/wave/docs/rest-api/README.md new file mode 100644 index 0000000000..c6d21823a7 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/README.md @@ -0,0 +1,76 @@ +# REST API - [WAVE Test Runner](../README.md) + +The REST API allows the WAVE server to be integrated into other systems. Every +call must be preceded with a namespace or web root, which is omitted in this +documentation. The default web root is `/_wave`, which can be changed in the +config.json using the keyword `web_root`. + +Additional [REST API Guides](./guides/README.md) can help to understand how to +use these endpoints in context. + +## Sessions API <a name="sessions-api"></a> + +| Name | Description | +| -------------------------------------------------- | -------------------------------------------------------------- | +| [`create`](./sessions-api/create.md) | Creates a new test session. | +| [`read session`](./sessions-api/read.md) | Reads a sessions configuration. | +| [`read sessions`](./sessions-api/read_sessions.md) | Reads all session tokens, expandable with configs and statuses | +| [`read public`](./sessions-api/read-public.md) | Reads all public sessions tokens. | +| [`update`](./sessions-api/update.md) | Updates a session configuration. | +| [`delete`](./sessions-api/delete.md) | Deletes a test session. | +| [`status`](./sessions-api/status.md) | Reads the status and progress of a session. | +| [`start`](./sessions-api/control.md#start) | Starts a test session. | +| [`stop`](./sessions-api/control.md#stop) | Stops a test session. | +| [`pause`](./sessions-api/control.md#pause) | Pauses a test session. | +| [`find`](./sessions-api/find.md) | Finds a session token by providing a token fragment. | +| [`labels`](./sessions-api/labels.md) | Attach labels to sessions for organization purposes. | +| [`listen events`](./sessions-api/events.md) | Register for sessions specific events. | +| [`push events`](./sessions-api/events.md) | Push session specific events. | + +## Tests API <a name="tests-api"></a> + +| Name | Description | +| --------------------------------------------------------------- | ------------------------------------------------------ | +| [`read all`](./tests-api/read-all.md) | Reads all tests available. | +| [`read session`](./tests-api/read-session.md) | Reads all tests that are part of a session. | +| [`read next`](./tests-api/read-next.md) | Reads the next test to run in a session. | +| [`read last completed`](./tests-api/read-last-completed.md) | Reads the last completed tests of a session. | +| [`read malfunctioning`](./tests-api/read-malfunctioning.md) | Reads the list of malfunctioning tests of a session. | +| [`update malfunctioning`](./tests-api/update-malfunctioning.md) | Updates the list of malfunctioning tests of a session. | +| [`read available apis`](./tests-api/read-available-apis.md) | Reads all available APIs names and paths. | + +## Results API <a name="results-api"></a> + +| Name | Description | +| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | +| [`create`](./results-api/create.md) | Create a new test result for a test in a session. | +| [`read`](./results-api/read.md) | Read all test results of a session. | +| [`read compact`](./results-api/read-compact.md) | Read the number of passed, failed, timed out and not run tests of a session. | +| [`import session`](./results-api/import.md#1-import-session) | Import session results. | +| [`import api results`](./results-api/import.md#2-import-api-results) | Import results of a specific API into existing session. | +| [`download`](./results-api/download.md#1-download) | Download all session results to import into other WMATS instance. | +| [`download api`](./results-api/download.md#2-download-api) | Download all results of an API. | +| [`download all apis`](./results-api/download.md#3-download-all-apis) | Download all results of all APIs. | +| [`view report`](./results-api/download.md#4-download-report) | View the WPT report of an API of a session. | +| [`view multi report`](./results-api/download.md#5-download-multi-report) | View the WPT report of an API of multiple sessions. | +| [`download overview`](./results-api/download.md#6-download-overview) | Download an overview of results of all APIs of a session. | +| [`view report`](./results-api/view.md#1-view-report) | Read an url to a hosted version of a WPT report for an API of a session. | +| [`view multi report`](./results-api/view.md#2-view-multi-report) | Read an url to a hosted version of a WPT report for an API of multiple session. | + +## Devices API <a name="devices-api"></a> + +| Name | Description | +| -------------------------------------------------------------------- | -------------------------------------- | +| [`create`](./devices-api/create.md) | Registers a new device. | +| [`read device`](./devices-api/read-device.md) | Read information of a specific device. | +| [`read devices`](./devices-api/read-devices.md) | Read a list of all available devices. | +| [`register event listener`](./devices-api/register.md) | Register for a device specific event. | +| [`send event`](./devices-api/send-event.md) | Sends a device specific event. | +| [`register global event listener`](./devices-api/register-global.md) | Register for a global device event. | +| [`send global event`](./devices-api/send-global-event.md) | Sends a global device event. | + +## General API <a name="general-api"></a> + +| Name | Description | +| ----------------------------------- | ---------------------------------------------------- | +| [`status`](./general-api/status.md) | Returns information on how the server is configured. | diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/create.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/create.md new file mode 100644 index 0000000000..5ed1355ad2 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/create.md @@ -0,0 +1,33 @@ +# `create` - [Devices API](../README.md#devices-api) + +The `create` method of the devices API registers a new devices to remotely +start test sessions on. The device will automatically be unregistered if its +not registering for an [event](./register.md) for more than a minute. + +## HTTP Request + +`POST /api/devices` + +## Response Payload + +```json +{ + "token": "<String>" +} +``` + +- **token** specifies the handle to reference the registered device by. + +## Example + +**Request:** + +`POST /api/devices` + +**Response:** + +```json +{ + "token": "e5f0b92e-8309-11ea-a1b1-0021ccd76152" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/event-types.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/event-types.md new file mode 100644 index 0000000000..ead1f0695b --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/event-types.md @@ -0,0 +1,37 @@ +# Device Event Types - [Devices API](../README.md#devices-api) + +Device events are events that are triggered by actions related to devices. +This document specifies what possible events can occur and what they mean. + +## Device specific <a name="device-specific"></a> + +Device specific events are always related to a specific device, referenced by +its token. + +### Start session + +**Type identifier**: `start_session` +**Payload**: +```json +{ + "session_token": "<String>" +} +``` +**Description**: Triggered by a companion device, this event starts a +pre-configured session on the registered device. + +## Global <a name="global"></a> + +Global device events have no special relation to any device. + +### Device added + +**Type identifier**: `device_added` +**Payload**: Same as response of [`read device`](./read-device.md) method. +**Description**: This event is triggered once a new device registers. + +### Device removed + +**Type identifier**: `device_removed` +**Payload**: Same as response of [`read device`](./read-device.md) method. +**Description**: This event is triggered once a device unregisters. diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/read-device.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/read-device.md new file mode 100644 index 0000000000..9762b17e71 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/read-device.md @@ -0,0 +1,41 @@ +# `read device` - [Devices API](../README.md#devices-api) + +The `read device` method of the devices API fetches available information regarding a +specific device. + +## HTTP Request + +`GET /api/devices/<device_token>` + +## Response Payload + +```json +{ + "token": "<String>", + "user_agent": "<String>", + "last_active": "<String>", + "name": "<String>" +} +``` + +- **token** is the unique identifier of the device. +- **user_agent** is the user agent of the request the device was registered with. +- **last_active** defines the point in time the device was last active. Expressed as ISO 8601 date and time format. +- **name** the name the device was assign based on its user agent. + +## Example + +**Request:** + +`GET /api/devices/1d9f5d30-830f-11ea-8dcb-0021ccd76152` + +**Response:** + +```json +{ + "token": "1d9f5d30-830f-11ea-8dcb-0021ccd76152", + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36", + "last_active": 1587391153295, + "name": "Chrome 81.0.4044" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/read-devices.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/read-devices.md new file mode 100644 index 0000000000..519b06c610 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/read-devices.md @@ -0,0 +1,47 @@ +# `read devices` - [Devices API](../README.md#devices-api) + +The `read devices` method of the devices API fetches a list of all registered +devices. + +## HTTP Request + +`GET /api/devices` + +## Response Payload + +```json +[ + { + "token": "<String>", + "user_agent": "<String>", + "last_active": "<String>", + "name": "<String>" + }, + ... +] +``` + +- **token** is the unique identifier of the device. +- **user_agent** is the user agent of the request the device was registered with. +- **last_active** defines the point in time the device was last active. Expressed as ISO 8601 date and time format. +- **name** the name the device was assign based on its user agent. + +## Example + +**Request:** + +`GET /api/devices` + +**Response:** + +```json +[ + { + "token": "1d9f5d30-830f-11ea-8dcb-0021ccd76152", + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36", + "last_active": 1587391153295, + "name": "Chrome 81.0.4044" + }, + ... +] +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/register-global.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/register-global.md new file mode 100644 index 0000000000..fd3a2ad998 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/register-global.md @@ -0,0 +1,54 @@ +# `register global event listener` - [Devices API](../README.md#devices-api) + +The `register global event listener` method of the devices API notifies a +registered listener upon global device events. It uses HTTP long polling in +send the event to this listener in real time, so upon receiving an event, the +connection has to be reestablished by the client to receive further events. + +## HTTP Request + +`GET /api/devices/events` + +## Query Parameters + +| Parameter | Desciption | Example | +| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | +| `device_token` | The token of the device which performed the request. (Optional) Lets the server know the registered device is still active. | `device_token=7dafeec0-c351-11e9-84c5-3d1ede2e7d2e` | + +## Response Payload + +```json +{ + "type": "<String>", + "data": "<Any>" +} +``` + +- **type** defines what type of event has been triggered. +- **data** contains the event specific payload. + +## Event Types + +See [global events](./event-types.md#global) + +## Example + +**Request:** + +`GET /api/devices/events` + +**Response:** + +```json +{ + "type": "device_added", + "data": { + "token": "1d9f5d30-830f-11ea-8dcb-0021ccd76152", + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36", + "last_active": 1587391153295, + "name": "Chrome 81.0.4044" + } +} +``` + + diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/register.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/register.md new file mode 100644 index 0000000000..adee3b4742 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/register.md @@ -0,0 +1,52 @@ +# `register event listener` - [Devices API](../README.md#devices-api) + +The `register event listener` method of the devices API notifies a registered +listener upon device specific events. It uses HTTP long polling in send the +event to this listener in real time, so upon receiving an event, the +connection has to be reestablished by the client to receive further events. + +## HTTP Request + +`GET /api/devices/<device_token>/events` + +## Query Parameters + +| Parameter | Desciption | Example | +| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | +| `device_token` | The token of the device which performed the request. (Optional) Lets the server know the registered device is still active. | `device_token=7dafeec0-c351-11e9-84c5-3d1ede2e7d2e` | + +## Response Payload + +```json +{ + "type": "<String>", + "data": "<Any>" +} +``` + +- **type** defines what type of event has been triggered. +- **data** contains the event specific payload. + +## Event Types + +### Start session + +See [device specific events](./event-types.md#device-specific) + +## Example + +**Request:** + +`GET /api/devices/1d9f5d30-830f-11ea-8dcb-0021ccd76152/events` + +**Response:** + +```json +{ + "type": "start_session", + "data": { + "session_token": "974c84e0-c35d-11e9-8f8d-47bb5bb0037d" + } +} +``` + diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/send-event.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/send-event.md new file mode 100644 index 0000000000..0ec74aec5c --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/send-event.md @@ -0,0 +1,43 @@ +# `send event` - [Devices API](../README.md#devices-api) + +The `send event` method of the devices API enables sending an event to +listeners of specific devices events. + +## HTTP Request + +`POST /api/devices/<device_token>/events` + +## Request Payload + +```json +{ + "type": "<String>", + "data": "<Any>" +} +``` + +- **type** defines what type of event has been triggered. +- **data** contains the event specific payload. + +## Event Types + +See [device specific events](./event-types.md#device-specific) + +## Example + +**Request:** + +`POST /api/devices/1d9f5d30-830f-11ea-8dcb-0021ccd76152/events` + +```json +{ + "type": "start_session", + "data": { + "session_token": "974c84e0-c35d-11e9-8f8d-47bb5bb0037d" + } +} +``` + +**Response:** + +`200 OK` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/send-global-event.md b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/send-global-event.md new file mode 100644 index 0000000000..aa5238dc7f --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/devices-api/send-global-event.md @@ -0,0 +1,46 @@ +# `send global event` - [Devices API](../README.md#devices-api) + +The `send global event` method of the devices API enables sending an event to +listeners of global device events. + +## HTTP Request + +`POST /api/devices/events` + +## Request Payload + +```json +{ + "type": "<String>", + "data": "<Any>" +} +``` + +- **type** defines what type of event has been triggered. +- **data** contains the event specific payload. + +## Event Types + +See [global events](./event-types.md#global) + +## Example + +**Request:** + +`POST /api/devices/1d9f5d30-830f-11ea-8dcb-0021ccd76152/events` + +```json +{ + "type": "device_added", + "data": { + "token": "1d9f5d30-830f-11ea-8dcb-0021ccd76152", + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36", + "last_active": 1587391153295, + "name": "Chrome 81.0.4044" + } +} +``` + +**Response:** + +`200 OK` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/general-api/status.md b/testing/web-platform/tests/tools/wave/docs/rest-api/general-api/status.md new file mode 100644 index 0000000000..8af510066a --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/general-api/status.md @@ -0,0 +1,41 @@ +# `status` - [General API](../README.md#general-api) + +The `status` method is used to ensure the server is reachable and to determine +what features of different server APIs are enabled. + +## HTTP Request + +``` +GET /api/status +``` + +### Response + +```json +{ + "version_string": "String", + "import_results_enabled": "Boolean", + "reports_enabled": "Boolean", + "read_sessions_enabled": "Boolean" +} +``` + +- **version_string**: The version of the server. +- **import_results_enabled**: If true the [`import result`](../results-api/import.md) endpoint is available +- **reports_enabled**: If true the server will generate reports for completed APIs in a given test session. +- **read_sessions_enabled**: If true it is possible to list all sessions using the [`read sessions`](../sessions-api/read_sessions.md) endpoint of the sessions API + +## Example + +``` +GET /api/status +``` + +```json +{ + "version_string": "v2.0.0", + "import_results_enabled": false, + "reports_enabled": true, + "read_sessions_enabled": false +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/guides/README.md b/testing/web-platform/tests/tools/wave/docs/rest-api/guides/README.md new file mode 100644 index 0000000000..c331bb8e66 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/guides/README.md @@ -0,0 +1,10 @@ +# REST API Guides - [WAVE Test Runner](../../README.md) + +In addition to the [REST API documentation](../README.md), these guide shall +provide a better understanding on how to properly use the different endpoints. + +[Starting sessions on a DUT using the devices API](./session-start-devices-api.md): +How to register a DUT and start a pre-configured session on it. + +[Sending and receiving session events](./session-events.md): +How to register for session events and push events to other listeners. diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/guides/session-events.md b/testing/web-platform/tests/tools/wave/docs/rest-api/guides/session-events.md new file mode 100644 index 0000000000..462737c652 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/guides/session-events.md @@ -0,0 +1,52 @@ +# Sending and receiving session events + +The session event endpoints allow to listen for events related to a specific +session and to send new events to all registered listeners. + +See all [REST API Guides](./README.md). + +## Register for session specific events + +To receive events of a session, simply perform a GET request to the desired +sessions event endpoint. For example, if we want to receive any events that +are related to the session with token `6fdbd1a0-c339-11e9-b775-6d49dd567772`: + +``` +GET /_wave/api/sessions/6fdbd1a0-c339-11e9-b775-6d49dd567772/events +``` + +```json +{ + "type": "status", + "data": "paused" +} +``` + +As this endpoint makes use of the HTTP long polling, you will not immediately +receive a response. The connection stays open either until an event gets +triggered, in which case the server will respond with that events data, or +there is no event within the timeout, which will return an empty response. + +With one request only one event can be received. To get any further events, +additional requests are necessary. To not miss any events, it is important to +perform the next request immediately after receiving a response. + +## Sending events + +To create a new event, simply send a POST request containing the event data to +the desired sessions event endpoint. For example, if you want to trigger a new +event for a session with token `6fdbd1a0-c339-11e9-b775-6d49dd567772`: + +``` +POST /_wave/api/sessions/6fdbd1a0-c339-11e9-b775-6d49dd567772/events +``` + +```json +{ + "type": "status", + "data": "paused" +} +``` + +This will cause any client, that currently has a connection open as described +in the preceding section, to receive the specified event. diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/guides/session-start-devices-api.md b/testing/web-platform/tests/tools/wave/docs/rest-api/guides/session-start-devices-api.md new file mode 100644 index 0000000000..a376d31617 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/guides/session-start-devices-api.md @@ -0,0 +1,60 @@ +# Starting sessions on a DUT using the devices API + +See all [REST API Guides](./README.md). + +## Connecting the DUT + +To start a session on a DUT using the devices API, first register the DUT at +the test runner. + +``` +POST /api/devices +``` + +```json +{ + "token": "fa3fb226-98ef-11ea-a21d-0021ccd76152" +} +``` + +Using the device token, you can listen for any events related to the device. + +``` +GET /api/devices/fa3fb226-98ef-11ea-a21d-0021ccd76152/events +``` + +Once an event occurs, the response to this call will contain the event data. +If no event occurs until the request times out, you have to perfom another call. + +```json +{ + "type": "start_session", + "data": { + "session_token": "98ed4b8e-98ed-11ea-9de7-0021ccd76152" + } +} +``` + +Using this data you can start the session and get the URL to the next test to +open. + +## Triggering the session start + +Once a device is registered and waits for events, you can use the device's +event channel to push an event to start a session on it. + +``` +POST /api/devices/fa3fb226-98ef-11ea-a21d-0021ccd76152/events +``` + +```json +{ + "type": "start_session", + "data": { + "session_token": "98ed4b8e-98ed-11ea-9de7-0021ccd76152" + } +} +``` + +The session related to the provided token can be a newly created one or may +already be running. diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/config.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/config.md new file mode 100644 index 0000000000..a60485dadc --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/config.md @@ -0,0 +1,34 @@ +# `config` - [Results API](../README.md#results-api) + +The `config` method is used to determine what features of the results API are +enabled. Features that can be enabled or disabled are the +[`import`](./import.md) method and the generation of reports and therefore +[`download and view`](./download.md) methods. + +## HTTP Request + +`GET /api/results/config` + +## Response + +```json +{ + "import_enabled": "Boolean", + "reports_enabled": "Boolean" +} +``` + +## Example + +**Request:** + +`GET /api/results/config` + +**Response:** + +```json +{ + "import_enabled": false, + "reports_enabled": true +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/create.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/create.md new file mode 100644 index 0000000000..5839702eda --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/create.md @@ -0,0 +1,65 @@ +# `create` - [Results API](../README.md#results-api) + +The `create` method of the results API creates a test result for a given test of a test session. + +## HTTP Request + +`POST /api/results/<session_token>` + +## Request Payload + +```json +{ + "test": "String", + "status": "Enum['OK', 'ERROR', 'TIMEOUT', 'NOT_RUN']", + "message": "String", + "subtests": [ + { + "name": "String", + "status": "Enum['PASS', 'FAIL', 'TIMEOUT', 'NOT_RUN']", + "message": "String" + } + ] +} +``` + +- **test** specifies the test to create the result for. +- **status** specifies the overall status of the test. It does not represent a result, but rather if the contained tests were executed as intended or if something went wrong running the test. + - **OK**: All tests were executed without problems. + - **ERROR**: There was an error running one or multiple tests. + - **TIMEOUT**: It took too long for the tests to execute. + - **NOT_RUN**: This test was skipped. +- **message** contains the reason for the overall status. If the status is `OK` the message should be `null`. +- **subtests** contains the actual results of the tests executed in this file. + - **name**: The name of the test. + - **status**: The status of the result: + - **PASS**: The test was executed successfully. + - **FAIL**: The test did not meet at least one assertion. + - **TIMEOUT**: It took too long for this test to execute. + - **NOT_RUN**: This test was skipped. + - **message** contains the reason for the tests failure. + +## Example + +**Request:** + +`POST /api/results/d89bcc00-c35b-11e9-8bb7-9e3d7595d40c` + +```json +{ + "test": "/apiOne/test/one.html", + "status": "OK", + "message": null, + "subtests": [ + { + "name": "Value should be X", + "status": "FAIL", + "message": "Expected value to be X but got Y" + } + ] +} +``` + +**Response:** + +`200 OK`
\ No newline at end of file diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/download.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/download.md new file mode 100644 index 0000000000..8cfa7e7531 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/download.md @@ -0,0 +1,127 @@ +# Downloading and viewing results and reports - [Results API](../README.md#results-api) + +There are multiple methods to download or view reports generated by the WPT +Report tool or just the plain json results with the structure as described in +the [`create`](./create.md) method of the results API. + +## 1. `download` + +Downloads all results of a session as ZIP, which other instances of the WMAS +Test Runner can import. + +### HTTP Request + +`GET /api/results/<session_token>/export` + +### Example + +`GET /api/results/f63700a0-c35f-11e9-af33-9e0d4c1f1370/export` + +## 2. `download api` + +Downloads all results of a single API in one json file. + +### HTTP Request + +`GET /api/results/<session_token>/<api_name>/json` + +### File Structure + +```json +{ + "results": [ + { + "test": "String", + "status": "Enum['OK', 'ERROR', 'TIMEOUT', 'NOT_RUN']", + "message": "String", + "subtests": [ + { + "name": "String", + "status": "Enum['PASS', 'FAIL', 'TIMEOUT', 'NOT_RUN']", + "message": "String" + } + ] + } + ] +} +``` + +Results are structured as explained in the [`create`](./create.md) method of the results API. + +### Example + +`GET /api/results/f63700a0-c35f-11e9-af33-9e0d4c1f1370/apiOne/json` + +## 3. `download all apis` + +Downloads all results of all APIs of a session as zip file containing one json file per API. + +### HTTP Request + +`GET /api/results/<session_token>/json` + +### File Structure + +There is one json file per API, each structured as described in the [`download api`](#download-api) method. + +### Example + +`GET /api/results/f63700a0-c35f-11e9-af33-9e0d4c1f1370/json` + +## 4. `view report` + +Returns a URL to a report of an API of a session, generated by the WPT Report tool, which is a static HTML page. + +### HTTP Request + +`GET /api/results/<session_token>/<api_name>/reporturl` + +### Example + +`GET /api/results/f63700a0-c35f-11e9-af33-9e0d4c1f1370/apiOne/reporturl` + +**Response** + +```json +{ + "uri": "/results/8f7f2fdc-62eb-11ea-8615-b8ca3a7b18ad/2dcontext/all.html" +} +``` + +## 5. `view multi report` + +Returns a URL to a report of an API of multiple session, generated by the WPT Report tool, which is a static HTML page. + +### HTTP Request + +`GET /api/results/<api_name>/reporturl` + +### Query Parameters + +| Parameter | Description | Default | Example | +| --------- | ------------------------------------------------------------ | ------- | -------------------------------- | +| `tokens` | Comma separated list of tokens to create a multi report for. | none | `tokens=token_a,token_b,token_c` | + +### Example + +`GET /api/results/apiOne/reporturl?tokens=8f7f2fdc-62eb-11ea-8615-b8ca3a7b18ad,990b4734-62eb-11ea-a9a5-b8ca3a7b18ad` + +**Response** + +```json +{ + "uri": "/results/comparison-8f7f2fdc-990b473401488e04/reporturl/all.html" +} +``` + +## 6. `download overview` + +Downloads a zip file containing an overview for all APIs results of a session as a static HTML page. + +### HTTP Request + +`GET /api/results/<session_token>/overview` + +### Example + +`GET /api/results/f63700a0-c35f-11e9-af33-9e0d4c1f1370/overview` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/import.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/import.md new file mode 100644 index 0000000000..08c7de6ff7 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/import.md @@ -0,0 +1,66 @@ +# Import results - [Results API](../README.md#results-api) + +If enabled, the WMAS Test Runner can import results exported by any arbitrary other instance. + +## 1. Import session + +Upload results and create a new, finished session + +### HTTP Request + +``` +POST /api/results/import +``` + +#### HTTP Response + +If successful, the server responds with the token of the imported session: + +```json +{ + "token": "String" +} +``` + +However, if an error occured, the server responds the error message: + +```json +{ + "error": "String" +} +``` + +## 2. Import API results + +Upload a results json file and overwrite results of a specific API. + +### HTTP Request + +``` +POST /api/results/<session_token>/<api_name>/json +``` + +### File structure + +```json +{ + "results": [ + { + "test": "String", + "status": "Enum['OK', 'ERROR', 'TIMEOUT', 'NOT_RUN']", + "message": "String", + "subtests": [ + { + "name": "String", + "status": "Enum['PASS', 'FAIL', 'TIMEOUT', 'NOT_RUN']", + "message": "String" + } + ] + } + ] +} +``` + +### HTTP Response + +If successful, the server responds with status code 200. diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/read-compact.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/read-compact.md new file mode 100644 index 0000000000..55b6f41d4b --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/read-compact.md @@ -0,0 +1,59 @@ +# `read compact` - [Results API](../README.md#results-api) + +The `read compact` method of the results API returns the number of passed, failed, timed out and not run tests per API of a session. + +## HTTP Request + +`GET /api/results/<session_token>/compact` + +## Response Payload + +```json +{ + "<api_name>": { + "pass": "Integer", + "fail": "Integer", + "timeout": "Integer", + "not_run": "Integer", + "total": "Integer", + "complete": "Integer" + } +} +``` + +## Example + +**Request:** + +`GET /api/results/620bbf70-c35e-11e9-bf9c-742c02629054/compact` + +**Response:** + +```json +{ + "apiOne": { + "pass": 311, + "fail": 59, + "timeout": 23, + "not_run": 20, + "total": 481, + "complete": 413 + }, + "apiTwo": { + "pass": 548, + "fail": 129, + "timeout": 53, + "not_run": 36, + "total": 766, + "complete": 766 + }, + "apiThree": { + "pass": 349, + "fail": 45, + "timeout": 14, + "not_run": 9, + "total": 523, + "complete": 417 + } +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/read.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/read.md new file mode 100644 index 0000000000..66894e69ee --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/read.md @@ -0,0 +1,63 @@ +# `read` - [Results API](../README.md#results-api) + +The `read` method of the results API returns all available results of a session, grouped by API. It is possible to filter the results to return by test directory or file. + +## HTTP Request + +`GET /api/results/<session_token>` + +## Query Parameters + +| Parameter | Description | Default | Example | +| --------- | ------------------------------ | ------- | --------------------------- | +| `path` | Path of test directory or file | `/` | `path=/apiOne/test/sub/dir` | + +## Response Payload + +```json +{ + "<api_name>": [ + { + "test": "String", + "status": "Enum['OK', 'ERROR', 'TIMEOUT', 'NOT_RUN']", + "message": "String", + "subtests": [ + { + "name": "String", + "status": "Enum['PASS', 'FAIL', 'TIMEOUT', 'NOT_RUN']", + "message": "String" + } + ] + } + ] +} +``` + +Arrays of results grouped by their respective APIs. Structure of results is the same as described in the [`create`](./create.md) method of the results API. + +## Example + +**Request:** + +`GET /api/results/974c84e0-c35d-11e9-8f8d-47bb5bb0037d?path=/apiOne/test/one.html` + +**Response:** + +```json +{ + "apiOne": [ + { + "test": "/apiOne/test/one.html", + "status": "OK", + "message": null, + "subtests": [ + { + "name": "Value should be X", + "status": "FAIL", + "message": "Expected value to be X but got Y" + } + ] + } + ] +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/view.md b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/view.md new file mode 100644 index 0000000000..5b60d2ccf2 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/results-api/view.md @@ -0,0 +1,61 @@ +# Viewing Reports - [Results API](../README.md#results-api) + +It is possible to view the reports generated by the WPT Report tool directly in the browser using a version of the report that is hosted by the WAVE server. The methods listed here return urls to those hosted reports. + +## 1. `view report` + +Returns a URL to a report for an API of a single session, generated by the WPT Report tool. + +### HTTP Request + +`GET /api/results/<session_token>/<api_name>/reporturl` + +### Response Payload + +```json +{ + "uri": "String" +} +``` + +### Example + +**Request:** + +`GET /api/results/d9caaae0-c362-11e9-943f-eedb305f22f6/apiOne/reporturl` + +**Response:** + +```json +{ + "uri": "/results/d9caaae0-c362-11e9-943f-eedb305f22f6/apiOne/all.html" +} +``` + +## 2. `view multi report` + +Returns a URL to a report for an API of multiple session, generated by the WPT Report tool. + +### HTTP Request + +`GET /api/results/<api_name>/reporturl` + +### Query Parameters + +| Parameter | Description | Default | Example | +| --------- | ------------------------------------------------------------ | ------- | -------------------------------- | +| `tokens` | Comma separated list of tokens to create a multi report for. | none | `tokens=token_a,token_b,token_c` | + +### Example + +**Request:** + +`GET /api/results/apiOne/reporturl?tokens=ce2dc080-c283-11e9-b4d6-e046513784c2,cd922410-c344-11e9-858f-9063f6dd878f` + +**Response:** + +```json +{ + "uri": "/results/comparison-cd922410-ce2dc080-1709d631/apiOne/all.html" +} +```
\ No newline at end of file diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/control.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/control.md new file mode 100644 index 0000000000..c439c118f8 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/control.md @@ -0,0 +1,25 @@ +# Controlling Sessions - [Sessions API](../README.md#sessions-api) + +It is possible to control the execution of tests on the device under test using the session APIs control methods. They change the status of a session and trigger the device under test to fetch a new url to change location to. Depending on the current status of the session this can be a test or a static page showing information about the current status. + +## `start` +The `start` method changes the status of a session from either `PENDING` or `PAUSED` to `RUNNING` and triggers the device under test to execute tests when resuming a paused session. + +### HTTP Request + +`POST /api/sessions/<session_token>/start` + +## `pause` +The `pause` method changes the status of a session from `RUNNING` to `PAUSED` and pauses the execution of tests on the device under test. + +### HTTP Request + +`POST /api/sessions/<session_token>/pause` + +## `stop` +The `stop` method finishes a session early by skipping all pending tests, causing a change of the status to `ABORTED`. It is not possible to undo this action and can only be performed on sessions that are not `ABORTED` or `COMPLETED`. + +### HTTP Request + +`POST /api/sessions/<session_token>/stop` + diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/create.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/create.md new file mode 100644 index 0000000000..86c0674afd --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/create.md @@ -0,0 +1,103 @@ +# `create` - [Sessions API](../README.md#sessions-api) + +The `create` method of the sessions API creates a new session. If provided with an configuration it creates a session accordingly. If no configuration is provided it uses default values. It returns the session token of the newly created session, which is the unique identifier of sessions. While a session has the status `PENDING` it is possible to modify the configuration using the [`update`](./update.md) method of the sessions API. As it is required to create the session from the device under test, this is really helpful, since it allows to configure the session using a second device. + +## HTTP Request + +`POST /api/sessions` + +## Request Payload + +```json +{ + "tests": { + "include": "Array<String>", + "exclude": "Array<String>" + }, + "types": "Enum['automatic', 'manual']", + "timeouts": { + "automatic": "Integer", + "manual": "Integer", + "<test_path>": "Integer" + }, + "reference_tokens": "Array<String>", + "labels": "Array<String>", + "type": "String" +} +``` + +- **tests** specifies the tests of the session: + - **include** specifies what tests should be selected from all available tests. Can be a path to a test file or directory. Provided query parameters will be added to all matching test urls. + - **exclude** specifies what tests should be removed from the included tests. Can be a path to a test file or directory. +- **types** what types of tests should be included. Possible values: + - **automatic** tests are tests that execute without user interaction. + - **manual** tests are tests that require user interaction. +- **timeouts** specifies the time to wait for a test to finish in milliseconds. + - **automatic**: Sets the default timeout for all automatic tests. + - **manual**: Sets the default timeout for all manual tests. + - **custom test paths**: Set the timeout for a test file or directory by putting the path with all dots removed as the key. +- **reference_tokens** specifies a set of completed sessions that is used to filter out all tests that have not passed in all those sessions from the session that is going to be created. +- **labels** specifies the initial set of labels for the session. +- **type** specifies the session type to trigger type specific behaviour like different control pages. + +### Default + +```json +{ + "tests": { + "include": ["/"], + "exclude": [] + }, + "types": ["automatic", "manual"], + "timeouts": { + "automatic": 60000, + "manual": 300000 + }, + "reference_tokens": [], + "labels": [] +} +``` + +## Response Payload + +If successful, the token of the new session is returned. + +```json +{ + "token": "String" +} +``` + +## Example + +**Request:** + +`POST /api/sessions` + +```json +{ + "tests": { + "include": ["/apiOne", "/apiTwo/sub"], + "exclude": ["/apiOne/specials"] + }, + "types": ["automatic"], + "timeouts": { + "automatic": 70000, + "/apiOne/example/dir": 30000, + "/apiOne/example/filehtml": 45000 + }, + "reference_tokens": [ + "ce2dc080-c283-11e9-b4d6-e046513784c2", + "430f47d0-c283-11e9-8776-fcbc36b81035" + ], + "labels": ["label1", "label2", "label3"] +} +``` + +**Response:** + +```json +{ + "token": "6fdbd1a0-c339-11e9-b775-6d49dd567772" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/delete.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/delete.md new file mode 100644 index 0000000000..f11d86035b --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/delete.md @@ -0,0 +1,25 @@ +# `delete` - [Sessions API](../README.md#sessions-api) + +The `delete` method of the sessions API is used to delete a session and single results associated with it. However artifacts like generated reports or JSON files containing results of a whole API remain, therefore urls to those resources are still working. + +## HTTP Request + +`DELETE /api/sessions/<session_token>` + +## Example + +**Request:** + +`DELETE /api/sessions/1592b880-c339-11e9-b414-61af09c491b1` + +**Response:** + +`200 OK` + +**Request:** + +`GET /api/sessions/1592b880-c339-11e9-b414-61af09c491b1` + +**Response:** + +`404 NOT FOUND` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/event-types.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/event-types.md new file mode 100644 index 0000000000..455e0a122b --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/event-types.md @@ -0,0 +1,27 @@ +# Session Event Types - [Sessions API](../README.md#sessions-api) + +Session events are events that are triggered by actions related to sessions. +The [`event`](./events.md) functions of the sessions API make use of these events. + +## Status change + +**Type identifier**: `status` +**Payload**: `"<String>"` +Possible Values: `paused`, `running`, `completed`, `aborted` +**Description**: Triggered once the status of the session changes. + +## Resume + +**Type identifier**: `resume` +**Payload**: `"<String>"` +Contains the token of the session to resume. +**Description**: Triggered when a specific session is supposed to be resumed. +This will discard the current session and continue executing the session with +the provided token. + +## Test Completed + +**Type identifier**: `test_completed` +**Payload**: `"<String>"` +Contains the test case that completed. +**Description**: Triggered when the test runner received a result for a test. diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/events.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/events.md new file mode 100644 index 0000000000..c1b693c88c --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/events.md @@ -0,0 +1,104 @@ +# `events` - [Sessions API](../README.md#sessions-api) + +Session events can be used to send messages related to a specific session for +others to receive. This can include status updates or action that running +session react on. + +For possible events see [Session Event Types](./event-types.md) + +## 1. `listen events` + +Listen for session specific events by registering on the `events` endpoint using HTTP long polling. + +### HTTP Request + +``` +GET /api/sessions/<token>/events +``` + +### Query Parameters + +| Parameter | Desciption | Default | Example | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------------- | +| `last_event` | The number of the last received event. All events that are newer than `last_event` are returned immediately. If there are no newer events, connection stays open until a new event is triggered. | None | `last_event=5` | + +#### Response Payload + +```json +[ + { + "type": "String", + "data": "String", + "number": "Number" + }, + ... +] +``` + +- **type**: the type of event that occurred. +- **data**: the actual payload of the event +- **number**: the number of the event + +#### Example + +``` +GET /api/sessions/6fdbd1a0-c339-11e9-b775-6d49dd567772/events?last_event=8 +``` + +```json +[ + { + "type": "status", + "data": "paused", + "number": 9 + }, + { + "type": "status", + "data": "running", + "number": 10 + }, + { + "type": "status", + "data": "paused", + "number": 11 + }, + { + "type": "status", + "data": "running", + "number": 12 + } +] +``` + +## 2. `push events` + +Push session specific events for any registered listeners to receive. + +### HTTP Request + +``` +POST /api/sessions/<token>/events +``` + +```json +{ + "type": "String", + "data": "String" +} +``` + +- **type**: the type of event that occurred. +- **data**: the actual payload of the event + +#### Example + +``` +POST /api/sessions/6fdbd1a0-c339-11e9-b775-6d49dd567772/events +``` + +```json +{ + "type": "status", + "data": "paused" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/find.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/find.md new file mode 100644 index 0000000000..3ffc6f58c4 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/find.md @@ -0,0 +1,29 @@ +# `find` - [Sessions API](../README.md#sessions-api) + +The `find` method of the sessions API searches for a session token using a provided token fragment, which is the beginning of a session token with at least 8 characters. Due to data protection, it is not possible to find multiple tokens using one fragment. If the server finds more than one session token, it returns none. In this case more characters need to be added to the fragment, until it matches only one session token. + +## HTTP Request + +`GET /api/sessions/<token_fragment>` + +## Response Payload + +```json +{ + "token": "String" +} +``` + +### Example + +**Request:** + +`GET /api/sessions/afd4ecb0` + +**Response:** + +```json +{ + "token": "afd4ecb0-c339-11e9-b66c-eca76c2bea9c" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/labels.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/labels.md new file mode 100644 index 0000000000..00a8defd98 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/labels.md @@ -0,0 +1,75 @@ +# `labels` - [Sessions API](../README.md#sessions-api) + +The `labels` methods of the sessions API allow for better organization of sessions. + +## Read labels + +Reads all labels of a session. + +### HTTP Request + +`GET /api/sessions/<token>/labels` + +### Response Payload + +```json +"Array<String>" +``` + +#### Example + +**Request:** + +`GET /api/sessions/afd4ecb0-c339-11e9-b66c-eca76c2bea9c/labels` + +**Response:** + +```json +["label1", "label2", "label3"] +``` + +## Update labels + +Update all labels of a session. + +### HTTP Request + +`PUT /api/sessions/<token>/labels` + +### Request Payload + +```json +"Array<String>" +``` + +The array of labels provided in the request payload will replace all existing labels of the session. + +#### Example + +**Request:** + +`GET /api/sessions/afd4ecb0-c339-11e9-b66c-eca76c2bea9c/labels` + +**Response:** + +```json +["label1", "label2", "label3"] +``` + +**Request:** + +`PUT /api/sessions/afd4ecb0-c339-11e9-b66c-eca76c2bea9c/labels` + +```json +["label4", "label5"] +``` + +**Request:** + +`GET /api/sessions/afd4ecb0-c339-11e9-b66c-eca76c2bea9c/labels` + +**Response:** + +```json +["label4", "label5"] +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read-public.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read-public.md new file mode 100644 index 0000000000..3e0e9089c4 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read-public.md @@ -0,0 +1,30 @@ +# `read public` - [Sessions API](../README.md#sessions-api) + +The `read public` method of the sessions API fetches a list of all sessions that are publicly available. It is not possible to delete those sessions using the user interface or the REST API. Currently there is no way to change is-public-state of a session using the API. + +## HTTP Request + +`GET /api/sessions/public` + +## Response Payload + +```json +"Array<String>" +``` + +## Example + +**Request:** + +`GET /api/sessions/public` + +**Response:** + +```json +[ + "bb7aafa0-6a92-11e9-8ec2-04f58dad2e4f", + "caf823e0-6a92-11e9-b732-3188d0065ebc", + "a50c6db0-6a94-11e9-8d1b-e23fc4555885", + "b2924d20-6a93-11e9-98b4-a11fb92a6d1c" +] +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read.md new file mode 100644 index 0000000000..4ec9c3f63f --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read.md @@ -0,0 +1,90 @@ +# `read session` - [Sessions API](../README.md#sessions-api) + +The `read` method of the sessions API fetches the configuration of a session, including values that can not be set by the user, but are created by the server upon creation. + +## HTTP Request + +`GET /api/sessions/<session_token>` + +## Response Payload + +```json +{ + "token": "String", + "tests": { + "include": "Array<String>", + "exclude": "Array<String>" + }, + "types": "Enum['automatic', 'manual']", + "timeouts": { + "automatic": "Integer", + "manual": "Integer", + "<test_path>": "Integer" + }, + "reference_tokens": "Array<String>", + "user_agent": "String", + "browser": { + "name": "String", + "version": "String" + }, + "is_public": "Boolean", + "date_created": "String", + "labels": "Array<String>" +} +``` + +- **token** is the unique identifier of the session. +- **tests** specifies the tests of the session: + - **include** specifies what tests should be selected from all available tests. Can be a path to a test file or directory. + - **exclude** specifies what tests should be removed from the included tests. Can be a path to a test file or directory. +- **types** what types of tests should be included. Possible values: + - **automatic** tests are tests that execute without user interaction. + - **manual** tests are tests that require user interaction. +- **timeouts** specifies the time to wait for a test to finish in milliseconds. + - **automatic**: Sets the default timeout for all automatic tests. + - **manual**: Sets the default timeout for all manual tests. + - **custom test paths**: Set the timeout for a test file or directory by putting the path with all dots removed as the key. +- **reference_tokens** specifies a set of completed sessions that is used to filter out all tests that have not passed in all those sessions from the session that is going to be created. +- **user_agent** is the user agent string of the request that created the session. The request to create the session should performed by the device under test. +- **browser** holds information about the browser, parsed from the user agent. + - **name**: The name of the browser. + - **version**: The version numbers of the browser. +- **is_public** defines whether or not the session is listed when fetching the list of public session using [`read public`](./read-public.md). +- **date_created**: The date the session was created in ISO 8601 format. +- **labels**: A list of the sessions labels. + +## Example + +**Request:** + +`GET /api/sessions/47a6fa50-c331-11e9-8709-a8eaa0ecfd0e` + +**Response:** + +```json +{ + "token": "47a6fa50-c331-11e9-8709-a8eaa0ecfd0e", + "tests": { + "include": ["/apiOne", "/apiTwo/sub"], + "exclude": ["/apiOne/specials"] + }, + "types": ["automatic"], + "timeouts": { + "automatic": 70000, + "/apiOne/example/dir": 30000, + "/apiOne/example/filehtml": 45000 + }, + "reference_tokens": [ + "ce2dc080-c283-11e9-b4d6-e046513784c2", + "430f47d0-c283-11e9-8776-fcbc36b81035" + ], + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/76.0.3809.100 Chrome/76.0.3809.100 Safari/537.36", + "browser": { + "name": "Chromium", + "version": "76" + }, + "is_public": "false", + "date_created": "2020-05-25T11:37:07", + "labels": ["labelA", "labelB"] +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read_sessions.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read_sessions.md new file mode 100644 index 0000000000..d29371f006 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/read_sessions.md @@ -0,0 +1,123 @@ +# `read sessions` - [Sessions API](../README.md#sessions-api) + +The `read sessions` method of the sessions API fetches a list of all available +session's token with the option expand the returned data by all retrieved +tokens corresponding session configurations or statuses. + +## HTTP Request + +``` +GET /api/sessions +``` + +### Query Parameters + +| Parameter | Description | Default | +| --------- | ---------------------------------------------------------------------------------------------------- | ------- | +| `index` | At what index of all session to start the returned list. | `0` | +| `count` | How many entries to return starting from `index`. | `10` | +| `expand` | Comma separated list of relations from `_links`. Includes additional data in the `_embedded` object. | none | + +### Response Payload + +``` +200 OK +Content-Type: application/json+hal +``` + +```json +{ + "_links": { + "<relation>": { + "href": "String" + } + ... + }, + "_embedded": { + "<relation>": "Array<Any>" + ... + }, + "items": "Array<String>" +} +``` + +- **\_links** contains URLs to related data. For more, see [HAL Specfication](https://tools.ietf.org/html/draft-kelly-json-hal). +- **\_embedded** additional content specified by `expand` query paramater. For more, see [HAL Specfication](https://tools.ietf.org/html/draft-kelly-json-hal). +- **items** contains the returned list of session tokens. + +## Example + +``` +GET /api/sessions?index=0&count=3&expand=status +``` + +``` +200 OK +Content-Type: application/json+hal +``` + +```json +{ + "_links": { + "first": { + "href": "/_wave/api/sessions?index=0&count=3" + }, + "last": { + "href": "/_wave/api/sessions?index=39&count=3" + }, + "self": { + "href": "/_wave/api/sessions?index=0&count=3" + }, + "next": { + "href": "/_wave/api/sessions?index=3&count=3" + }, + "configuration": { + "href": "/_wave/api/sessions/{token}", + "templated": true + }, + "status": { + "href": "/_wave/api/sessions/{token}/status", + "templated": true + } + }, + "items": [ + "13f80c84-9046-11ea-9c80-0021ccd76152", + "34db08e4-903b-11ea-89ce-0021ccd76152", + "a355f846-9465-11ea-ae9e-0021ccd76152" + ], + "_embedded": { + "status": [ + { + "status": "completed", + "expiration_date": null, + "labels": [], + "date_finished": 1588844145897.157, + "token": "13f80c84-9046-11ea-9c80-0021ccd76152", + "date_created": null, + "date_started": 1588844127000, + "type": "wmas" + }, + { + "status": "completed", + "expiration_date": null, + "labels": [], + "date_finished": 1588839822768.9568, + "token": "34db08e4-903b-11ea-89ce-0021ccd76152", + "date_created": null, + "date_started": 1588839522000, + "type": "wmas" + }, + { + "status": "completed", + "expiration_date": null, + "labels": [], + "date_finished": 1589297485065.1802, + "token": "a355f846-9465-11ea-ae9e-0021ccd76152", + "date_created": null, + "date_started": 1589297484000, + "type": "wmas" + } + ] + } +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/status.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/status.md new file mode 100644 index 0000000000..b2604a41df --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/status.md @@ -0,0 +1,48 @@ +# `status` - [Sessions API](../README.md#sessions-api) + +The `status` method of the results API returns information about a sessions current status and progress. + +## HTTP Request + +`GET /api/sessions/<session_token>/status` + +## Response Payload + +```json +{ + "token": "String", + "status": "Enum['pending', 'running', 'paused', 'completed', 'aborted']", + "date_started": "String", + "date_finished": "String", + "expiration_date": "String" +} +``` + +- **token** contains the token of the session corresponding to this status. +- **status** specifies the current status of the session: + - **pending**: The session was created, can receive updates, however cannot execute tests. + - **running**: The session currently executes tests. + - **paused**: The execution of tests in this session is currently paused. + - **completed**: All tests files include in this session were executed and have a result. + - **aborted**: The session was finished before all tests were executed. +- **date_started** contains the time the status changed from `PENDING` to `RUNNING` in ISO 8601. +- **date_finished** contains the time the status changed to either `COMPLETED` or `ABORTED` in ISO 8601. +- **expiration_date** contains the time at which the sessions will be deleted in ISO 8601. + +## Example + +**Request:** + +`GET /api/sessions/d9caaae0-c362-11e9-943f-eedb305f22f6/status` + +**Response:** + +```json +{ + "token": "d9caaae0-c362-11e9-943f-eedb305f22f6", + "status": "running", + "date_started": "2019-09-04T14:21:19", + "date_finished": null, + "expiration_date": "2019-09-04T14:26:19" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/update.md b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/update.md new file mode 100644 index 0000000000..3817c11593 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/sessions-api/update.md @@ -0,0 +1,102 @@ +# `update` - [Sessions API](../README.md#sessions-api) + +The `update` method of the sessions API makes it possible to modify a sessions configuration while its status is `PENDING`. This can be used to configure the session on a second device, rather than on the device under test. + +## HTTP Request + +`PUT /api/sessions/<session_token>` + +## Request Payload + +The request payload is the same as in the [`create`](./sessions-api/create.md) method of the sessions API. Only keys that are an inherent part of the configuration will stay the same if not specified in the `update` payload. All others will be deleted if not included. + +## Example + +**Request:** + +`GET /api/sessions/47a6fa50-c331-11e9-8709-a8eaa0ecfd0e` + +**Response:** + +```json +{ + "token": "47a6fa50-c331-11e9-8709-a8eaa0ecfd0e", + "tests": { + "include": ["/apiOne", "/apiTwo/sub"], + "exclude": ["/apiOne/specials"] + }, + "types": ["automatic"], + "timeouts": { + "automatic": 70000, + "/apiOne/example/dir": 30000, + "/apiOne/example/filehtml": 45000 + }, + "reference_tokens": [ + "ce2dc080-c283-11e9-b4d6-e046513784c2", + "430f47d0-c283-11e9-8776-fcbc36b81035" + ], + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/76.0.3809.100 Chrome/76.0.3809.100 Safari/537.36", + "browser": { + "name": "Chromium", + "version": "76" + }, + "is_public": "false", + "labels": [] +} +``` + +**Request:** + +`PUT /api/sessions/47a6fa50-c331-11e9-8709-a8eaa0ecfd0e` + +```json +{ + "tests": { + "include": ["/apiOne", "/apiThree"] + }, + "timeouts": { + "automatic": 60000 + }, + "reference_tokens": [ + "bb7aafa0-6a92-11e9-8ec2-04f58dad2e4f", + "a50c6db0-6a94-11e9-8d1b-e23fc4555885" + ], + "labels": ["label1", "label2"] +} +``` + +**Response:** + +`200 OK` + +**Request:** + +`GET /api/sessions/47a6fa50-c331-11e9-8709-a8eaa0ecfd0e` + +**Response:** + +```json +{ + "token": "47a6fa50-c331-11e9-8709-a8eaa0ecfd0e", + "tests": { + "include": ["/apiOne", "/apiThree"], + "exclude": ["/apiOne/specials"] + }, + "types": ["automatic"], + "timeouts": { + "automatic": 60000, + "manual": 360000 + }, + "reference_tokens": [ + "bb7aafa0-6a92-11e9-8ec2-04f58dad2e4f", + "a50c6db0-6a94-11e9-8d1b-e23fc4555885" + ], + "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/76.0.3809.100 Chrome/76.0.3809.100 Safari/537.36", + "browser": { + "name": "Chromium", + "version": "76" + }, + "is_public": "false", + "labels": ["label1", "label2"] +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-all.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-all.md new file mode 100644 index 0000000000..8480981b66 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-all.md @@ -0,0 +1,43 @@ +# `read all` - [Tests API](../README.md#tests-api) + +The `read all` method of the tests API fetches all tests available to include into a test session. + +## HTTP Request + +`GET /api/tests` + +## Response Payload + +```json +{ + "<api_name>": "Array<String>" +} +``` + +## Example + +**Request:** + +`GET /api/tests` + +**Response:** + +```json +{ + "apiOne": [ + "/apiOne/test/one.html", + "/apiOne/test/two.html", + "/apiOne/test/three.html" + ], + "apiTwo": [ + "/apiTwo/test/one.html", + "/apiTwo/test/two.html", + "/apiTWo/test/three.html" + ], + "apiThree": [ + "/apiThree/test/one.html", + "/apiThree/test/two.html", + "/apiThree/test/three.html" + ] +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-available-apis.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-available-apis.md new file mode 100644 index 0000000000..d197c2c21a --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-available-apis.md @@ -0,0 +1,43 @@ +# `read available apis` - [Tests API](../README.md#tests-api) + +The `read available apis` method return a list of all web APIs that the DUT +can be tested for. It returns the human readable API name, as well as the +directory name under which all corresponding tests reside. + +## HTTP Request + +`GET /api/tests/apis` + +## Response Payload + +```json +[ + { + "path": "String", + "name": "String" + }, + ... +] +``` + +## Example + +**Request:** + +`GET /api/tests/apis` + +**Response:** + +```json +[ + { + "path": "/2dcontext", + "name": "2D Context" + }, + { + "path": "/media-source", + "name": "Media Source" + }, + ... +] +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-last-completed.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-last-completed.md new file mode 100644 index 0000000000..1c1f87a726 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-last-completed.md @@ -0,0 +1,47 @@ +# `read last completed` - [Tests API](../README.md#tests-api) + +The `read last completed` method of the tests API returns a list of test files, which most recently finished and have a result. The files are grouped by the status their respective result had. + +## HTTP Request + +`GET /api/tests/<session_token>/last_completed` + +## Query Parameters + +| Parameter | Desciption | Default | Example | +| --------- | ------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------- | +| `count` | Number of files per status to return | 5 | `count=5` | +| `status` | The status the files results must have. Comma separated list. Possible values: `all`, `pass`, `fail` and `timeout` | `all` | `status=timeout,pass` | + +## Response Payload + +```json +{ + "pass": "Array<String>", + "fail": "Array<String>", + "timeout": "Array<String>" +} +``` + +## Example + +**Request:** + +`GET /api/tests/7dafeec0-c351-11e9-84c5-3d1ede2e7d2e/last_completed?count=3&status=fail,timeout` + +**Response:** + +```json +{ + "fail": [ + "/apiTwo/test/four.html", + "/apiOne/test/twentyfour.html", + "/apiOne/test/nineteen.html" + ], + "timeout": [ + "/apiFive/test/eight.html", + "/apiThree/test/five.html", + "/apiThree/test/two.html" + ] +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-malfunctioning.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-malfunctioning.md new file mode 100644 index 0000000000..755b2f7897 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-malfunctioning.md @@ -0,0 +1,30 @@ +# `read malfunctioning` - [Tests API](../README.md#tests-api) + +The `read malfunctioning` method of the tests API returns a list of test files, which were flagged as not working properly in a specific session. This is useful to [add them to the exclude list](../../usage/excluding-tests.md) of further test sessions. + +## HTTP Request + +`GET /api/tests/<session_token>/malfunctioning` + +## Response Payload + +```json +"Array<String>" +``` + +## Example + +**Request:** + +`GET /api/tests/7dafeec0-c351-11e9-84c5-3d1ede2e7d2e/malfunctioning` + +**Response:** + +```json +[ + "/apiOne/test/one.html", + "/apiOne/test/five.html", + "/apiThree/test/two.html", + "/apiThree/test/twenty.html" +] +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-next.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-next.md new file mode 100644 index 0000000000..c10ba81a12 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-next.md @@ -0,0 +1,29 @@ +# `read next` - [Tests API](../README.md#tests-api) + +The `read next` method of the tests API returns the next test of a test session, that is due to be executed. If the sessions status is not `RUNNING` it returns a static page containing information about the session and its current status. + +## HTTP Request + +`GET /api/tests/<session_token>/next` + +## Response Payload + +```json +{ + "next_test": "String" +} +``` + +## Example + +**Request:** + +`GET /api/tests/d6667670-c350-11e9-b504-4ac471cdd99d/next` + +**Response:** + +```json +{ + "next_test": "http://web-platform.test:8000/apiOne/test/one.html?&token=d6667670-c350-11e9-b504-4ac471cdd99d&timeout=60000" +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-session.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-session.md new file mode 100644 index 0000000000..59344d5e68 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/read-session.md @@ -0,0 +1,61 @@ +# `read session` - [Tests API](../README.md#tests-api) + +The `read session` method of the tests API fetches all tests contained in a test session grouped by their status. + +## HTTP Request + +`GET /api/tests/<session_token>` + +## Response Payload + +```json +{ + "token": "String", + "pending_tests": { + "<api_name>": "Array<String>" + }, + "running_tests": { + "<api_name>": "Array<String>" + }, + "completed_tests": { + "<api_name>": "Array<String>" + } +} +``` + +- **pending_tests** are tests that have yet to be executed. +- **running_tests** are tests that are currently executed by the device under test. Although only one test at a time is executed, test that time out or fail to send a result may still wait for the time out to occur. In this case there are multiple tests in this list. +- **completed_tests** are tests that are finished and have a result. + +## Example + +**Request:** + +`GET /api/tests/cd922410-c344-11e9-858f-9063f6dd878f` + +**Response:** + +```json +{ + "token": "cd922410-c344-11e9-858f-9063f6dd878f", + "pending_tests": { + "apiTwo": ["/apiTwo/test/three.html"], + "apiThree": [ + "/apiThree/test/one.html", + "/apiThree/test/two.html", + "/apiThree/test/three.html" + ] + }, + "running_tests": { + "apiTwo": ["/apiTwo/test/two.html"] + }, + "completed_tests": { + "apiOne": [ + "/apiOne/test/one.html", + "/apiOne/test/two.html", + "/apiOne/test/three.html" + ], + "apiTwo": ["/apiTwo/test/one.html"] + } +} +``` diff --git a/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/update-malfunctioning.md b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/update-malfunctioning.md new file mode 100644 index 0000000000..327d1c14d1 --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/rest-api/tests-api/update-malfunctioning.md @@ -0,0 +1,56 @@ +# `update malfunctioning` - [Tests API](../README.md#tests-api) + +The `update malfunctioning` method of the tests API sets the list of test files, that are flagged as not working properly in a specific session. It replaces the existing list with the new provided list. + +## HTTP Request + +`PUT /api/tests/<session_token>/malfunctioning` + +## Request Payload + +```json +"Array<String>" +``` + +## Example + +**Request:** + +`GET /api/tests/7dafeec0-c351-11e9-84c5-3d1ede2e7d2e/malfunctioning` + +**Response:** + +```json +[ + "/apiOne/test/one.html", + "/apiOne/test/five.html", + "/apiThree/test/two.html", + "/apiThree/test/twenty.html" +] +``` + +**Request:** + +`PUT /api/tests/7dafeec0-c351-11e9-84c5-3d1ede2e7d2e/malfunctioning` + +```json +[ + "/apiOne/test/three.html", + "/apiOne/test/eight.html", + "/apiThree/test/one.html" +] +``` + +**Request:** + +`GET /api/tests/7dafeec0-c351-11e9-84c5-3d1ede2e7d2e/malfunctioning` + +**Response:** + +```json +[ + "/apiOne/test/three.html", + "/apiOne/test/eight.html", + "/apiThree/test/one.html" +] +``` diff --git a/testing/web-platform/tests/tools/wave/docs/usage/usage.md b/testing/web-platform/tests/tools/wave/docs/usage/usage.md new file mode 100644 index 0000000000..50a99e0d5a --- /dev/null +++ b/testing/web-platform/tests/tools/wave/docs/usage/usage.md @@ -0,0 +1,231 @@ +# Usage Guide - [WAVE Test Runner](../README.md) + +With WAVE Test Runner v1.0.0 all files and REST API endpoints are served under +a configurable namespace, by default `/_wave/`, which will be used in this +usage guide. + +In this document the usage is explained using screenshots from the context of +the WMAS project. However, practices can be applied to different contexts as well. + +## Contents + +1. [Creating test sessions](#1-creating-test-sessions) + 1. [The landing page](#11-the-landing-page) + 2. [Configuring a new session](#12-configuring-a-new-session) + 3. [Exclude tests](#13-exclude-tests) + 1. [Manually specify tests to exclude](#131-manually-specify-tests-to-exclude) + 2. [Use a session's malfunctioning list to add tests to exclude](#132-use-a-sessions-malfunctioning-list-to-add-tests-to-exclude) + 3. [Use a previous session's exclude list to add tests to exclude](#133-use-a-previous-sessions-exclude-list-to-add-tests-to-exclude) +2. [Resuming test sessions](#2-resuming-test-sessions) + 1. [Using the webinterface](#21-using-the-webinterface) + 2. [Using a URL](#22-using-a-url) +3. [Monitoring test sessions](#3-monitoring-test-sessions) +4. [Managing test sessions](#4-managing-test-sessions) + +# 1. Creating test sessions + +Test sessions hold information about one test run one a particular device, like the current status. +Each session is identified using a UUIDv1 token string to gather these information or perform actions on it. +Each new session is configured using several parameters before the run starts. + +## 1.1 The landing page + +Every new session is created from the landing page. +It is recommended to create a new session from the device that is tested, as the user agent is part of the displayed information, as well as the browser and version, which gets parsed from it. +However, this does not influence the execution of tests or the creation of test results. +To create a new session, open the landing page on the URI path `/_wave/index.html`. + +![landing_page] + +The landing page is divided into two section, one to create a new session and one to resume a session. +As soon as the landing is opened, a new test session is created. +Its token is displayed next to the QR-Code on the right, along with the expiration date. +As the session was created automatically, it gets removed automatically once it expires. +However, if you start the session, the expiration date gets removed and the sessions is available until you delete it. + +## 1.2 Configuring a new session + +To configure and start the session, either click on "Configure Session" or scan the QR-Code. +In most cases it is recommended to scan the QR-Code, as it does not require any interaction with the landing page on the DUT. + +![configuration_page] + +In the configuration screen you can set parameters for the new session and start it. +At the top the session's token and expiration date is displayed. Next there is the "Labels" option, which allows adding any number of labels to the session, helping to better organize sessions and allowing to apply filters while searching. +Labels can be added and modified at any point in the future. +Next there is the API selection, which allows defining the set of APIs to test in the new session. To exclude specific test or subdirectories of those selected APIs, there is the "Excluded Tests" option right below it. Here you can specify what tests to exclude in three distinct ways. (More details in [1.3 Exclude tests](#13-exclude-tests)) + +![configuration_page_bottom] + +With the "Test Types" option you specify what types of test should be included into the session: in contrast to automatic tests, manual tests require user interaction to execute properly. +The "Reference Browsers" option lets you select browsers that are used to further filter the set of tests included in the session. +Only tests that have passed the reference test session in all selected browsers are included. +The reference browsers represent the status of implementation of all WAVE APIs in modern desktop browsers, at about the time the WAVE specification was published. +To start the session press "Start Session", note that the landing page has to stay opened, as the test are going to be execute in the same window. + +[To the top](#usage-guide---wave-test-runner) + +## 1.3 Exclude tests + +To have a fine control over what test cases are executed when configuring a session, it is possible to provide a list of test cases, that are omitted in the run. + +### 1.3.1 Manually specify tests to exclude + +To add tests to exclude by providing a plain text list, click on "Add Raw" in the "Excluded Tests" setting. +This opens a input field, where you can enter multiple full paths to test files or directories. + +![Exclude List Add Raw][configuration_page_add_raw] + +Each line will be interpreted as a path to exclude a single or a group of tests. +All tests that have a path starting with one of the provided, will be excluded in the session. +Lines starting with a # symbol will be ignored, in case you want to organize test paths in a text file using comments. +Click "Add" and you will see the paths listed in the table below. + +### 1.3.2 Use a session's malfunctioning list to add tests to exclude + +When flagging tests in a running session as malfunctioning, e.g. when crashing the device, it is possible to add these test to the exclude list of the new session. +To do this, click on "Add Malfunctioning" in the "Excluded Tests" section. + +![Exclude List Add Malfunctioning][configuration_page_add_malfunctioning] + +Enter the first eight characters or more into the text field labelled "Session Token" to import all tests from the session's malfunctioning list into the new session's exclude list. +Click "Add" to confirm. +The tests should now appear in the list below. + +### 1.3.3 Use a previous session's exclude list to add tests to exclude + +If you have already specified a suitable exclude list or want to expand an existing, you can apply the exclude list of a previous session. +Click on "Add Previous Excluded" in the "Excluded Tests" section to open the corresponding controls. + +![Exclude List Add Previously Excluded][configuration_page_add_prev_excluded] + +Enter the first eight characters or more into the text field labelled "Session Token" to import all tests from the previous session's exclude list into the new session's exclude list. +Click "Add" to confirm. +The tests should now appear in the list below. + +[To the top](#usage-guide---wave-test-runner) + +# 2. Resuming test sessions + +Certain test cases may cause some devices to crash, which makes the test runner unable to automatically run the next test. +In this case, external interaction is necessary. +To alleviate the process of resuming the test session, the are two mechanisms integrated into the web interface that reduce interaction with the device to a minimum. +There is also a mechanism that can be useful if a test framework with access to the tested browser is utilized. + +## 2.1 Using the webinterface + +In any case, it is necessary to open the landing page on the device, in order to resume the session. + +![Landing Page][landing_page] + +On the landing page, in the section "Resume running session", you can see the token of the last session this device has run. +To resume this particular session, click on the "Resume" button next to it, or simply press enter or space. +If the presented token is not the one of the session you want to resume, you can change it from the configuration screen. +To get there, press the "Configure Session" button or scan the QR-Code. + +![Configuration Page][configuration_page] + +At the very bottom of the configuration page, there is a section called "Resume session", where you can see the token that was previously displayed on the landing page in a text box. +Here you can change the token of the session to resume, just enter the first eight characters or more of the token. +When you're done, press the "Resume" button. +Note that it is necessary to keep the landing page open in order to automatically run the next test, as it is loaded in the same window. + +## 2.2 Using a URL + +If you have access to the DUTs browser programmatically, you may want to resume a crashed test session automatically. +To load the next test of a specific session, simply open the following URL: + +`/next.html?token=<session_token>` + +For example: + +`/_wave/next.html?token=24fcd360-ef4d-11e9-a95f-d6e1ad4c5fdb` + +[To the top](#usage-guide---wave-test-runner) + +# 3. Monitoring test sessions + +While running test sessions, the results page for second screen devices provide a convenient summary of the sessions current state, as well as controls to manipulate the test execution. +Additionally, you can flag tests in case they interrupt the test execution by, e.g. crashing the test, to exclude them in future sessions and download test results and reports. + +![results_page_top] + +On the top right-hand side, there are controls to stop, pause or delete the session. +Stopping, as well as deleting the session is irreversible. +Below you find the session's details, including the token, user agent, test paths, excluded test paths, total test file count, status, the different test timeouts, the date and time the session has been started, the date and time the session has finished, the duration and labels. + +![results_page_last_completed] + +Right below, tests that have recently completed with result status TIMEOUT are listed to add them to the list of malfunctioning tests by clicking the button with the + symbol. +Now that test appears in the list of malfunctioning tests at the very bottom of the result page. +This list can be used to exclude tests when creating a new session. (more details in [1.3.2 Use a session's malfunctioning list to add tests to exclude](#132-use-a-sessions-malfunctioning-list-to-add-tests-to-exclude)) + +![results_page_api_results] + +In the section "API Results" you can see the progress of each individual API selected for the session. +As each test file can contain multiple subtests, the count of passed, failed, timed out and not run tests does not correlate to the count of test files run, which indicates the overall progress. +Keep in mind that only test files that received a result will count as run, so even if all tests finished executing on the device, some may have failed to send the result, in which case the internal timeout has to run out to create it. + +![results_page_api_results_export] + +Once all test files of an API have received a result, it is possible to download the result data or view a report for that API, by clicking the corresponding button in the far right column of the table. + +![results_page_bottom] + +Below the table of API results, there are more options to download the results of the session. +The first option downloads the results the same way it is persisted on the serverside, along with some meta data. +This form is especially useful if you want to import the session details with the results into other instances of the WAVE Test Runner. +Furthermore, there is the option to download the raw result in JSON format of all finished APIs. +This the same JSON you get by clicking on the "JSON" button in the API results column, but of all finished APIs in a ZIP file. +Lastly, you can download a static HTML page, similiar to the results view. +Finally, at the bottom of the page you can find the list of malfunctioning tests that have been added from the list of last timed-out test files. +Remove tests by clicking their corresponding button with the trashcan icon. + +[To the top](#usage-guide---wave-test-runner) + +# 4. Managing test sessions + +The overview page provides features that help to manage and organize multiple sessions. You can access it from the URL `/_wave/overview.html`. + +![overview_page] + +In the "Manage Sessions" section you can add more sessions to the list below by entering the first eight or more characters of the token. +Clicking on "Add Session" will add the session to the list if it was the only one that could be associated with the provided token. +If there are multiple sessions that match the provided input, none will be added. +Additionally, you can compare multiple session, given that they are completed, used the same reference sessions and share tested APIs. +Simply select the desired session from the list below and click "Compare Selected". +You can also import sessions in the "Import Sessions" section, however, this feature has to be enabled in the server configuration. +Below the "Manage Sessions" section, there is the list of reference and recent sessions. + +![overview_page_sessions] + +In the sessions list, sessions are organized in three lists: Reference Browsers, which are test results everyone can see, containing the results of the reference browsers for the corresponding WAVE specification, recent sessions, which are sessions there have recently been viewed or executed on the device, and pinned sessions, which are sessions pinned by the user from the list of recent sessions. +Add label filters to show only matching sessions. + +![overview_page_sessions_pinned_recent] + +You can pin a session by clicking the button with the tag on a session in the recent sessions list and unpin them the same way from the pinned sessions list. +Click the trashcan icon to remove a session from its list, this will not delete the session results. +Sort the list of sessions by clicking on the column to filter them by. + +![overview_page_sessions_filtered] + +Add one or more tags to the filter to conveniently find the sessions you are looking for. Add labels to session when creating them or in their corresponding results page. + +[To the top](#usage-guide---wave-test-runner) + +[landing_page]: ../res/landing_page.jpg "Landing Page" +[configuration_page]: ../res/configuration_page_top.jpg "Configuration Page" +[configuration_page_bottom]: ../res/configuration_page_bottom.jpg "Configuration Page" +[configuration_page_add_raw]: ../res/configuration_page_exclude_add_raw.jpg "Exclude Tests - Add Raw" +[configuration_page_add_malfunctioning]: ../res/configuration_page_exclude_add_malfunctioning.jpg "Exclude Tests - Add Malfunctioning" +[configuration_page_add_prev_excluded]: ../res/configuration_page_exclude_add_prev_excluded.jpg "Exclude Tests - Add Previously Excluded" +[results_page_top]: ../res/results_page_top.jpg "Results Page" +[results_page_last_completed]: ../res/results_page_last_timed_out.jpg "Results Page" +[results_page_api_results]: ../res/results_page_api_results.jpg "Results Page" +[results_page_api_results_export]: ../res/results_page_api_results_export.jpg "Results Page" +[results_page_bottom]: ../res/results_page_bottom.jpg "Results Page" +[overview_page]: ../res/overview_page_top.jpg "Overview Page" +[overview_page_sessions]: ../res/overview_page_sessions.jpg "Overview Page Sessions" +[overview_page_sessions_pinned_recent]: ../res/overview_page_sessions_pinned_recent.jpg "Overview Page Sessions" +[overview_page_sessions_filtered]: ../res/overview_page_sessions_filtered.jpg "Overview Page Filter" |
