diff options
Diffstat (limited to 'testing/web-platform/tests/docs/running-tests')
13 files changed, 669 insertions, 0 deletions
diff --git a/testing/web-platform/tests/docs/running-tests/android_webview.md b/testing/web-platform/tests/docs/running-tests/android_webview.md new file mode 100644 index 0000000000..4a86814fdf --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/android_webview.md @@ -0,0 +1,51 @@ +# Android WebView + +To run WPT on WebView on an Android device, some additional set-up is required. + +Currently, Android WebView support is experimental. + +## Prerequisites + +Please check [Chrome for Android](chrome_android.md) for the common instructions for Android support first. + +Ensure you have a userdebug or eng Android build installed on the device. + +Install an up-to-date version of system webview shell: +1. Go to [chromium-browser-snapshots](https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Android/) +2. Find the subdirectory with the highest number and click it, this number can be found + in the "Commit Position" column of row "LAST_CHANGE" (at bottom of page). +3. Download `chrome-android.zip` file and unzip it. +4. Install `SystemWebViewShell.apk`. +5. On emulator, system webview shell may already be installed by default. Then you may need to remove the existing apk: + * Choose a userdebug build. + * Run an emulator with + [writable system partition from command line](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/android_emulator.md/) + +If you have an issue with ChromeDriver version mismatch, try one of the following: + * Try removing `_venv/bin/chromedriver` such that wpt runner can install a matching version + automatically. Failing that, please check your environment path and make + sure that no other ChromeDriver is used. + * Download the [ChromeDriver binary](https://chromedriver.chromium.org/) matching your WebView's major version and specify it on the command line + ``` + ./wpt run --webdriver-binary <binary path> ... + ``` + +Configure host remap rules in the [webview commandline file](https://cs.chromium.org/chromium/src/android_webview/docs/commandline-flags.md?l=57): +``` +adb shell "echo '_ --host-resolver-rules=\"MAP nonexistent.*.test ~NOTFOUND, MAP *.test 127.0.0.1\"' > /data/local/tmp/webview-command-line" +``` + +Ensure that `adb` can be found on your system's PATH. + +## Running Tests + +Example command line: + +```bash +./wpt run --test-type=testharness android_webview <TESTS> +``` + +* Note that there is no support for channel or automatic installation. The test + will be run against the current WebView version installed on the device. + +* Reftests are not supported at the moment. diff --git a/testing/web-platform/tests/docs/running-tests/chrome-chromium-installation-detection.md b/testing/web-platform/tests/docs/running-tests/chrome-chromium-installation-detection.md new file mode 100644 index 0000000000..63fcd8688d --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/chrome-chromium-installation-detection.md @@ -0,0 +1,96 @@ +# Detection and Installation of Browser and WebDriver Binaries and for Chrome and Chromium + +This is a detailed description of the process in which WPT detects and installs the browser +components for Chrome and Chromium. This process can seem convoluted and difficult to +understand at first glance, but the reason for this process is to best ensure these components +are compatible with each other and are the intended items that the user is trying to test. + +## Chrome + +### Detection +**Browser**: Because WPT does not offer installation of Chrome browser binaries, it will +not attempt to detect a Chrome browser binary in the virtual environment directory. +Instead, commonly-used installation locations on various operating systems are checked to +detect a valid Chrome binary. This detection process is only used if the user has not passed +a binary path as an argument using the `--binary` flag. + +**WebDriver**: ChromeDriver detection for Chrome will only occur if a valid browser binary +has been found. Once the browser binary version is detected, the virtual environment +directory will be checked to see if a matching ChromeDriver version is already installed. +If the browser and ChromeDriver versions do not match, the ChromeDriver binary will be +removed from the directory and the user will be prompted to begin the webdriver installation +process. A ChromeDriver version is considered matching the browser version if ChromeDriver shares +the same major version, or next major version when testing Chrome Dev. For example, Chrome 98.x.x.x +is considered to match ChromeDriver version 98.x.x.x, or also ChromeDriver 99.x.x.x if testing +Chrome Dev. + +Note: Both Chrome and Chromium’s versions of ChromeDriver are stored in separate +directories in the virtual environment directory i.e +`_venv3/bin/{chrome|chromium}/{chromedriver}`. This safeguards from accidentally +using Chromium’s ChromeDriver for a Chrome run and vice versa. Additionally, there +is no need to reinstall ChromeDriver versions if switching between testing Chrome and Chromium. + +### Installation +**Browser**: Browser binary installation is not provided through WPT and will throw a +`NotImplementedError` if attempted via `./wpt install`. The user will need to +have a browser binary on their system that can be detected or provide a path explicitly +using the `--binary` flag. + +**WebDriver**: A version of ChromeDriver will only be installed once a Chrome browser binary +has been given or detected. A `FileNotFoundError` will be raised if the user tries to download +ChromeDriver via `./wpt install` and a browser binary is not located. After browser binary +detection, a version of ChromeDriver that matches the browser binary will be installed. +The download source for this ChromeDriver is +[described here](https://chromedriver.chromium.org/downloads/version-selection). +If a matching ChromeDriver version cannot be found using this process, it is assumed that +the Chrome browser binary is a dev version which does not have a ChromeDriver version available +through official releases. In this case, the Chromium revision associated with this version is +detected from [OmahaProxy](https://omahaproxy.appspot.com/) and used to download +Chromium's version of ChromeDriver for use from Chromium snapshots, as this is currently +the closest version we can match for Chrome Dev. Finally, if the revision number detected is +not available in Chromium snapshots, or if the version does not match any revision number, +the latest revision of Chromium's ChromeDriver is installed from Chromium snapshots. + +## Chromium + +### Detection +**Browser**: Chromium browser binary detection is only done in the virtual +environment directory `_venv3/browsers/{channel}/`, not on the user’s system +outside of this directory. This detection process is only used if the user has +not passed a binary path as an argument using the `--binary` flag. + +**WebDriver**: ChromeDriver detection for Chromium will only occur if a valid browser binary has +been found. Once the browser binary version is detected, the virtual environment directory will +be checked to see if a matching ChromeDriver version is already installed. If the versions do not +match, the ChromeDriver binary will be removed from the directory and the user will be prompted to +begin the webdriver installation process. For Chromium, the ChromeDriver and browser versions must be +the same to be considered matching. For example, Chromium 99.0.4844.74 will only match ChromeDriver +99.0.4844.74. + +### Installation +**Browser**: Chromium’s browser binary will be installed from +[Chromium snapshots storage](https://storage.googleapis.com/chromium-browser-snapshots/index.html). +The last revision associated with the user’s operating system will be downloaded +(this revision is obtained by the LAST_CHANGE designation from the snapshots bucket). +Chromium does not have varying channels, so the installation uses the default `nightly` +designation. The install path is `_venv3/browsers/nightly/{chromium_binary}`. + +Note: If this download process is successful, the Chromium snapshot URL that the browser +binary was downloaded from will be kept during the current invocation. If a Chromium ChromeDriver +is also downloaded later to match this browser binary, the same URL is used for that download to +ensure both components are downloaded from the same source. + +**WebDriver**: A version of ChromeDriver will only be installed once a Chromium browser binary +has been given or detected. A FileNotFoundError will be raised if the user tries to download +ChromeDriver via the install command and a browser binary is not located. A version of +ChromeDriver that matches the version of the browser binary will be installed. The download +source for this ChromeDriver will be the Chromium snapshots. If a Chromium browser +binary and webdriver are installed in the same invocation of `./wpt run` +(for example, by passing both `--install-browser` and `--install-webdriver` flags), then the +browser binary and ChromeDriver will be pulled from the same Chromium snapshots URL (see Note +from browser installation). Although unusual, if a Chromium browser binary is detected and +it is not the tip-of-tree revision and the browser binary was not downloaded and installed +during this invocation of `./wpt run` and the currently installed ChromeDriver version does +not match the browser version, then an attempt will be made to detect the revision number from +the browser binary version using the [OmahaProxy](https://omahaproxy.appspot.com/) +and download the matching ChromeDriver using this revision number from Chromium snapshots. diff --git a/testing/web-platform/tests/docs/running-tests/chrome.md b/testing/web-platform/tests/docs/running-tests/chrome.md new file mode 100644 index 0000000000..45293af65e --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/chrome.md @@ -0,0 +1,33 @@ +# Chrome and Chromium + +When running Chrome, there are some useful command line arguments. + +You can inform `wpt` of the release channel of Chrome using `--channel`. +`wpt` is able to find the correct binary in the following cases: +* On Linux for stable, beta and dev channels if + `google-chrome-{stable,beta,unstable}` are in `PATH`; +* On Mac for stable and canary channels if the official DMGs are installed. + +In other cases, you will need to specify the path to the Chrome binary with +`--binary`. For example: + +```bash +./wpt run --channel dev --binary /path/to/non-default/google-chrome chrome +``` + +Note: when the channel is "dev", `wpt` will *automatically* enable all +[experimental web platform features][1] +(chrome://flags/#enable-experimental-web-platform-features) by passing +`--enable-experimental-web-platform-features` to Chrome. + +If you want to enable a specific [runtime enabled feature][1], use +`--binary-arg` to specify the flag(s) that you want to pass to Chrome: + +```bash +./wpt run --binary-arg=--enable-blink-features=AsyncClipboard chrome clipboard-apis/ +``` + +[A detailed explanation is available](running-tests/chrome-chromium-installation-detection.html) +for more information on how wpt detects and installs the components for Chrome and Chromium. + +[1]: https://chromium.googlesource.com/chromium/src/+/main/third_party/blink/renderer/platform/RuntimeEnabledFeatures.md diff --git a/testing/web-platform/tests/docs/running-tests/chrome_android.md b/testing/web-platform/tests/docs/running-tests/chrome_android.md new file mode 100644 index 0000000000..a216a8a68b --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/chrome_android.md @@ -0,0 +1,22 @@ +# Chrome for Android + +To run WPT on Chrome on an Android device, some additional set up is required. + +As with usual Android development, you need to have `adb` and be able to +connect to the device. Run `adb devices` to verify. + +Currently, Android support is a prototype with some known issues: + +* If you have previously run `./wpt run` against Chrome, you might need to + remove `_venv/bin/chromedriver` so that we can install the correct + ChromeDriver corresponding to your Chrome for Android version. +* We do not support reftests at the moment. +* You will need to manually kill Chrome (all channels) before running tests. + +Note: rooting the device or installing a root CA is no longer required. + +Example (assuming you have Chrome Canary installed on your phone): + +```bash +./wpt run --test-type=testharness --channel=canary chrome_android TESTS +``` diff --git a/testing/web-platform/tests/docs/running-tests/command-line-arguments.md b/testing/web-platform/tests/docs/running-tests/command-line-arguments.md new file mode 100644 index 0000000000..598c9da2a1 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/command-line-arguments.md @@ -0,0 +1,14 @@ +# Command-Line Arguments + +The `wpt` command-line application offers a number of features for interacting +with WPT. The functionality is organized into "sub-commands", and each accepts +a different set of command-line arguments. + +This page documents all of the available sub-commands and associated arguments. + +```eval_rst +.. argparse:: + :module: tools.wpt.wpt + :func: create_complete_parser + :prog: wpt +``` diff --git a/testing/web-platform/tests/docs/running-tests/custom-runner.md b/testing/web-platform/tests/docs/running-tests/custom-runner.md new file mode 100644 index 0000000000..029a8771c1 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/custom-runner.md @@ -0,0 +1,21 @@ +# Writing Your Own Runner + +Most test runners have two stages: finding all tests, followed by +executing them (or a subset thereof). + +To find all tests in the repository, it is **strongly** recommended to +use the included `wpt manifest` tool: the required behaviors are more +complex than what are documented (especially when it comes to +precedence of the various possibilities and some undocumented legacy +ways to define test types), and hence its behavior should be +considered the canonical definition of how to enumerate tests and find +their type in the repository. + +For test execution, please read the documentation for the various test +types very carefully and then check your understanding on the [mailing +list][public-test-infra] or [matrix][matrix]. It's possible edge-case +behavior isn't properly documented! + +[public-test-infra]: https://lists.w3.org/Archives/Public/public-test-infra/ +[matrix]: https://app.element.io/#/room/#wpt:matrix.org +[web irc]: http://irc.w3.org diff --git a/testing/web-platform/tests/docs/running-tests/from-ci.md b/testing/web-platform/tests/docs/running-tests/from-ci.md new file mode 100644 index 0000000000..9ea142bb4b --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/from-ci.md @@ -0,0 +1,32 @@ +# Running Tests on CI + +Contributors with write access to the repository can trigger full runs in the +same CI systems used to produce results for [wpt.fyi](https://wpt.fyi). The runs +are triggered by pushing to branch names on the form `triggers/$browser_$channel` +and the results will be automatically submitted to wpt.fyi. + +This is useful when making infrastructure changes that could affect very many +tests, in order to avoid regressions. + +Note: Full runs use a lot of CI resources, so please take care to not trigger +them more than necessary. + +Instructions: + + * Base your changes on a commit for which there are already results in wpt.fyi. + + * Determine which branch name to push to by looking for `refs/heads/triggers/` + in `.azure-pipelines.yml` and `.taskcluster.yml`. For example, to trigger a + full run of Safari Technology Preview, the branch name is + `triggers/safari_preview`. + + * Force push to the branch, for example: + `git push --force-with-lease origin HEAD:triggers/safari_preview`. + The `--force-with-lease` argument is to detect if someone else has just + pushed. When this happens wait for the checkout step of their triggered run + to finish before you force push again. + +You can see if the run started from the commit status on GitHub's commits listing +([example](https://github.com/web-platform-tests/wpt/commits/triggers/safari_preview)) +and if successful the results will show up on wpt.fyi within 10 minutes +([example](https://wpt.fyi/runs?product=safari)). diff --git a/testing/web-platform/tests/docs/running-tests/from-local-system.md b/testing/web-platform/tests/docs/running-tests/from-local-system.md new file mode 100644 index 0000000000..3865038ef6 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/from-local-system.md @@ -0,0 +1,215 @@ +# Running Tests from the Local System + +The tests are designed to be run from your local computer. + +## System Setup + +Running the tests requires `python`, `pip` and `virtualenv`, as well as updating +the system `hosts` file. + +WPT requires Python 3.7 or higher. + +The required setup is different depending on your operating system. + +### Linux Setup + +If not already present, use the system package manager to install `python`, +`pip` and `virtualenv`. + +On Debian or Ubuntu: + +```bash +sudo apt-get install python python-pip virtualenv +``` + +It is important to have a package that provides a `python` binary. On Fedora, +for example, that means installing the `python-unversioned-command` package. On +Ubuntu Focal and later, the package is called `python-is-python3`. + +### macOS Setup + +The system-provided Python can be used, while `pip` and `virtualenv` can be +installed for the user only: + +```bash +python -m ensurepip --user +export PATH="$PATH:$( python3 -m site --user-base )/bin" +pip install --user virtualenv +``` + +To make the `PATH` change persistent, add it to your `~/.bash_profile` file or +wherever you currently set your PATH. + +See also [additional setup required to run Safari](safari.md). + +### Windows Setup + +Download and install [Python 3](https://www.python.org/downloads). The +installer includes `pip` by default. + +Add `C:\Python39` and `C:\Python39\Scripts` to your `%Path%` +[environment variable](http://www.computerhope.com/issues/ch000549.htm). + +Finally, install `virtualenv`: + +```bash +pip install virtualenv +``` + +The standard Windows shell requires that all `wpt` commands are prefixed +by the Python binary i.e. assuming `python` is on your path the server is +started using: + +```bash +python wpt serve +``` + +#### Windows Subsystem for Linux + +Optionally on Windows you can use the [Windows Subsystem for +Linux](https://docs.microsoft.com/en-us/windows/wsl/about) (WSL). If doing so, +installation and usage are similar to the Linux instructions. Be aware that WSL +may attempt to override `/etc/hosts` each time it is launched, which would then +require you to re-run [`hosts` File Setup](#hosts-file-setup). This behavior +[can be configured](https://docs.microsoft.com/en-us/windows/wsl/wsl-config#network). + +### `hosts` File Setup + +To get the tests running, you need to set up the test domains in your +[`hosts` file](http://en.wikipedia.org/wiki/Hosts_%28file%29%23Location_in_the_file_system). + +On Linux, macOS or other UNIX-like system: + +```bash +./wpt make-hosts-file | sudo tee -a /etc/hosts +``` + +And on Windows (this must be run in a PowerShell session with Administrator privileges): + +``` +python wpt make-hosts-file | Out-File $env:SystemRoot\System32\drivers\etc\hosts -Encoding ascii -Append +``` + +If you are behind a proxy, you also need to make sure the domains above are +excluded from your proxy lookups. + +## Via the browser + +The test environment can then be started using + + ./wpt serve + +This will start HTTP servers on two ports and a websockets server on +one port. By default the web servers start on ports 8000 and 8443 and the other +ports are randomly-chosen free ports. Tests must be loaded from the +*first* HTTP server in the output. To change the ports, +create a `config.json` file in the wpt root directory, and add +port definitions of your choice e.g.: + +``` +{ + "ports": { + "http": [1234, "auto"], + "https":[5678] + } +} +``` + +After your `hosts` file is configured, the servers will be locally accessible at: + +http://web-platform.test:8000/<br> +https://web-platform.test:8443/ * + +To use the web-based runner point your browser to: + +http://web-platform.test:8000/tools/runner/index.html<br> +https://web-platform.test:8443/tools/runner/index.html * + +This server has all the capabilities of the publicly-deployed version--see +[Running the Tests from the Web](from-web.md). + +\**See [Trusting Root CA](../tools/certs/README.md)* + +## Via the command line + +Many tests can be automatically executed in a new browser instance using + + ./wpt run [browsername] [tests] + +This will automatically load the tests in the chosen browser and extract the +test results. For example to run the `dom/historical.html` tests in a local +copy of Chrome: + + ./wpt run chrome dom/historical.html + +Or to run in a specified copy of Firefox: + + ./wpt run --binary ~/local/firefox/firefox firefox dom/historical.html + +For details on the supported products and a large number of other options for +customising the test run: + + ./wpt run --help + +[A complete listing of the command-line arguments is available +here](command-line-arguments.html#run). + +```eval_rst +.. toctree:: + :hidden: + + command-line-arguments +``` + +### Browser-specific instructions + +```eval_rst +.. toctree:: + + chrome + chrome_android + android_webview + safari + webkitgtk_minibrowser +``` + +### Running in parallel + +To speed up the testing process, use the `--processes` option to run multiple +browser instances in parallel. For example, to run the tests in dom/ with six +Firefox instances in parallel: + + ./wpt run --processes=6 firefox dom/ + +But note that behaviour in this mode is necessarily less deterministic than with +a single process (the default), so there may be more noise in the test results. + +### Output formats + +By default, `./wpt run` outputs test results and a summary in a human readable +format. For debugging, `--log-mach` can give more verbose output. For example: + + ./wpt run --log-mach=- --log-mach-level=info firefox dom/ + +A machine readable JSON report can be produced using `--log-wptreport`. This +together with `--log-wptscreenshot` is what is used to produce results for +[wpt.fyi](https://wpt.fyi). For example: + + ./wpt run --log-wptreport=report.json --log-wptscreenshot=screenshots.txt firefox css/css-grid/ + +(See [wpt.fyi documentation](https://github.com/web-platform-tests/wpt.fyi/blob/main/api/README.md#results-creation) +for how results are uploaded.) + +### Expectation data + +For use in continuous integration systems, and other scenarios where regression +tracking is required, the command-line interface supports storing and loading +the expected result of each test in a test run. See [Expectations +Data](../../tools/wptrunner/docs/expectation) for more information on creating +and maintaining these files. + +## Testing polyfills + +Polyfill scripts can be tested using the `--inject-script` argument to either +`wpt run` or `wpt serve`. See [Testing Polyfills](testing-polyfills) for +details. diff --git a/testing/web-platform/tests/docs/running-tests/from-web.md b/testing/web-platform/tests/docs/running-tests/from-web.md new file mode 100644 index 0000000000..157598da36 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/from-web.md @@ -0,0 +1,27 @@ +# Running Tests from the Web + +Tests that have been merged on GitHub are mirrored at +[wpt.live](https://wpt.live) and [w3c-test.org](https://w3c-test.org). +[On properly-configured systems](from-local-system), local files may also be +served from the URL [http://web-platform.test](http://web-platform.test). + +Not all tests can be executed in-browser, as some tests rely on automation +(e.g. via [testdriver.js](../writing-tests/testdriver)) that is not available +when running a browser in a normal user session. + +## Web test runner + +For running multiple tests inside a browser, there is a test runner +located at `/tools/runner/index.html`. + +This allows all the tests, or those matching a specific prefix +(e.g. all tests under `/dom/`) to be run. For testharness.js tests, +the results will be automatically collected, while the runner +provides a simple UI for manually comparing reftest rendering and +running manual tests. + +Note, however, it does not currently handle more complex reftests with +more than one reference involved. + +Because it runs entirely in-browser, this runner cannot deal with +edge-cases like tests that cause the browser to crash or hang. diff --git a/testing/web-platform/tests/docs/running-tests/index.md b/testing/web-platform/tests/docs/running-tests/index.md new file mode 100644 index 0000000000..17b361dde8 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/index.md @@ -0,0 +1,24 @@ +# Running Tests + +```eval_rst +.. toctree:: + + from-web + from-local-system + from-ci + custom-runner + ../tools/certs/README.md +``` + +The simplest way to run the tests is via the public website. More detail on +that approach is available in [Running tests from the Web](from-web). + +Contributors who are interested in modifying and creating tests should refer to +[Running Tests from the Local System](from-local-system). + +Contributors with write access to the repository can also trigger full runs +in our CI setups, see [Running Tests on CI](from-ci). + +Advanced use cases may call for a customized method of executing the tests. +Guidelines for writing a custom "runner" are available at [Writing Your Own +Runner](custom-runner). diff --git a/testing/web-platform/tests/docs/running-tests/safari.md b/testing/web-platform/tests/docs/running-tests/safari.md new file mode 100644 index 0000000000..eed5225471 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/safari.md @@ -0,0 +1,47 @@ +# Safari + +To run Safari on macOS, some manual setup is required. Some steps are different +for Safari and Safari Technology Preview. + + * Allow Safari to be controlled by SafariDriver: + * `safaridriver --enable` or + * `"/Applications/Safari Technology Preview.app/Contents/MacOS/safaridriver" --enable` + + * Allow pop-up windows: + * `defaults write com.apple.Safari WebKitJavaScriptCanOpenWindowsAutomatically 1` or + * `defaults write com.apple.SafariTechnologyPreview WebKitJavaScriptCanOpenWindowsAutomatically 1` + + * Turn on additional experimental features Safari Technology Preview: + * `defaults write com.apple.SafariTechnologyPreview ExperimentalServerTimingEnabled 1` + + * Trust the certificate: + * `security add-trusted-cert -k "$(security default-keychain | cut -d\" -f2)" tools/certs/cacert.pem` + + * Set `no_proxy='*'` in your environment. This is a + workaround for a known + [macOS High Sierra issue](https://github.com/web-platform-tests/wpt/issues/9007). + +Now, run the tests using the `safari` product: +``` +./wpt run safari [test_list] +``` + +This will use the `safaridriver` found on the path, which will be stable Safari. +To run Safari Technology Preview instead, use the `--channel=preview` argument: +``` +./wpt run --channel=preview safari [test_list] +``` + +## Debugging + +To debug problems with `safaridriver`, add the `--webdriver-arg=--diagnose` +argument: +``` +./wpt run --channel=preview --webdriver-arg=--diagnose safari [test_list] +``` + +The logs will be in `~/Library/Logs/com.apple.WebDriver/`. +See `man safaridriver` for more information. + +To enable safaridriver diagnostics in Azure Pipelines, set +`safaridriver_diagnose` to `true` in `.azure-pipelines.yml`. diff --git a/testing/web-platform/tests/docs/running-tests/testing-polyfills.md b/testing/web-platform/tests/docs/running-tests/testing-polyfills.md new file mode 100644 index 0000000000..468bfa2e03 --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/testing-polyfills.md @@ -0,0 +1,67 @@ +# Testing polyfills + +## Preparing the polyfill + +The polyfill script-injection feature currently only supports scripts which +are immediately invoked. The script must be prepared as a single file whose +contents will be inlined into a script tag served as part of every test page. + +If your polyfill is only available as an asynchronous module with dependent +scripts, you can use a tool such as +[microbundle](https://github.com/developit/microbundle) to repackage it as a +single synchronous script file, e.g.: + +```bash +microbundle -f iife -i polyfill/src/main.js -o polyfill.js +``` + +## Running the tests + +Follow the steps for [Running Tests from the Local System](from-local-system) to +set up your test environment. When running tests via the browser or via the +command line, add the `--inject-script=polyfill.js` to either command, e.g. + +Via the browser: + +```bash +./wpt serve --inject-script=polyfill.js +``` + +Then visit http://web-platform.test:8000/ or https://web-platform.test:8443/ to +run the tests in your browser. + +Via the command line: + +```bash +./wpt run --inject-script=polyfill.js [browsername] [tests] +``` + +## Limitations + +Polyfill scripts are injected to an inline script tag which removes itself from +the DOM after executing. This is done by modifying the server response for +documents with a `text/html` MIME type to insert the following before the first tag in +the served response: + +```html +<script> +// <-- The polyfill file is inlined here +// Remove the injected script tag from the DOM. +document.currentScript.remove(); +``` + +This approach has a couple limitations: +* This requires that the polyfill is self-contained and executes +synchronously in a single inline script. See [Preparing the +polyfill](#preparing-the-polyfill) for suggestions on transforming polyfills to +run in that way. +* Does not inject into python handlers which write directly to the output + stream. +* Does not inject into the worker context of `.any.js` tests. + +### Observability + +The script tag is removed from the DOM before any other script has run, and runs +from an inline script. As such, it should not affect mutation observers on the +same page or resource timing APIs, as it is not a separate resource. The polyfill +may be observable by a mutation observer added by a parent frame before load. diff --git a/testing/web-platform/tests/docs/running-tests/webkitgtk_minibrowser.md b/testing/web-platform/tests/docs/running-tests/webkitgtk_minibrowser.md new file mode 100644 index 0000000000..7aac81e5fc --- /dev/null +++ b/testing/web-platform/tests/docs/running-tests/webkitgtk_minibrowser.md @@ -0,0 +1,20 @@ +# WebKitGTK MiniBrowser + + +To be able to run tests with the WebKitGTK MiniBrowser you need the +following packages installed: + +* Fedora: `webkit2gtk3-devel` +* Debian or Ubuntu: `webkit2gtk-driver` + + +The WebKitGTK MiniBrowser is not installed on the default binary path. +The `wpt` script will try to automatically locate it, but if you need +to run it manually you can find it on any of this paths: + +* Fedora: `/usr/libexec/webkit2gtk-4.0/MiniBrowser` +* Debian or Ubuntu: `/usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/MiniBrowser` + * Note: if the machine architecture is not `x86_64`, then it will be located + inside: + `/usr/lib/${TRIPLET}/webkit2gtk-4.0/MiniBrowser` + where `TRIPLET=$(gcc -dumpmachine)` |