From 36d22d82aa202bb199967e9512281e9a53db42c9 Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sun, 7 Apr 2024 21:33:14 +0200 Subject: Adding upstream version 115.7.0esr. Signed-off-by: Daniel Baumann --- dom/webgpu/tests/cts/checkout/docs/organization.md | 166 +++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 dom/webgpu/tests/cts/checkout/docs/organization.md (limited to 'dom/webgpu/tests/cts/checkout/docs/organization.md') diff --git a/dom/webgpu/tests/cts/checkout/docs/organization.md b/dom/webgpu/tests/cts/checkout/docs/organization.md new file mode 100644 index 0000000000..fd7020afd6 --- /dev/null +++ b/dom/webgpu/tests/cts/checkout/docs/organization.md @@ -0,0 +1,166 @@ +# Test Organization + +## `src/webgpu/` + +Because of the glorious amount of test needed, the WebGPU CTS is organized as a tree of arbitrary +depth (a filesystem with multiple tests per file). + +Each directory may have a `README.txt` describing its contents. +Tests are grouped in large families (each of which has a `README.txt`); +the root and first few levels looks like the following (some nodes omitted for simplicity): + +- **`api`** with tests for full coverage of the Javascript API surface of WebGPU. + - **`validation`** with positive and negative tests for all the validation rules of the API. + - **`operation`** with tests that checks the result of performing valid WebGPU operations, + taking advantage of parametrization to exercise interactions between parts of the API. + - **`regression`** for one-off tests that reproduce bugs found in implementations to prevent + the bugs from appearing again. +- **`shader`** with tests for full coverage of the shaders that can be passed to WebGPU. + - **`validation`**. + - **`execution`** similar to `api/operation`. + - **`regression`**. +- **`idl`** with tests to check that the WebGPU IDL is correctly implemented, for examples that + objects exposed exactly the correct members, and that methods throw when passed incomplete + dictionaries. +- **`web-platform`** with tests for Web platform-specific interactions like `GPUSwapChain` and + ``, WebXR and `GPUQueue.copyExternalImageToTexture`. + +At the same time test hierarchies can be used to split the testing of a single sub-object into +several file for maintainability. For example `GPURenderPipeline` has a large descriptor and some +parts could be tested independently like `vertex_input` vs. `primitive_topology` vs. `blending` +but all live under the `render_pipeline` directory. + +In addition to the test tree, each test can be parameterized. For coverage it is important to +test all enums values, for example for `GPUTextureFormat`. Instead of having a loop to iterate +over all the `GPUTextureFormat`, it is better to parameterize the test over them. Each format +will have a different entry in the test list which will help WebGPU implementers debug the test, +or suppress the failure without losing test coverage while they fix the bug. + +Extra capabilities (limits and features) are often tested in the same files as the rest of the API. +For example, a compressed texture format capability would simply add a `GPUTextureFormat` to the +parametrization lists of many tests, while a capability adding significant new functionality +like ray-tracing could have a separate subtree. + +Operation tests for optional features should be skipped using `t.selectDeviceOrSkipTestCase()` or +`t.skip()`. Validation tests should be written that test the behavior with and without the +capability enabled via `t.selectDeviceOrSkipTestCase()`, to ensure the functionality is valid +only with the capability enabled. + +### Validation tests + +Validation tests check the validation rules that are (or will be) set by the +WebGPU spec. Validation tests try to carefully trigger the individual validation +rules in the spec, without simultaneously triggering other rules. + +Validation errors *generally* generate WebGPU errors, not exceptions. +But check the spec on a case-by-case basis. + +Like all `GPUTest`s, `ValidationTest`s are wrapped in both types of error scope. These +"catch-all" error scopes look for any errors during the test, and report them as test failures. +Since error scopes can be nested, validation tests can nest an error scope to expect that there +*are* errors from specific operations. + +#### Parameterization + +Test parameterization can help write many validation tests more succinctly, +while making it easier for both authors and reviewers to be confident that +an aspect of the API is tested fully. Examples: + +- [`webgpu:api,validation,render_pass,resolve:resolve_attachment:*`](https://github.com/gpuweb/cts/blob/ded3b7c8a4680a1a01621a8ac859facefadf32d0/src/webgpu/api/validation/render_pass/resolve.spec.ts#L35) +- [`webgpu:api,validation,createBindGroupLayout:bindingTypeSpecific_optional_members:*`](https://github.com/gpuweb/cts/blob/ded3b7c8a4680a1a01621a8ac859facefadf32d0/src/webgpu/api/validation/createBindGroupLayout.spec.ts#L68) + +Use your own discretion when deciding the balance between heavily parameterizing +a test and writing multiple separate tests. + +#### Guidelines + +There are many aspects that should be tested in all validation tests: + +- each individual argument to a method call (including `this`) or member of a descriptor + dictionary should be tested including: + - what happens when an error object is passed. + - what happens when an optional feature enum or method is used. + - what happens for numeric values when they are at 0, too large, too small, etc. +- each validation rule in the specification should be checked both with a control success case, + and error cases. +- each set of arguments or state that interact for validation. + +When testing numeric values, it is important to check on both sides of the boundary: if the error +happens for value N and not N - 1, both should be tested. Alignment of integer values should also +be tested but boundary testing of alignment should be between a value aligned to 2^N and a value +aligned to 2^(N-1). + +Finally, this is probably also where we would test that extensions follow the rule that: if the +browser supports a feature but it is not enabled on the device, then calling methods from that +feature throws `TypeError`. + +- Test providing unknown properties *that are definitely not part of any feature* are + valid/ignored. (Unfortunately, due to the rules of IDL, adding a member to a dictionary is + always a breaking change. So this is how we have to test this unless we can get a "strict" + dictionary type in IDL. We can't test adding members from non-enabled extensions.) + +### Operation tests + +Operation tests test the actual results of using the API. They execute +(sometimes significant) code and check that the result is within the expected +set of behaviors (which can be quite complex to compute). + +Note that operation tests need to test a lot of interactions between different +parts of the API, and so can become quite complex. Try to reduce the complexity by +utilizing combinatorics and [helpers](./helper_index.txt), and splitting/merging test files as needed. + +#### Errors + +Operation tests are usually `GPUTest`s. As a result, they automatically fail on any validation +errors that occur during the test. + +When it's easier to write an operation test with invalid cases, use +`ParamsBuilder.filter`/`.unless` to avoid invalid cases, or detect and +`expect` validation errors in some cases. + +#### Implementation + +Use helpers like `expectContents` (and more to come) to check the values of data on the GPU. +(These are "eventual expectations" - the harness will wait for them to finish at the end). + +When testing something inside a shader, it's not always necessary to output the result to a +render output. In fragment shaders, you can output to a storage buffer. In vertex shaders, you +can't - but you can render with points (simplest), send the result to the fragment shader, and +output it from there. (Someday, we may end up wanting a helper for this.) + +#### Testing Default Values + +Default value tests (for arguments and dictionary members) should usually be operation tests - +all you have to do is include `undefined` in parameterizations of other tests to make sure the +behavior with `undefined` has the same expected result that you have when the default value is +specified explicitly. + +### IDL tests + +TODO: figure out how to implement these. https://github.com/gpuweb/cts/issues/332 + +These tests test only rules that come directly from WebIDL. For example: + +- Values out of range for `[EnforceRange]` cause exceptions. +- Required function arguments and dictionary members cause exceptions if omitted. +- Arguments and dictionary members cause exceptions if passed the wrong type. + +They may also test positive cases like the following, but the behavior of these should be tested in +operation tests. + +- OK to omit optional arguments/members. +- OK to pass the correct argument/member type (or of any type in a union type). + +Every overload of every method should be tested. + +## `src/stress/`, `src/manual/` + +Stress tests and manual tests for WebGPU that are not intended to be run in an automated way. + +## `src/unittests/` + +Unit tests for the test framework (`src/common/framework/`). + +## `src/demo/` + +A demo of test hierarchies for the purpose of testing the `standalone` test runner page. -- cgit v1.2.3