1 files changed, 166 insertions, 0 deletions
diff --git a/dom/webgpu/tests/cts/checkout/docs/organization.md b/dom/webgpu/tests/cts/checkout/docs/organization.md
new file mode 100644
index 0000000000..fd7020afd6
--- /dev/null
+++ b/dom/webgpu/tests/cts/checkout/docs/organization.md
@@ -0,0 +1,166 @@
+# Test Organization
+
+## `src/webgpu/`
+
+Because of the glorious amount of test needed, the WebGPU CTS is organized as a tree of arbitrary
+depth (a filesystem with multiple tests per file).
+
+Each directory may have a `README.txt` describing its contents.
+Tests are grouped in large families (each of which has a `README.txt`);
+the root and first few levels looks like the following (some nodes omitted for simplicity):
+
+- **`api`** with tests for full coverage of the Javascript API surface of WebGPU.
+    - **`validation`** with positive and negative tests for all the validation rules of the API.
+    - **`operation`** with tests that checks the result of performing valid WebGPU operations,
+      taking advantage of parametrization to exercise interactions between parts of the API.
+    - **`regression`** for one-off tests that reproduce bugs found in implementations to prevent
+      the bugs from appearing again.
+- **`shader`** with tests for full coverage of the shaders that can be passed to WebGPU.
+    - **`validation`**.
+    - **`execution`** similar to `api/operation`.
+    - **`regression`**.
+- **`idl`** with tests to check that the WebGPU IDL is correctly implemented, for examples that
+  objects exposed exactly the correct members, and that methods throw when passed incomplete
+  dictionaries.
+- **`web-platform`** with tests for Web platform-specific interactions like `GPUSwapChain` and
+  `<canvas>`, WebXR and `GPUQueue.copyExternalImageToTexture`.
+
+At the same time test hierarchies can be used to split the testing of a single sub-object into
+several file for maintainability. For example `GPURenderPipeline` has a large descriptor and some
+parts could be tested independently like `vertex_input` vs. `primitive_topology` vs. `blending`
+but all live under the `render_pipeline` directory.
+
+In addition to the test tree, each test can be parameterized. For coverage it is important to
+test all enums values, for example for `GPUTextureFormat`. Instead of having a loop to iterate
+over all the `GPUTextureFormat`, it is better to parameterize the test over them. Each format
+will have a different entry in the test list which will help WebGPU implementers debug the test,
+or suppress the failure without losing test coverage while they fix the bug.
+
+Extra capabilities (limits and features) are often tested in the same files as the rest of the API.
+For example, a compressed texture format capability would simply add a `GPUTextureFormat` to the
+parametrization lists of many tests, while a capability adding significant new functionality
+like ray-tracing could have a separate subtree.
+
+Operation tests for optional features should be skipped using `t.selectDeviceOrSkipTestCase()` or
+`t.skip()`. Validation tests should be written that test the behavior with and without the
+capability enabled via `t.selectDeviceOrSkipTestCase()`, to ensure the functionality is valid
+only with the capability enabled.
+
+### Validation tests
+
+Validation tests check the validation rules that are (or will be) set by the
+WebGPU spec. Validation tests try to carefully trigger the individual validation
+rules in the spec, without simultaneously triggering other rules.
+
+Validation errors *generally* generate WebGPU errors, not exceptions.
+But check the spec on a case-by-case basis.
+
+Like all `GPUTest`s, `ValidationTest`s are wrapped in both types of error scope. These
+"catch-all" error scopes look for any errors during the test, and report them as test failures.
+Since error scopes can be nested, validation tests can nest an error scope to expect that there
+*are* errors from specific operations.
+
+#### Parameterization
+
+Test parameterization can help write many validation tests more succinctly,
+while making it easier for both authors and reviewers to be confident that
+an aspect of the API is tested fully. Examples:
+
+- [`webgpu:api,validation,render_pass,resolve:resolve_attachment:*`](https://github.com/gpuweb/cts/blob/ded3b7c8a4680a1a01621a8ac859facefadf32d0/src/webgpu/api/validation/render_pass/resolve.spec.ts#L35)
+- [`webgpu:api,validation,createBindGroupLayout:bindingTypeSpecific_optional_members:*`](https://github.com/gpuweb/cts/blob/ded3b7c8a4680a1a01621a8ac859facefadf32d0/src/webgpu/api/validation/createBindGroupLayout.spec.ts#L68)
+
+Use your own discretion when deciding the balance between heavily parameterizing
+a test and writing multiple separate tests.
+
+#### Guidelines
+
+There are many aspects that should be tested in all validation tests:
+
+- each individual argument to a method call (including `this`) or member of a descriptor
+  dictionary should be tested including:
+    - what happens when an error object is passed.
+    - what happens when an optional feature enum or method is used.
+    - what happens for numeric values when they are at 0, too large, too small, etc.
+- each validation rule in the specification should be checked both with a control success case,
+  and error cases.
+- each set of arguments or state that interact for validation.
+
+When testing numeric values, it is important to check on both sides of the boundary: if the error
+happens for value N and not N - 1, both should be tested. Alignment of integer values should also
+be tested but boundary testing of alignment should be between a value aligned to 2^N and a value
+aligned to 2^(N-1).
+
+Finally, this is probably also where we would test that extensions follow the rule that: if the
+browser supports a feature but it is not enabled on the device, then calling methods from that
+feature throws `TypeError`.
+
+- Test providing unknown properties *that are definitely not part of any feature* are
+  valid/ignored. (Unfortunately, due to the rules of IDL, adding a member to a dictionary is
+  always a breaking change. So this is how we have to test this unless we can get a "strict"
+  dictionary type in IDL. We can't test adding members from non-enabled extensions.)
+
+### Operation tests
+
+Operation tests test the actual results of using the API. They execute
+(sometimes significant) code and check that the result is within the expected
+set of behaviors (which can be quite complex to compute).
+
+Note that operation tests need to test a lot of interactions between different
+parts of the API, and so can become quite complex. Try to reduce the complexity by
+utilizing combinatorics and [helpers](./helper_index.txt), and splitting/merging test files as needed.
+
+#### Errors
+
+Operation tests are usually `GPUTest`s. As a result, they automatically fail on any validation
+errors that occur during the test.
+
+When it's easier to write an operation test with invalid cases, use
+`ParamsBuilder.filter`/`.unless` to avoid invalid cases, or detect and
+`expect` validation errors in some cases.
+
+#### Implementation
+
+Use helpers like `expectContents` (and more to come) to check the values of data on the GPU.
+(These are "eventual expectations" - the harness will wait for them to finish at the end).
+
+When testing something inside a shader, it's not always necessary to output the result to a
+render output. In fragment shaders, you can output to a storage buffer. In vertex shaders, you
+can't - but you can render with points (simplest), send the result to the fragment shader, and
+output it from there. (Someday, we may end up wanting a helper for this.)
+
+#### Testing Default Values
+
+Default value tests (for arguments and dictionary members) should usually be operation tests -
+all you have to do is include `undefined` in parameterizations of other tests to make sure the
+behavior with `undefined` has the same expected result that you have when the default value is
+specified explicitly.
+
+### IDL tests
+
+TODO: figure out how to implement these. https://github.com/gpuweb/cts/issues/332
+
+These tests test only rules that come directly from WebIDL. For example:
+
+- Values out of range for `[EnforceRange]` cause exceptions.
+- Required function arguments and dictionary members cause exceptions if omitted.
+- Arguments and dictionary members cause exceptions if passed the wrong type.
+
+They may also test positive cases like the following, but the behavior of these should be tested in
+operation tests.
+
+- OK to omit optional arguments/members.
+- OK to pass the correct argument/member type (or of any type in a union type).
+
+Every overload of every method should be tested.
+
+## `src/stress/`, `src/manual/`
+
+Stress tests and manual tests for WebGPU that are not intended to be run in an automated way.
+
+## `src/unittests/`
+
+Unit tests for the test framework (`src/common/framework/`).
+
+## `src/demo/`
+
+A demo of test hierarchies for the purpose of testing the `standalone` test runner page.