1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
|
# Adding Timing Metadata
## listing_meta.json files
`listing_meta.json` files are SEMI AUTO-GENERATED.
The raw data may be edited manually, to add entries or change timing values.
The list of tests in this file is **not** guaranteed to stay up to date.
Use the generated `gen/*_variant_list*.json` if you need a complete list.
The `subcaseMS` values are estimates. They can be set to 0 or omitted if for some reason
you can't estimate the time (or there's an existing test with a long name and
slow subcases that would result in query strings that are too long).
It's OK if the number is estimated too high.
These entries are estimates for the amount of time that subcases take to run,
and are used as inputs into the WPT tooling to attempt to portion out tests into
approximately same-sized chunks. High estimates are OK, they just may generate
more chunks than necessary.
To check for missing or 0 entries, run
`tools/validate --print-metadata-warnings src/webgpu`
and look at the resulting warnings.
### Performance
Note this data is typically captured by developers using higher-end
computers, so typical test machines might execute more slowly. For this
reason, the WPT chunking should be configured to generate chunks much shorter
than 5 seconds (a typical default time limit in WPT test executors) so they
should still execute in under 5 seconds on lower-end computers.
## Problem
When renaming or removing tests from the CTS you will see an error like this
when running `npm test` or `npm run standalone`:
```
ERROR: Non-existent tests found in listing_meta.json. Please update:
webgpu:api,operation,adapter,requestAdapter:old_test_that_got_renamed:*
```
## Solution
This means there is a stale line in `src/webgpu/listing_meta.json` that needs
to be deleted, or updated to match the rename that you did.
## Problem
You run `tools/validate --print-metadata-warnings src/webgpu`
and want to fix the warnings.
## Solution 1 (manual, best for one-off updates of simple tests)
If you're developing new tests and need to update this file, it is sometimes
easiest to do so manually. Run your tests under your usual development workflow
and see how long they take. In the standalone web runner `npm start`, the total
time for a test case is reported on the right-hand side when the case logs are
expanded.
Record the average time per *subcase* across all cases of the test (you may need
to compute this) into the `listing_meta.json` file.
## Solution 2 (semi-automated)
There exists tooling in the CTS repo for generating appropriate estimates for
these values, though they do require some manual intervention. The rest of this
doc will be a walkthrough of running these tools.
Timing data can be captured in bulk and "merged" into this file using
the `merge_listing_times` tool. This is
This is useful when a large number of tests
change or otherwise a lot of tests need to be updated, but it also automates the
manual steps above.
The tool can also be used without any inputs to reformat `listing_meta.json`.
Please read the help message of `merge_listing_times` for more information.
### Websocket Logger
The first tool that needs to be run is `websocket-logger`, which receives data
on a WebSocket channel at `localhost:59497` to capture timing data when CTS is run. This
should be run in a separate process/terminal, since it needs to stay running
throughout the following steps.
In the `tools/websocket-logger/` directory:
```
npm ci
npm start
```
The output from this command will indicate where the results are being logged,
which will be needed later. For example:
```
...
Writing to wslog-2023-09-12T18-57-34.txt
...
```
See also [tools/websocket-logger/README.md](../tools/websocket-logger/README.md).
### Running CTS
Now we need to run the specific cases in CTS that we need to time.
This should be possible under any development workflow by logging through a
side-channel (as long as its runtime environment, like Node, supports WebSockets).
Regardless of development workflow, you need to enable logToWebSocket flag
(`?log_to_web_socket=1` in browser, `--log-to-web-socket` on command line, or
just hack it in by switching the default in `options.ts`).
The most well-tested way to do this is using the standalone web runner.
This requires serving the CTS locally. In the project root:
```
npm run standalone
npm start
```
Once this is started you can then direct a WebGPU enabled browser to the
specific CTS entry and run the tests, for example:
```
http://localhost:8080/standalone/?log_to_web_socket=1&q=webgpu:*
```
If the tests have a high variance in runtime, you can run them multiple times.
The longest recorded time will be used.
### Merging metadata
The final step is to merge the new data that has been captured into the JSON
file.
This can be done using the following command:
```
tools/merge_listing_times webgpu -- tools/websocket-logger/wslog-2023-09-12T18-57-34.txt
tools/merge_listing_times webgpu -- tools/websocket-logger/wslog-*.txt
```
Or, you can point it to one of the log files from a specific invocation of websocket-logger.
Now you just need to commit the pending diff in your repo.
|
