path: root/src/doc/rustc-dev-guide/src/tests/ui.md

# UI tests

<!-- toc -->

UI tests are a particular [test suite](compiletest.md#test-suites) of compiletest.

## Introduction

The tests in [`tests/ui`] are a collection of general-purpose tests which
primarily focus on validating the console output of the compiler, but can be
used for many other purposes.
For example, tests can also be configured to [run the resulting
program](#controlling-passfail-expectations) to verify its behavior.

[`tests/ui`]: https://github.com/rust-lang/rust/blob/master/tests/ui

## General structure of a test

A test consists of a Rust source file located anywhere in the `tests/ui` directory.
For example, [`tests/ui/hello.rs`] is a basic hello-world test.

Compiletest will use `rustc` to compile the test, and compare the output
against the expected output which is stored in a `.stdout` or `.stderr` file
located next to the test.
See [Output comparison](#output-comparison) for more.

Additionally, errors and warnings should be annotated with comments within
the source file.
See [Error annotations](#error-annotations) for more.

[Headers](headers.md) in the form of comments at the top of the file control
how the test is compiled and what the expected behavior is.

Tests are expected to fail to compile, since most tests are testing compiler
errors.
You can change that behavior with a header, see [Controlling pass/fail
expectations](#controlling-passfail-expectations).

By default, a test is built as an executable binary.
If you need a different crate type, you can use the `#![crate_type]` attribute
to set it as needed.

[`tests/ui/hello.rs`]: https://github.com/rust-lang/rust/blob/master/tests/ui/hello.rs

## Output comparison

UI tests store the expected output from the compiler in `.stderr` and
`.stdout` files next to the test.
You normally generate these files with the `--bless` CLI option, and then
inspect them manually to verify they contain what you expect.

The output is normalized to ignore unwanted differences, see the
[Normalization](#normalization) section.
If the file is missing, then compiletest expects the corresponding output to
be empty.

There can be multiple stdout/stderr files.
The general form is:

*test-name*`.`*revision*`.`*compare_mode*`.`*extension*

* *revision* is the [revision](#cfg-revisions) name.
  This is not included when not using revisions.
* *compare_mode* is the [compare mode](#compare-modes).
  This will only be checked when the given compare mode is active.
  If the file does not exist, then compiletest will check for a file without
  the compare mode.
* *extension* is the kind of output being checked:
  * `stderr` — compiler stderr
  * `stdout` — compiler stdout
  * `run.stderr` — stderr when running the test
  * `run.stdout` — stdout when running the test
  * `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 64-bit target
  * `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 32-bit target

A simple example would be `foo.stderr` next to a `foo.rs` test.
A more complex example would be `foo.my-revision.polonius.stderr`.

There are several [headers](headers.md) which will change how compiletest will
check for output files:

* `stderr-per-bitwidth` — checks separate output files based on the target
  pointer width. Consider using the `normalize-stderr` header instead (see
  [Normalization](#normalization)).
* `dont-check-compiler-stderr` — Ignores stderr from the compiler.
* `dont-check-compiler-stdout` — Ignores stdout from the compiler.
* `compare-output-lines-by-subset` — Checks that the output contains the
  contents of the stored output files by lines opposed to checking for strict
  equality.

UI tests run with with `-Zdeduplicate-diagnostics=no` flag which disables
rustc's built-in diagnostic deduplication mechanism.
This means you may see some duplicate messages in the output.
This helps illuminate situations where duplicate diagnostics are being
generated.

### Normalization

The compiler output is normalized to eliminate output difference between
platforms, mainly about filenames.

Compiletest makes the following replacements on the compiler output:

- The directory where the test is defined is replaced with `$DIR`.
  Example: `/path/to/rust/tests/ui/error-codes`
- The directory to the standard library source is replaced with `$SRC_DIR`.
  Example: `/path/to/rust/library`
- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:COL`.
  This helps ensure that changes to the layout of the standard library do not
  cause widespread changes to the `.stderr` files.
  Example: `$SRC_DIR/alloc/src/sync.rs:53:46`
- The base directory where the test's output goes is replaced with `$TEST_BUILD_DIR`.
  This only comes up in a few rare circumstances.
  Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
- Tabs are replaced with `\t`.
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using
  a heuristic). This helps normalize differences with Windows-style paths.
- CRLF newlines are converted to LF.
- Error line annotations like `//~ ERROR some message` are removed.
- Various v0 and legacy symbol hashes are replaced with placeholders like
  `[HASH]` or `<SYMBOL_HASH>`.

Additionally, the compiler is run with the `-Z ui-testing` flag which causes
the compiler itself to apply some changes to the diagnostic output to make it
more suitable for UI testing.
For example, it will anonymize line numbers in the output (line numbers
prefixing each source line are replaced with `LL`).
In extremely rare situations, this mode can be disabled with the header
command `// compile-flags: -Z ui-testing=no`.

Note: The line and column numbers for `-->` lines pointing to the test are
*not* normalized, and left as-is. This ensures that the compiler continues
to point to the correct location, and keeps the stderr files readable.
Ideally all line/column information would be retained, but small changes to
the source causes large diffs, and more frequent merge conflicts and test
errors.

Sometimes these built-in normalizations are not enough. In such cases, you
may provide custom normalization rules using the header commands, e.g.

```rust,ignore
// normalize-stdout-test: "foo" -> "bar"
// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)"
// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
```

This tells the test, on 32-bit platforms, whenever the compiler writes
`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
instead. Similar for 64-bit. The replacement is performed by regexes using
default regex flavor provided by `regex` crate.

The corresponding reference file will use the normalized output to test both
32-bit and 64-bit platforms:

```text
...
   |
   = note: source type: fn() ($PTR bits)
   = note: target type: u16 (16 bits)
...
```

Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a
concrete usage example.

[mrs]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.rs
[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.stderr

Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
information or stage supported by [`ignore-X`](headers.md#ignoring-tests)
here as well (e.g. `normalize-stderr-windows` or simply
`normalize-stderr-test` for unconditional replacement).


## Error annotations

Error annotations specify the errors that the compiler is expected to emit.
They are "attached" to the line in source where the error is located.

```rust,ignore
fn main() {
    boom  //~ ERROR cannot find value `boom` in this scope [E0425]
}
```

Although UI tests have a `.stderr` file which contains the entire compiler output,
UI tests require that errors are also annotated within the source.
This redundancy helps avoid mistakes since the `.stderr` files are usually
auto-generated.
It also helps to directly see where the error spans are expected to point to
by looking at one file instead of having to compare the `.stderr` file with
the source.
Finally, they ensure that no additional unexpected errors are generated.

They have several forms, but generally are a comment with the diagnostic
level (such as `ERROR`) and a substring of the expected error output.
You don't have to write out the entire message, just make sure to include the
important part of the message to make it self-documenting.

The error annotation needs to match with the line of the diagnostic.
There are several ways to match the message with the line (see the examples below):

* `~`: Associates the error level and message with the current line
* `~^`: Associates the error level and message with the previous error
  annotation line.
  Each caret (`^`) that you add adds a line to this, so `~^^^` is three lines
  above the error annotation line.
* `~|`: Associates the error level and message with the same line as the
  previous comment.
  This is more convenient than using multiple carets when there are multiple
  messages associated with the same line.

The space character between `//~` (or other variants) and the subsequent text
is negligible (i.e. there is no semantic difference between `//~ ERROR` and
`//~ERROR` although the former is more common in the codebase).

### Error annotation examples

Here are examples of error annotations on different lines of UI test
source.

#### Positioned on error line

Use the `//~ ERROR` idiom:

```rust,ignore
fn main() {
    let x = (1, 2, 3);
    match x {
        (_a, _x @ ..) => {} //~ ERROR `_x @` is not allowed in a tuple
        _ => {}
    }
}
```

#### Positioned below error line

Use the `//~^` idiom with number of carets in the string to indicate the
number of lines above.
In the example below, the error line is four lines above the error annotation
line so four carets are included in the annotation.

```rust,ignore
fn main() {
    let x = (1, 2, 3);
    match x {
        (_a, _x @ ..) => {}  // <- the error is on this line
        _ => {}
    }
}
//~^^^^ ERROR `_x @` is not allowed in a tuple
```

#### Use same error line as defined on error annotation line above

Use the `//~|` idiom to define the same error line as the error annotation
line above:

```rust,ignore
struct Binder(i32, i32, i32);

fn main() {
    let x = Binder(1, 2, 3);
    match x {
        Binder(_a, _x @ ..) => {}  // <- the error is on this line
        _ => {}
    }
}
//~^^^^ ERROR `_x @` is not allowed in a tuple struct
//~| ERROR this pattern has 1 field, but the corresponding tuple struct has 3 fields [E0023]
```

### `error-pattern`

The `error-pattern` [header](headers.md) can be used for
messages that don't have a specific span.

Let's think about this test:

```rust,ignore
fn main() {
    let a: *const [_] = &[1, 2, 3];
    unsafe {
        let _b = (*a)[3];
    }
}
```

We want to ensure this shows "index out of bounds" but we cannot use the
`ERROR` annotation since the error doesn't have any span.
Then it's time to use the `error-pattern` header:

```rust,ignore
// error-pattern: index out of bounds
fn main() {
    let a: *const [_] = &[1, 2, 3];
    unsafe {
        let _b = (*a)[3];
    }
}
```

But for strict testing, try to use the `ERROR` annotation as much as possible.

### Error levels

The error levels that you can have are:

1. `ERROR`
2. `WARN` or `WARNING`
3. `NOTE`
4. `HELP` and `SUGGESTION`

You are allowed to not include a level, but you should include it at least for
the primary message.

The `SUGGESTION` level is used for specifying what the expected replacement
text should be for a diagnostic suggestion.

UI tests use the `-A unused` flag by default to ignore all unused warnings, as
unused warnings are usually not the focus of a test.
However, simple code samples often have unused warnings.
If the test is specifically testing an unused warning, just add the
appropriate `#![warn(unused)]` attribute as needed.

### cfg revisions

When using [revisions](compiletest.md#revisions), different messages can be
conditionally checked based on the current revision.
This is done by placing the revision cfg name in brackets like this:

```rust,ignore
// edition:2018
// revisions: mir thir
// [thir]compile-flags: -Z thir-unsafeck

async unsafe fn f() {}

async fn g() {
    f(); //~ ERROR call to unsafe function is unsafe
}

fn main() {
    f(); //[mir]~ ERROR call to unsafe function is unsafe
}
```

In this example, the second error message is only emitted in the `mir` revision.
The `thir` revision only emits the first error.

If the cfg causes the compiler to emit different output, then a test can have
multiple `.stderr` files for the different outputs.
In the example above, there would be a `.mir.stderr` and `.thir.stderr` file
with the different outputs of the different revisions.


## Controlling pass/fail expectations

By default, a UI test is expected to **generate a compile error** because most
of the tests are checking for invalid input and error diagnostics.
However, you can also make UI tests where compilation is expected to succeed,
and you can even run the resulting program.
Just add one of the following [header commands](headers.md):

* Pass headers:
  * `// check-pass` — compilation should succeed but skip codegen
    (which is expensive and isn't supposed to fail in most cases).
  * `// build-pass` — compilation and linking should succeed but do
    not run the resulting binary.
  * `// run-pass` — compilation should succeed and running the resulting
    binary should also succeed.
* Fail headers:
  * `// check-fail` — compilation should fail (the codegen phase is skipped).
    This is the default for UI tests.
  * `// build-fail` — compilation should fail during the codegen phase.
    This will run `rustc` twice, once to verify that it compiles successfully
    without the codegen phase, then a second time the full compile should
    fail.
  * `// run-fail` — compilation should succeed, but running the resulting
    binary should fail.

For `run-pass` and `run-fail` tests, by default the output of the program
itself is not checked.
If you want to check the output of running the program, include the
`check-run-results` header.
This will check for a `.run.stderr` and `.run.stdout` files to compare
against the actual output of the program.

Tests with the `*-pass` headers can be overridden with the `--pass`
command-line option:

```sh
./x.py test tests/ui --pass check
```

The `--pass` option only affects UI tests.
Using `--pass check` can run the UI test suite much faster (roughly twice as
fast on my system), though obviously not exercising as much.

The `ignore-pass` header can be used to ignore the `--pass` CLI flag if the
test won't work properly with that override.


## Known bugs

The `known-bug` header may be used for tests that demonstrate a known bug that
has not yet been fixed.
Adding tests for known bugs is helpful for several reasons, including:

1. Maintaining a functional test that can be conveniently reused when the bug is fixed.
2. Providing a sentinel that will fail if the bug is incidentally fixed.
   This can alert the developer so they know that the associated issue has
   been fixed and can possibly be closed.

Do not include [error annotations](#error-annotations) in a test with `known-bug`.
The test should still include other normal headers and stdout/stderr files.


## Test organization

When deciding where to place a test file, please try to find a subdirectory
that best matches what you are trying to exercise.
Do your best to keep things organized.
Admittedly it can be difficult as some tests can overlap different categories,
and the existing layout may not fit well.

For regression tests – basically, some random snippet of code that came in
from the internet – we often name the test after the issue plus a short
description.
Ideally, the test should be added to a directory that helps identify what
piece of code is being tested here (e.g.,
`tests/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`)

When writing a new feature, **create a subdirectory to store your tests**.
For example, if you are implementing RFC 1234 ("Widgets"), then it might make
sense to put the tests in a directory like `tests/ui/rfc1234-widgets/`.

In other cases, there may already be a suitable directory. (The proper
directory structure to use is actually an area of active debate.)

Over time, the [`tests/ui`] directory has grown very fast.
There is a check in [tidy](intro.md#tidy) that will ensure none of the
subdirectories has more than 1000 entries.
Having too many files causes problems because it isn't editor/IDE friendly and
the GitHub UI won't show more than 1000 entries.
However, since `tests/ui` (UI test root directory) and `tests/ui/issues`
directories have more than 1000 entries, we set a different limit for those
directories.
So, please avoid putting a new test there and try to find a more relevant
place.

For example, if your test is related to closures, you should put it in
`tests/ui/closures`.
If you're not sure where is the best place, it's still okay to add to
`tests/ui/issues/`.
When you reach the limit, you could increase it by tweaking [here][ui test
tidy].

[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs


## Rustfix tests

UI tests can validate that diagnostic suggestions apply correctly
and that the resulting changes compile correctly.
This can be done with the `run-rustfix` header:

```rust,ignore
// run-rustfix
// check-pass
#![crate_type = "lib"]

pub struct not_camel_case {}
//~^ WARN `not_camel_case` should have an upper camel case name
//~| HELP convert the identifier to upper camel case
//~| SUGGESTION NotCamelCase
```

Rustfix tests should have a file with the `.fixed` extension which contains
the source file after the suggestion has been applied.

When the test is run, compiletest first checks that the correct
lint/warning is generated.
Then, it applies the suggestion and compares against `.fixed` (they must match).
Finally, the fixed source is compiled, and this compilation is required to succeed.

Usually when creating a rustfix test you will generate the `.fixed` file
automatically with the `x.py test --bless` option.

The `run-rustfix` header will cause *all* suggestions to be applied, even
if they are not [`MachineApplicable`](../diagnostics.md#suggestions).
If this is a problem, then you can instead use the `rustfix-only-machine-applicable`
header.
This should be used if there is a mixture of different suggestion levels, and
some of the non-machine-applicable ones do not apply cleanly.


## Compare modes

[Compare modes](compiletest.md#compare-modes) can be used to run all tests
with different flags from what they are normally compiled with.
In some cases, this might result in different output from the compiler.
To support this, different output files can be saved which contain the
output based on the compare mode.

For example, when using the Polonius mode, a test `foo.rs` will
first look for expected output in `foo.polonius.stderr`, falling back to the usual
`foo.stderr` if not found.
This is useful as different modes can sometimes result in different
diagnostics and behavior.
This can help track which tests have differences between the modes, and to
visually inspect those diagnostic differences.

If in the rare case you encounter a test that has different behavior, you can
run something like the following to generate the alternate stderr file:

```sh
./x.py test tests/ui --compare-mode=polonius --bless
```

Currently none of the compare modes are checked in CI for UI tests.