diff options
Diffstat (limited to 'Documentation/dev-tools/kunit')
| -rw-r--r-- | Documentation/dev-tools/kunit/api/index.rst | 20 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/api/resource.rst | 13 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/api/test.rst | 10 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/architecture.rst | 195 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/faq.rst | 104 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/index.rst | 113 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg | 81 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/run_manual.rst | 57 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/run_wrapper.rst | 323 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/running_tips.rst | 264 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/start.rst | 306 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/style.rst | 202 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/tips.rst | 190 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/usage.rst | 705 |
14 files changed, 2583 insertions, 0 deletions
diff --git a/Documentation/dev-tools/kunit/api/index.rst b/Documentation/dev-tools/kunit/api/index.rst new file mode 100644 index 000000000..45ce04823 --- /dev/null +++ b/Documentation/dev-tools/kunit/api/index.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============= +API Reference +============= +.. toctree:: + + test + resource + +This section documents the KUnit kernel testing API. It is divided into the +following sections: + +Documentation/dev-tools/kunit/api/test.rst + + - documents all of the standard testing API + +Documentation/dev-tools/kunit/api/resource.rst + + - documents the KUnit resource API diff --git a/Documentation/dev-tools/kunit/api/resource.rst b/Documentation/dev-tools/kunit/api/resource.rst new file mode 100644 index 000000000..0a94f8312 --- /dev/null +++ b/Documentation/dev-tools/kunit/api/resource.rst @@ -0,0 +1,13 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +Resource API +============ + +This file documents the KUnit resource API. + +Most users won't need to use this API directly, power users can use it to store +state on a per-test basis, register custom cleanup actions, and more. + +.. kernel-doc:: include/kunit/resource.h + :internal: diff --git a/Documentation/dev-tools/kunit/api/test.rst b/Documentation/dev-tools/kunit/api/test.rst new file mode 100644 index 000000000..c5eca423e --- /dev/null +++ b/Documentation/dev-tools/kunit/api/test.rst @@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======== +Test API +======== + +This file documents all of the standard testing API. + +.. kernel-doc:: include/kunit/test.h + :internal: diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst new file mode 100644 index 000000000..8efe792bd --- /dev/null +++ b/Documentation/dev-tools/kunit/architecture.rst @@ -0,0 +1,195 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================== +KUnit Architecture +================== + +The KUnit architecture can be divided into two parts: + +- `In-Kernel Testing Framework`_ +- `kunit_tool (Command Line Test Harness)`_ + +In-Kernel Testing Framework +=========================== + +The kernel testing library supports KUnit tests written in C using +KUnit. KUnit tests are kernel code. KUnit does several things: + +- Organizes tests +- Reports test results +- Provides test utilities + +Test Cases +---------- + +The fundamental unit in KUnit is the test case. The KUnit test cases are +grouped into KUnit suites. A KUnit test case is a function with type +signature ``void (*)(struct kunit *test)``. +These test case functions are wrapped in a struct called +struct kunit_case. + +.. note: + ``generate_params`` is optional for non-parameterized tests. + +Each KUnit test case gets a ``struct kunit`` context +object passed to it that tracks a running test. The KUnit assertion +macros and other KUnit utilities use the ``struct kunit`` context +object. As an exception, there are two fields: + +- ``->priv``: The setup functions can use it to store arbitrary test + user data. + +- ``->param_value``: It contains the parameter value which can be + retrieved in the parameterized tests. + +Test Suites +----------- + +A KUnit suite includes a collection of test cases. The KUnit suites +are represented by the ``struct kunit_suite``. For example: + +.. code-block:: c + + static struct kunit_case example_test_cases[] = { + KUNIT_CASE(example_test_foo), + KUNIT_CASE(example_test_bar), + KUNIT_CASE(example_test_baz), + {} + }; + + static struct kunit_suite example_test_suite = { + .name = "example", + .init = example_test_init, + .exit = example_test_exit, + .test_cases = example_test_cases, + }; + kunit_test_suite(example_test_suite); + +In the above example, the test suite ``example_test_suite``, runs the +test cases ``example_test_foo``, ``example_test_bar``, and +``example_test_baz``. Before running the test, the ``example_test_init`` +is called and after running the test, ``example_test_exit`` is called. +The ``kunit_test_suite(example_test_suite)`` registers the test suite +with the KUnit test framework. + +Executor +-------- + +The KUnit executor can list and run built-in KUnit tests on boot. +The Test suites are stored in a linker section +called ``.kunit_test_suites``. For code, see: +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/asm-generic/vmlinux.lds.h?h=v5.15#n945. +The linker section consists of an array of pointers to +``struct kunit_suite``, and is populated by the ``kunit_test_suites()`` +macro. To run all tests compiled into the kernel, the KUnit executor +iterates over the linker section array. + +.. kernel-figure:: kunit_suitememorydiagram.svg + :alt: KUnit Suite Memory + + KUnit Suite Memory Diagram + +On the kernel boot, the KUnit executor uses the start and end addresses +of this section to iterate over and run all tests. For code, see: +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/executor.c + +When built as a module, the ``kunit_test_suites()`` macro defines a +``module_init()`` function, which runs all the tests in the compilation +unit instead of utilizing the executor. + +In KUnit tests, some error classes do not affect other tests +or parts of the kernel, each KUnit case executes in a separate thread +context. For code, see: +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/try-catch.c?h=v5.15#n58 + +Assertion Macros +---------------- + +KUnit tests verify state using expectations/assertions. +All expectations/assertions are formatted as: +``KUNIT_{EXPECT|ASSERT}_<op>[_MSG](kunit, property[, message])`` + +- ``{EXPECT|ASSERT}`` determines whether the check is an assertion or an + expectation. + + - For an expectation, if the check fails, marks the test as failed + and logs the failure. + + - An assertion, on failure, causes the test case to terminate + immediately. + + - Assertions call function: + ``void __noreturn kunit_abort(struct kunit *)``. + + - ``kunit_abort`` calls function: + ``void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch)``. + + - ``kunit_try_catch_throw`` calls function: + ``void kthread_complete_and_exit(struct completion *, long) __noreturn;`` + and terminates the special thread context. + +- ``<op>`` denotes a check with options: ``TRUE`` (supplied property + has the boolean value “true”), ``EQ`` (two supplied properties are + equal), ``NOT_ERR_OR_NULL`` (supplied pointer is not null and does not + contain an “err” value). + +- ``[_MSG]`` prints a custom message on failure. + +Test Result Reporting +--------------------- +KUnit prints test results in KTAP format. KTAP is based on TAP14, see: +https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md. +KTAP (yet to be standardized format) works with KUnit and Kselftest. +The KUnit executor prints KTAP results to dmesg, and debugfs +(if configured). + +Parameterized Tests +------------------- + +Each KUnit parameterized test is associated with a collection of +parameters. The test is invoked multiple times, once for each parameter +value and the parameter is stored in the ``param_value`` field. +The test case includes a KUNIT_CASE_PARAM() macro that accepts a +generator function. +The generator function is passed the previous parameter and returns the next +parameter. It also provides a macro to generate common-case generators based on +arrays. + +kunit_tool (Command Line Test Harness) +====================================== + +kunit_tool is a Python script ``(tools/testing/kunit/kunit.py)`` +that can be used to configure, build, exec, parse and run (runs other +commands in order) test results. You can either run KUnit tests using +kunit_tool or can include KUnit in kernel and parse manually. + +- ``configure`` command generates the kernel ``.config`` from a + ``.kunitconfig`` file (and any architecture-specific options). + For some architectures, additional config options are specified in the + ``qemu_config`` Python script + (For example: ``tools/testing/kunit/qemu_configs/powerpc.py``). + It parses both the existing ``.config`` and the ``.kunitconfig`` files + and ensures that ``.config`` is a superset of ``.kunitconfig``. + If this is not the case, it will combine the two and run + ``make olddefconfig`` to regenerate the ``.config`` file. It then + verifies that ``.config`` is now a superset. This checks if all + Kconfig dependencies are correctly specified in ``.kunitconfig``. + ``kunit_config.py`` includes the parsing Kconfigs code. The code which + runs ``make olddefconfig`` is a part of ``kunit_kernel.py``. You can + invoke this command via: ``./tools/testing/kunit/kunit.py config`` and + generate a ``.config`` file. +- ``build`` runs ``make`` on the kernel tree with required options + (depends on the architecture and some options, for example: build_dir) + and reports any errors. + To build a KUnit kernel from the current ``.config``, you can use the + ``build`` argument: ``./tools/testing/kunit/kunit.py build``. +- ``exec`` command executes kernel results either directly (using + User-mode Linux configuration), or via an emulator such + as QEMU. It reads results from the log via standard + output (stdout), and passes them to ``parse`` to be parsed. + If you already have built a kernel with built-in KUnit tests, + you can run the kernel and display the test results with the ``exec`` + argument: ``./tools/testing/kunit/kunit.py exec``. +- ``parse`` extracts the KTAP output from a kernel log, parses + the test results, and prints a summary. For failed tests, any + diagnostic output will be included. diff --git a/Documentation/dev-tools/kunit/faq.rst b/Documentation/dev-tools/kunit/faq.rst new file mode 100644 index 000000000..fae426f26 --- /dev/null +++ b/Documentation/dev-tools/kunit/faq.rst @@ -0,0 +1,104 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================== +Frequently Asked Questions +========================== + +How is this different from Autotest, kselftest, and so on? +========================================================== +KUnit is a unit testing framework. Autotest, kselftest (and some others) are +not. + +A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is supposed to +test a single unit of code in isolation and hence the name *unit test*. A unit +test should be the finest granularity of testing and should allow all possible +code paths to be tested in the code under test. This is only possible if the +code under test is small and does not have any external dependencies outside of +the test's control like hardware. + +There are no testing frameworks currently available for the kernel that do not +require installing the kernel on a test machine or in a virtual machine. All +testing frameworks require tests to be written in userspace and run on the +kernel under test. This is true for Autotest, kselftest, and some others, +disqualifying any of them from being considered unit testing frameworks. + +Does KUnit support running on architectures other than UML? +=========================================================== + +Yes, mostly. + +For the most part, the KUnit core framework (what we use to write the tests) +can compile to any architecture. It compiles like just another part of the +kernel and runs when the kernel boots, or when built as a module, when the +module is loaded. However, there is infrastructure, like the KUnit Wrapper +(``tools/testing/kunit/kunit.py``) that might not support some architectures +(see :ref:`kunit-on-qemu`). + +In short, yes, you can run KUnit on other architectures, but it might require +more work than using KUnit on UML. + +For more information, see :ref:`kunit-on-non-uml`. + +.. _kinds-of-tests: + +What is the difference between a unit test and other kinds of tests? +==================================================================== +Most existing tests for the Linux kernel would be categorized as an integration +test, or an end-to-end test. + +- A unit test is supposed to test a single unit of code in isolation. A unit + test should be the finest granularity of testing and, as such, allows all + possible code paths to be tested in the code under test. This is only possible + if the code under test is small and does not have any external dependencies + outside of the test's control like hardware. +- An integration test tests the interaction between a minimal set of components, + usually just two or three. For example, someone might write an integration + test to test the interaction between a driver and a piece of hardware, or to + test the interaction between the userspace libraries the kernel provides and + the kernel itself. However, one of these tests would probably not test the + entire kernel along with hardware interactions and interactions with the + userspace. +- An end-to-end test usually tests the entire system from the perspective of the + code under test. For example, someone might write an end-to-end test for the + kernel by installing a production configuration of the kernel on production + hardware with a production userspace and then trying to exercise some behavior + that depends on interactions between the hardware, the kernel, and userspace. + +KUnit is not working, what should I do? +======================================= + +Unfortunately, there are a number of things which can break, but here are some +things to try. + +1. Run ``./tools/testing/kunit/kunit.py run`` with the ``--raw_output`` + parameter. This might show details or error messages hidden by the kunit_tool + parser. +2. Instead of running ``kunit.py run``, try running ``kunit.py config``, + ``kunit.py build``, and ``kunit.py exec`` independently. This can help track + down where an issue is occurring. (If you think the parser is at fault, you + can run it manually against ``stdin`` or a file with ``kunit.py parse``.) +3. Running the UML kernel directly can often reveal issues or error messages, + ``kunit_tool`` ignores. This should be as simple as running ``./vmlinux`` + after building the UML kernel (for example, by using ``kunit.py build``). + Note that UML has some unusual requirements (such as the host having a tmpfs + filesystem mounted), and has had issues in the past when built statically and + the host has KASLR enabled. (On older host kernels, you may need to run + ``setarch `uname -m` -R ./vmlinux`` to disable KASLR.) +4. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test + (e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config + around, so you can see what config was used after running ``kunit.py run``. + It also preserves any config changes you might make, so you can + enable/disable things with ``make ARCH=um menuconfig`` or similar, and then + re-run kunit_tool. +5. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This + may help clean up any residual config items which could be causing problems. +6. Finally, try running KUnit outside UML. KUnit and KUnit tests can be + built into any kernel, or can be built as a module and loaded at runtime. + Doing so should allow you to determine if UML is causing the issue you're + seeing. When tests are built-in, they will execute when the kernel boots, and + modules will automatically execute associated tests when loaded. Test results + can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and + can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`. + +If none of the above tricks help, you are always welcome to email any issues to +kunit-dev@googlegroups.com. diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst new file mode 100644 index 000000000..f5d13f1d3 --- /dev/null +++ b/Documentation/dev-tools/kunit/index.rst @@ -0,0 +1,113 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================= +KUnit - Linux Kernel Unit Testing +================================= + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + start + architecture + run_wrapper + run_manual + usage + api/index + style + faq + tips + running_tips + +This section details the kernel unit testing framework. + +Introduction +============ + +KUnit (Kernel unit testing framework) provides a common framework for +unit tests within the Linux kernel. Using KUnit, you can define groups +of test cases called test suites. The tests either run on kernel boot +if built-in, or load as a module. KUnit automatically flags and reports +failed test cases in the kernel log. The test results appear in +:doc:`KTAP (Kernel - Test Anything Protocol) format</dev-tools/ktap>`. +It is inspired by JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock +(C++ unit testing framework). + +KUnit tests are part of the kernel, written in the C (programming) +language, and test parts of the Kernel implementation (example: a C +language function). Excluding build time, from invocation to +completion, KUnit can run around 100 tests in less than 10 seconds. +KUnit can test any kernel component, for example: file system, system +calls, memory management, device drivers and so on. + +KUnit follows the white-box testing approach. The test has access to +internal system functionality. KUnit runs in kernel space and is not +restricted to things exposed to user-space. + +In addition, KUnit has kunit_tool, a script (``tools/testing/kunit/kunit.py``) +that configures the Linux kernel, runs KUnit tests under QEMU or UML +(:doc:`User Mode Linux </virt/uml/user_mode_linux_howto_v2>`), +parses the test results and +displays them in a user friendly manner. + +Features +-------- + +- Provides a framework for writing unit tests. +- Runs tests on any kernel architecture. +- Runs a test in milliseconds. + +Prerequisites +------------- + +- Any Linux kernel compatible hardware. +- For Kernel under test, Linux kernel version 5.5 or greater. + +Unit Testing +============ + +A unit test tests a single unit of code in isolation. A unit test is the finest +granularity of testing and allows all possible code paths to be tested in the +code under test. This is possible if the code under test is small and does not +have any external dependencies outside of the test's control like hardware. + + +Write Unit Tests +---------------- + +To write good unit tests, there is a simple but powerful pattern: +Arrange-Act-Assert. This is a great way to structure test cases and +defines an order of operations. + +- Arrange inputs and targets: At the start of the test, arrange the data + that allows a function to work. Example: initialize a statement or + object. +- Act on the target behavior: Call your function/code under test. +- Assert expected outcome: Verify that the result (or resulting state) is as + expected. + +Unit Testing Advantages +----------------------- + +- Increases testing speed and development in the long run. +- Detects bugs at initial stage and therefore decreases bug fix cost + compared to acceptance testing. +- Improves code quality. +- Encourages writing testable code. + +Read also :ref:`kinds-of-tests`. + +How do I use it? +================ + +* Documentation/dev-tools/kunit/start.rst - for KUnit new users. +* Documentation/dev-tools/kunit/architecture.rst - KUnit architecture. +* Documentation/dev-tools/kunit/run_wrapper.rst - run kunit_tool. +* Documentation/dev-tools/kunit/run_manual.rst - run tests without kunit_tool. +* Documentation/dev-tools/kunit/usage.rst - write tests. +* Documentation/dev-tools/kunit/tips.rst - best practices with + examples. +* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs + used for testing. +* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and + answers. diff --git a/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg b/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg new file mode 100644 index 000000000..cf8fddc27 --- /dev/null +++ b/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg @@ -0,0 +1,81 @@ +<?xml version="1.0" encoding="UTF-8"?> +<svg width="796.93" height="555.73" version="1.1" viewBox="0 0 796.93 555.73" xmlns="http://www.w3.org/2000/svg"> + <g transform="translate(-13.724 -17.943)"> + <g fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"> + <rect x="323.56" y="18.443" width="115.75" height="41.331"/> + <rect x="323.56" y="463.09" width="115.75" height="41.331"/> + <rect x="323.56" y="531.84" width="115.75" height="41.331"/> + <rect x="323.56" y="88.931" width="115.75" height="74.231"/> + </g> + <g> + <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/> + <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text> + </g> + <g transform="translate(0 -258.6)"> + <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/> + <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text> + </g> + <g transform="translate(0 -217.27)"> + <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/> + <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text> + </g> + <g transform="translate(0 -175.94)"> + <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/> + <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text> + </g> + <g transform="translate(0 -134.61)"> + <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/> + <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text> + </g> + <g transform="translate(0 -41.331)"> + <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/> + <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text> + </g> + <g transform="translate(3.4459e-5 -.71088)"> + <rect x="502.19" y="143.16" width="201.13" height="41.331" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/> + <text x="512.02319" y="168.02026" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="512.02319" y="168.02026" font-family="monospace">_kunit_suites_start</tspan></text> + </g> + <g transform="translate(3.0518e-5 -3.1753)"> + <rect x="502.19" y="445.69" width="201.13" height="41.331" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/> + <text x="521.61694" y="470.54846" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="521.61694" y="470.54846" font-family="monospace">_kunit_suites_end</tspan></text> + </g> + <rect x="14.224" y="277.78" width="134.47" height="41.331" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/> + <text x="32.062176" y="304.41287" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="32.062176" y="304.41287" font-family="monospace">.init.data</tspan></text> + <g transform="translate(217.98 145.12)" stroke="#1a1a1a"> + <circle cx="149.97" cy="373.01" r="3.4012"/> + <circle cx="163.46" cy="373.01" r="3.4012"/> + <circle cx="176.95" cy="373.01" r="3.4012"/> + </g> + <g transform="translate(217.98 -298.66)" stroke="#1a1a1a"> + <circle cx="149.97" cy="373.01" r="3.4012"/> + <circle cx="163.46" cy="373.01" r="3.4012"/> + <circle cx="176.95" cy="373.01" r="3.4012"/> + </g> + <g stroke="#1a1a1a"> + <rect x="323.56" y="328.49" width="115.75" height="51.549" fill="#b9dbc6"/> + <g transform="translate(217.98 -18.75)"> + <circle cx="149.97" cy="373.01" r="3.4012"/> + <circle cx="163.46" cy="373.01" r="3.4012"/> + <circle cx="176.95" cy="373.01" r="3.4012"/> + </g> + </g> + <g transform="scale(1.0933 .9147)" stroke-width="32.937" aria-label="{"> + <path d="m275.49 545.57c-35.836-8.432-47.43-24.769-47.957-64.821v-88.536c-0.527-44.795-10.54-57.97-49.538-67.456 38.998-10.013 49.011-23.715 49.538-67.983v-88.536c0.527-40.052 12.121-56.389 47.957-64.821v-5.797c-65.348 0-85.901 17.391-86.955 73.253v93.806c-0.527 36.89-10.013 50.065-44.795 59.551 34.782 10.013 44.268 23.188 44.795 60.078v93.279c1.581 56.389 21.607 73.78 86.955 73.78z"/> + </g> + <g transform="scale(1.1071 .90325)" stroke-width="14.44" aria-label="{"> + <path d="m461.46 443.55c-15.711-3.6967-20.794-10.859-21.025-28.418v-38.815c-0.23104-19.639-4.6209-25.415-21.718-29.574 17.097-4.3898 21.487-10.397 21.718-29.805v-38.815c0.23105-17.559 5.314-24.722 21.025-28.418v-2.5415c-28.649 0-37.66 7.6244-38.122 32.115v41.126c-0.23105 16.173-4.3898 21.949-19.639 26.108 15.249 4.3898 19.408 10.166 19.639 26.339v40.895c0.69313 24.722 9.4728 32.346 38.122 32.346z"/> + </g> + <path d="m449.55 161.84v2.5h49.504v-2.5z" color="#000000" style="-inkscape-stroke:none"/> + <g fill-rule="evenodd"> + <path d="m443.78 163.09 8.65-5v10z" color="#000000" stroke-width="1pt" style="-inkscape-stroke:none"/> + <path d="m453.1 156.94-10.648 6.1543 0.99804 0.57812 9.6504 5.5781zm-1.334 2.3125v7.6856l-6.6504-3.8438z" color="#000000" style="-inkscape-stroke:none"/> + </g> + <path d="m449.55 461.91v2.5h49.504v-2.5z" color="#000000" style="-inkscape-stroke:none"/> + <g fill-rule="evenodd"> + <path d="m443.78 463.16 8.65-5v10z" color="#000000" stroke-width="1pt" style="-inkscape-stroke:none"/> + <path d="m453.1 457-10.648 6.1562 0.99804 0.57617 9.6504 5.5781zm-1.334 2.3125v7.6856l-6.6504-3.8438z" color="#000000" style="-inkscape-stroke:none"/> + </g> + <rect x="515.64" y="223.9" width="294.52" height="178.49" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/> + <text x="523.33319" y="262.52542" font-family="monospace" font-size="14.667px" style="line-height:1.25" xml:space="preserve"><tspan x="523.33319" y="262.52542"><tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit_suite {</tspan><tspan x="523.33319" y="280.8588"><tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold"> const char</tspan> name[<tspan fill="#ff00ff" font-size="14.667px">256</tspan>];</tspan><tspan x="523.33319" y="299.19217"> <tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">int</tspan> (*init)(<tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit *);</tspan><tspan x="523.33319" y="317.52554"> <tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">void</tspan> (*exit)(<tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit *);</tspan><tspan x="523.33319" y="335.85892"> <tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit_case *test_cases;</tspan><tspan x="523.33319" y="354.19229"> ...</tspan><tspan x="523.33319" y="372.52567">};</tspan></text> + </g> +</svg> diff --git a/Documentation/dev-tools/kunit/run_manual.rst b/Documentation/dev-tools/kunit/run_manual.rst new file mode 100644 index 000000000..e7b46421f --- /dev/null +++ b/Documentation/dev-tools/kunit/run_manual.rst @@ -0,0 +1,57 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================ +Run Tests without kunit_tool +============================ + +If we do not want to use kunit_tool (For example: we want to integrate +with other systems, or run tests on real hardware), we can +include KUnit in any kernel, read out results, and parse manually. + +.. note:: KUnit is not designed for use in a production system. It is + possible that tests may reduce the stability or security of + the system. + +Configure the Kernel +==================== + +KUnit tests can run without kunit_tool. This can be useful, if: + +- We have an existing kernel configuration to test. +- Need to run on real hardware (or using an emulator/VM kunit_tool + does not support). +- Wish to integrate with some existing testing systems. + +KUnit is configured with the ``CONFIG_KUNIT`` option, and individual +tests can also be built by enabling their config options in our +``.config``. KUnit tests usually (but don't always) have config options +ending in ``_KUNIT_TEST``. Most tests can either be built as a module, +or be built into the kernel. + +.. note :: + + We can enable the ``KUNIT_ALL_TESTS`` config option to + automatically enable all tests with satisfied dependencies. This is + a good way of quickly testing everything applicable to the current + config. + +Once we have built our kernel (and/or modules), it is simple to run +the tests. If the tests are built-in, they will run automatically on the +kernel boot. The results will be written to the kernel log (``dmesg``) +in TAP format. + +If the tests are built as modules, they will run when the module is +loaded. + +.. code-block :: bash + + # modprobe example-test + +The results will appear in TAP format in ``dmesg``. + +.. note :: + + If ``CONFIG_KUNIT_DEBUGFS`` is enabled, KUnit test results will + be accessible from the ``debugfs`` filesystem (if mounted). + They will be in ``/sys/kernel/debug/kunit/<test_suite>/results``, in + TAP format. diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst new file mode 100644 index 000000000..dafe8eb28 --- /dev/null +++ b/Documentation/dev-tools/kunit/run_wrapper.rst @@ -0,0 +1,323 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================= +Running tests with kunit_tool +============================= + +We can either run KUnit tests using kunit_tool or can run tests +manually, and then use kunit_tool to parse the results. To run tests +manually, see: Documentation/dev-tools/kunit/run_manual.rst. +As long as we can build the kernel, we can run KUnit. + +kunit_tool is a Python script which configures and builds a kernel, runs +tests, and formats the test results. + +Run command: + +.. code-block:: + + ./tools/testing/kunit/kunit.py run + +We should see the following: + +.. code-block:: + + Configuring KUnit Kernel ... + Building KUnit kernel... + Starting KUnit kernel... + +We may want to use the following options: + +.. code-block:: + + ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all` + +- ``--timeout`` sets a maximum amount of time for tests to run. +- ``--jobs`` sets the number of threads to build the kernel. + +kunit_tool will generate a ``.kunitconfig`` with a default +configuration, if no other ``.kunitconfig`` file exists +(in the build directory). In addition, it verifies that the +generated ``.config`` file contains the ``CONFIG`` options in the +``.kunitconfig``. +It is also possible to pass a separate ``.kunitconfig`` fragment to +kunit_tool. This is useful if we have several different groups of +tests we want to run independently, or if we want to use pre-defined +test configs for certain subsystems. + +To use a different ``.kunitconfig`` file (such as one +provided to test a particular subsystem), pass it as an option: + +.. code-block:: + + ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig + +To view kunit_tool flags (optional command-line arguments), run: + +.. code-block:: + + ./tools/testing/kunit/kunit.py run --help + +Creating a ``.kunitconfig`` file +================================ + +If we want to run a specific set of tests (rather than those listed +in the KUnit ``defconfig``), we can provide Kconfig options in the +``.kunitconfig`` file. For default .kunitconfig, see: +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/kunit/configs/default.config. +A ``.kunitconfig`` is a ``minconfig`` (a .config +generated by running ``make savedefconfig``), used for running a +specific set of tests. This file contains the regular Kernel configs +with specific test targets. The ``.kunitconfig`` also +contains any other config options required by the tests (For example: +dependencies for features under tests, configs that enable/disable +certain code blocks, arch configs and so on). + +To create a ``.kunitconfig``, using the KUnit ``defconfig``: + +.. code-block:: + + cd $PATH_TO_LINUX_REPO + cp tools/testing/kunit/configs/default.config .kunit/.kunitconfig + +We can then add any other Kconfig options. For example: + +.. code-block:: + + CONFIG_LIST_KUNIT_TEST=y + +kunit_tool ensures that all config options in ``.kunitconfig`` are +set in the kernel ``.config`` before running the tests. It warns if we +have not included the options dependencies. + +.. note:: Removing something from the ``.kunitconfig`` will + not rebuild the ``.config file``. The configuration is only + updated if the ``.kunitconfig`` is not a subset of ``.config``. + This means that we can use other tools + (For example: ``make menuconfig``) to adjust other config options. + The build dir needs to be set for ``make menuconfig`` to + work, therefore by default use ``make O=.kunit menuconfig``. + +Configuring, building, and running tests +======================================== + +If we want to make manual changes to the KUnit build process, we +can run part of the KUnit build process independently. +When running kunit_tool, from a ``.kunitconfig``, we can generate a +``.config`` by using the ``config`` argument: + +.. code-block:: + + ./tools/testing/kunit/kunit.py config + +To build a KUnit kernel from the current ``.config``, we can use the +``build`` argument: + +.. code-block:: + + ./tools/testing/kunit/kunit.py build + +If we already have built UML kernel with built-in KUnit tests, we +can run the kernel, and display the test results with the ``exec`` +argument: + +.. code-block:: + + ./tools/testing/kunit/kunit.py exec + +The ``run`` command discussed in section: **Running tests with kunit_tool**, +is equivalent to running the above three commands in sequence. + +Parsing test results +==================== + +KUnit tests output displays results in TAP (Test Anything Protocol) +format. When running tests, kunit_tool parses this output and prints +a summary. To see the raw test results in TAP format, we can pass the +``--raw_output`` argument: + +.. code-block:: + + ./tools/testing/kunit/kunit.py run --raw_output + +If we have KUnit results in the raw TAP format, we can parse them and +print the human-readable summary with the ``parse`` command for +kunit_tool. This accepts a filename for an argument, or will read from +standard input. + +.. code-block:: bash + + # Reading from a file + ./tools/testing/kunit/kunit.py parse /var/log/dmesg + # Reading from stdin + dmesg | ./tools/testing/kunit/kunit.py parse + +Filtering tests +=============== + +By passing a bash style glob filter to the ``exec`` or ``run`` +commands, we can run a subset of the tests built into a kernel . For +example: if we only want to run KUnit resource tests, use: + +.. code-block:: + + ./tools/testing/kunit/kunit.py run 'kunit-resource*' + +This uses the standard glob format with wildcard characters. + +.. _kunit-on-qemu: + +Running tests on QEMU +===================== + +kunit_tool supports running tests on qemu as well as +via UML. To run tests on qemu, by default it requires two flags: + +- ``--arch``: Selects a configs collection (Kconfig, qemu config options + and so on), that allow KUnit tests to be run on the specified + architecture in a minimal way. The architecture argument is same as + the option name passed to the ``ARCH`` variable used by Kbuild. + Not all architectures currently support this flag, but we can use + ``--qemu_config`` to handle it. If ``um`` is passed (or this flag + is ignored), the tests will run via UML. Non-UML architectures, + for example: i386, x86_64, arm and so on; run on qemu. + +- ``--cross_compile``: Specifies the Kbuild toolchain. It passes the + same argument as passed to the ``CROSS_COMPILE`` variable used by + Kbuild. As a reminder, this will be the prefix for the toolchain + binaries such as GCC. For example: + + - ``sparc64-linux-gnu`` if we have the sparc toolchain installed on + our system. + + - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux`` + if we have downloaded the microblaze toolchain from the 0-day + website to a directory in our home directory called toolchains. + +This means that for most architectures, running under qemu is as simple as: + +.. code-block:: bash + + ./tools/testing/kunit/kunit.py run --arch=x86_64 + +When cross-compiling, we'll likely need to specify a different toolchain, for +example: + +.. code-block:: bash + + ./tools/testing/kunit/kunit.py run \ + --arch=s390 \ + --cross_compile=s390x-linux-gnu- + +If we want to run KUnit tests on an architecture not supported by +the ``--arch`` flag, or want to run KUnit tests on qemu using a +non-default configuration; then we can write our own``QemuConfig``. +These ``QemuConfigs`` are written in Python. They have an import line +``from..qemu_config import QemuArchParams`` at the top of the file. +The file must contain a variable called ``QEMU_ARCH`` that has an +instance of ``QemuArchParams`` assigned to it. See example in: +``tools/testing/kunit/qemu_configs/x86_64.py``. + +Once we have a ``QemuConfig``, we can pass it into kunit_tool, +using the ``--qemu_config`` flag. When used, this flag replaces the +``--arch`` flag. For example: using +``tools/testing/kunit/qemu_configs/x86_64.py``, the invocation appear +as + +.. code-block:: bash + + ./tools/testing/kunit/kunit.py run \ + --timeout=60 \ + --jobs=12 \ + --qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py + +Running command-line arguments +============================== + +kunit_tool has a number of other command-line arguments which can +be useful for our test environment. Below are the most commonly used +command line arguments: + +- ``--help``: Lists all available options. To list common options, + place ``--help`` before the command. To list options specific to that + command, place ``--help`` after the command. + + .. note:: Different commands (``config``, ``build``, ``run``, etc) + have different supported options. +- ``--build_dir``: Specifies kunit_tool build directory. It includes + the ``.kunitconfig``, ``.config`` files and compiled kernel. + +- ``--make_options``: Specifies additional options to pass to make, when + compiling a kernel (using ``build`` or ``run`` commands). For example: + to enable compiler warnings, we can pass ``--make_options W=1``. + +- ``--alltests``: Enable a predefined set of options in order to build + as many tests as possible. + + .. note:: The list of enabled options can be found in + ``tools/testing/kunit/configs/all_tests.config``. + + If you only want to enable all tests with otherwise satisfied + dependencies, instead add ``CONFIG_KUNIT_ALL_TESTS=y`` to your + ``.kunitconfig``. + +- ``--kunitconfig``: Specifies the path or the directory of the ``.kunitconfig`` + file. For example: + + - ``lib/kunit/.kunitconfig`` can be the path of the file. + + - ``lib/kunit`` can be the directory in which the file is located. + + This file is used to build and run with a predefined set of tests + and their dependencies. For example, to run tests for a given subsystem. + +- ``--kconfig_add``: Specifies additional configuration options to be + appended to the ``.kunitconfig`` file. For example: + + .. code-block:: + + ./tools/testing/kunit/kunit.py run --kconfig_add CONFIG_KASAN=y + +- ``--arch``: Runs tests on the specified architecture. The architecture + argument is same as the Kbuild ARCH environment variable. + For example, i386, x86_64, arm, um, etc. Non-UML architectures run on qemu. + Default is `um`. + +- ``--cross_compile``: Specifies the Kbuild toolchain. It passes the + same argument as passed to the ``CROSS_COMPILE`` variable used by + Kbuild. This will be the prefix for the toolchain + binaries such as GCC. For example: + + - ``sparc64-linux-gnu-`` if we have the sparc toolchain installed on + our system. + + - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux`` + if we have downloaded the microblaze toolchain from the 0-day + website to a specified path in our home directory called toolchains. + +- ``--qemu_config``: Specifies the path to a file containing a + custom qemu architecture definition. This should be a python file + containing a `QemuArchParams` object. + +- ``--qemu_args``: Specifies additional qemu arguments, for example, ``-smp 8``. + +- ``--jobs``: Specifies the number of jobs (commands) to run simultaneously. + By default, this is set to the number of cores on your system. + +- ``--timeout``: Specifies the maximum number of seconds allowed for all tests to run. + This does not include the time taken to build the tests. + +- ``--kernel_args``: Specifies additional kernel command-line arguments. May be repeated. + +- ``--run_isolated``: If set, boots the kernel for each individual suite/test. + This is useful for debugging a non-hermetic test, one that + might pass/fail based on what ran before it. + +- ``--raw_output``: If set, generates unformatted output from kernel. Possible options are: + + - ``all``: To view the full kernel output, use ``--raw_output=all``. + + - ``kunit``: This is the default option and filters to KUnit output. Use ``--raw_output`` or ``--raw_output=kunit``. + +- ``--json``: If set, stores the test results in a JSON format and prints to `stdout` or + saves to a file if a filename is specified. diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst new file mode 100644 index 000000000..8e8c493f1 --- /dev/null +++ b/Documentation/dev-tools/kunit/running_tips.rst @@ -0,0 +1,264 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================ +Tips For Running KUnit Tests +============================ + +Using ``kunit.py run`` ("kunit tool") +===================================== + +Running from any directory +-------------------------- + +It can be handy to create a bash function like: + +.. code-block:: bash + + function run_kunit() { + ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run "$@" ) + } + +.. note:: + Early versions of ``kunit.py`` (before 5.6) didn't work unless run from + the kernel root, hence the use of a subshell and ``cd``. + +Running a subset of tests +------------------------- + +``kunit.py run`` accepts an optional glob argument to filter tests. The format +is ``"<suite_glob>[.test_glob]"``. + +Say that we wanted to run the sysctl tests, we could do so via: + +.. code-block:: bash + + $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig + $ ./tools/testing/kunit/kunit.py run 'sysctl*' + +We can filter down to just the "write" tests via: + +.. code-block:: bash + + $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig + $ ./tools/testing/kunit/kunit.py run 'sysctl*.*write*' + +We're paying the cost of building more tests than we need this way, but it's +easier than fiddling with ``.kunitconfig`` files or commenting out +``kunit_suite``'s. + +However, if we wanted to define a set of tests in a less ad hoc way, the next +tip is useful. + +Defining a set of tests +----------------------- + +``kunit.py run`` (along with ``build``, and ``config``) supports a +``--kunitconfig`` flag. So if you have a set of tests that you want to run on a +regular basis (especially if they have other dependencies), you can create a +specific ``.kunitconfig`` for them. + +E.g. kunit has one for its tests: + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig + +Alternatively, if you're following the convention of naming your +file ``.kunitconfig``, you can just pass in the dir, e.g. + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit + +.. note:: + This is a relatively new feature (5.12+) so we don't have any + conventions yet about on what files should be checked in versus just + kept around locally. It's up to you and your maintainer to decide if a + config is useful enough to submit (and therefore have to maintain). + +.. note:: + Having ``.kunitconfig`` fragments in a parent and child directory is + iffy. There's discussion about adding an "import" statement in these + files to make it possible to have a top-level config run tests from all + child directories. But that would mean ``.kunitconfig`` files are no + longer just simple .config fragments. + + One alternative would be to have kunit tool recursively combine configs + automagically, but tests could theoretically depend on incompatible + options, so handling that would be tricky. + +Setting kernel commandline parameters +------------------------------------- + +You can use ``--kernel_args`` to pass arbitrary kernel arguments, e.g. + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py run --kernel_args=param=42 --kernel_args=param2=false + + +Generating code coverage reports under UML +------------------------------------------ + +.. note:: + TODO(brendanhiggins@google.com): There are various issues with UML and + versions of gcc 7 and up. You're likely to run into missing ``.gcda`` + files or compile errors. + +This is different from the "normal" way of getting coverage information that is +documented in Documentation/dev-tools/gcov.rst. + +Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options: + +.. code-block:: none + + CONFIG_DEBUG_KERNEL=y + CONFIG_DEBUG_INFO=y + CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y + CONFIG_GCOV=y + + +Putting it together into a copy-pastable sequence of commands: + +.. code-block:: bash + + # Append coverage options to the current config + $ ./tools/testing/kunit/kunit.py run --kunitconfig=.kunit/ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config + # Extract the coverage information from the build dir (.kunit/) + $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ + + # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y + # E.g. can generate an HTML report in a tmp dir like so: + $ genhtml -o /tmp/coverage_html coverage.info + + +If your installed version of gcc doesn't work, you can tweak the steps: + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6 + $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6 + + +Running tests manually +====================== + +Running tests without using ``kunit.py run`` is also an important use case. +Currently it's your only option if you want to test on architectures other than +UML. + +As running the tests under UML is fairly straightforward (configure and compile +the kernel, run the ``./linux`` binary), this section will focus on testing +non-UML architectures. + + +Running built-in tests +---------------------- + +When setting tests to ``=y``, the tests will run as part of boot and print +results to dmesg in TAP format. So you just need to add your tests to your +``.config``, build and boot your kernel as normal. + +So if we compiled our kernel with: + +.. code-block:: none + + CONFIG_KUNIT=y + CONFIG_KUNIT_EXAMPLE_TEST=y + +Then we'd see output like this in dmesg signaling the test ran and passed: + +.. code-block:: none + + TAP version 14 + 1..1 + # Subtest: example + 1..1 + # example_simple_test: initializing + ok 1 - example_simple_test + ok 1 - example + +Running tests as modules +------------------------ + +Depending on the tests, you can build them as loadable modules. + +For example, we'd change the config options from before to + +.. code-block:: none + + CONFIG_KUNIT=y + CONFIG_KUNIT_EXAMPLE_TEST=m + +Then after booting into our kernel, we can run the test via + +.. code-block:: none + + $ modprobe kunit-example-test + +This will then cause it to print TAP output to stdout. + +.. note:: + The ``modprobe`` will *not* have a non-zero exit code if any test + failed (as of 5.13). But ``kunit.py parse`` would, see below. + +.. note:: + You can set ``CONFIG_KUNIT=m`` as well, however, some features will not + work and thus some tests might break. Ideally tests would specify they + depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case + most test authors won't think about. + As of 5.13, the only difference is that ``current->kunit_test`` will + not exist. + +Pretty-printing results +----------------------- + +You can use ``kunit.py parse`` to parse dmesg for test output and print out +results in the same familiar format that ``kunit.py run`` does. + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg + + +Retrieving per suite results +---------------------------- + +Regardless of how you're running your tests, you can enable +``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results: + +.. code-block:: none + + CONFIG_KUNIT=y + CONFIG_KUNIT_EXAMPLE_TEST=m + CONFIG_KUNIT_DEBUGFS=y + +The results for each suite will be exposed under +``/sys/kernel/debug/kunit/<suite>/results``. +So using our example config: + +.. code-block:: bash + + $ modprobe kunit-example-test > /dev/null + $ cat /sys/kernel/debug/kunit/example/results + ... <TAP output> ... + + # After removing the module, the corresponding files will go away + $ modprobe -r kunit-example-test + $ cat /sys/kernel/debug/kunit/example/results + /sys/kernel/debug/kunit/example/results: No such file or directory + +Generating code coverage reports +-------------------------------- + +See Documentation/dev-tools/gcov.rst for details on how to do this. + +The only vaguely KUnit-specific advice here is that you probably want to build +your tests as modules. That way you can isolate the coverage from tests from +other code executed during boot, e.g. + +.. code-block:: bash + + # Reset coverage counters before running the test. + $ echo 0 > /sys/kernel/debug/gcov/reset + $ modprobe kunit-example-test diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst new file mode 100644 index 000000000..f4f504f1f --- /dev/null +++ b/Documentation/dev-tools/kunit/start.rst @@ -0,0 +1,306 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Getting Started +=============== + +This page contains an overview of the kunit_tool and KUnit framework, +teaching how to run existing tests and then how to write a simple test case, +and covers common problems users face when using KUnit for the first time. + +Installing Dependencies +======================= +KUnit has the same dependencies as the Linux kernel. As long as you can +build the kernel, you can run KUnit. + +Running tests with kunit_tool +============================= +kunit_tool is a Python script, which configures and builds a kernel, runs +tests, and formats the test results. From the kernel repository, you +can run kunit_tool: + +.. code-block:: bash + + ./tools/testing/kunit/kunit.py run + +.. note :: + You may see the following error: + "The source tree is not clean, please run 'make ARCH=um mrproper'" + + This happens because internally kunit.py specifies ``.kunit`` + (default option) as the build directory in the command ``make O=output/dir`` + through the argument ``--build_dir``. Hence, before starting an + out-of-tree build, the source tree must be clean. + + There is also the same caveat mentioned in the "Build directory for + the kernel" section of the :doc:`admin-guide </admin-guide/README>`, + that is, its use, it must be used for all invocations of ``make``. + The good news is that it can indeed be solved by running + ``make ARCH=um mrproper``, just be aware that this will delete the + current configuration and all generated files. + +If everything worked correctly, you should see the following: + +.. code-block:: + + Configuring KUnit Kernel ... + Building KUnit Kernel ... + Starting KUnit Kernel ... + +The tests will pass or fail. + +.. note :: + Because it is building a lot of sources for the first time, + the ``Building KUnit Kernel`` step may take a while. + +For detailed information on this wrapper, see: +Documentation/dev-tools/kunit/run_wrapper.rst. + +Selecting which tests to run +---------------------------- + +By default, kunit_tool runs all tests reachable with minimal configuration, +that is, using default values for most of the kconfig options. However, +you can select which tests to run by: + +- `Customizing Kconfig`_ used to compile the kernel, or +- `Filtering tests by name`_ to select specifically which compiled tests to run. + +Customizing Kconfig +~~~~~~~~~~~~~~~~~~~ +A good starting point for the ``.kunitconfig`` is the KUnit default config. +If you didn't run ``kunit.py run`` yet, you can generate it by running: + +.. code-block:: bash + + cd $PATH_TO_LINUX_REPO + tools/testing/kunit/kunit.py config + cat .kunit/.kunitconfig + +.. note :: + ``.kunitconfig`` lives in the ``--build_dir`` used by kunit.py, which is + ``.kunit`` by default. + +Before running the tests, kunit_tool ensures that all config options +set in ``.kunitconfig`` are set in the kernel ``.config``. It will warn +you if you have not included dependencies for the options used. + +There are many ways to customize the configurations: + +a. Edit ``.kunit/.kunitconfig``. The file should contain the list of kconfig + options required to run the desired tests, including their dependencies. + You may want to remove CONFIG_KUNIT_ALL_TESTS from the ``.kunitconfig`` as + it will enable a number of additional tests that you may not want. + If you need to run on an architecture other than UML see :ref:`kunit-on-qemu`. + +b. Enable additional kconfig options on top of ``.kunit/.kunitconfig``. + For example, to include the kernel's linked-list test you can run:: + + ./tools/testing/kunit/kunit.py run \ + --kconfig_add CONFIG_LIST_KUNIT_TEST=y + +c. Provide the path of one or more .kunitconfig files from the tree. + For example, to run only ``FAT_FS`` and ``EXT4`` tests you can run:: + + ./tools/testing/kunit/kunit.py run \ + --kunitconfig ./fs/fat/.kunitconfig \ + --kunitconfig ./fs/ext4/.kunitconfig + +d. If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the + ``.config`` file. But you can edit the ``.config`` file directly or with + tools like ``make menuconfig O=.kunit``. As long as its a superset of + ``.kunitconfig``, kunit.py won't overwrite your changes. + + +.. note :: + + To save a .kunitconfig after finding a satisfactory configuration:: + + make savedefconfig O=.kunit + cp .kunit/defconfig .kunit/.kunitconfig + +Filtering tests by name +~~~~~~~~~~~~~~~~~~~~~~~ +If you want to be more specific than Kconfig can provide, it is also possible +to select which tests to execute at boot-time by passing a glob filter +(read instructions regarding the pattern in the manpage :manpage:`glob(7)`). +If there is a ``"."`` (period) in the filter, it will be interpreted as a +separator between the name of the test suite and the test case, +otherwise, it will be interpreted as the name of the test suite. +For example, let's assume we are using the default config: + +a. inform the name of a test suite, like ``"kunit_executor_test"``, + to run every test case it contains:: + + ./tools/testing/kunit/kunit.py run "kunit_executor_test" + +b. inform the name of a test case prefixed by its test suite, + like ``"example.example_simple_test"``, to run specifically that test case:: + + ./tools/testing/kunit/kunit.py run "example.example_simple_test" + +c. use wildcard characters (``*?[``) to run any test case that matches the pattern, + like ``"*.*64*"`` to run test cases containing ``"64"`` in the name inside + any test suite:: + + ./tools/testing/kunit/kunit.py run "*.*64*" + +Running Tests without the KUnit Wrapper +======================================= +If you do not want to use the KUnit Wrapper (for example: you want code +under test to integrate with other systems, or use a different/ +unsupported architecture or configuration), KUnit can be included in +any kernel, and the results are read out and parsed manually. + +.. note :: + ``CONFIG_KUNIT`` should not be enabled in a production environment. + Enabling KUnit disables Kernel Address-Space Layout Randomization + (KASLR), and tests may affect the state of the kernel in ways not + suitable for production. + +Configuring the Kernel +---------------------- +To enable KUnit itself, you need to enable the ``CONFIG_KUNIT`` Kconfig +option (under Kernel Hacking/Kernel Testing and Coverage in +``menuconfig``). From there, you can enable any KUnit tests. They +usually have config options ending in ``_KUNIT_TEST``. + +KUnit and KUnit tests can be compiled as modules. The tests in a module +will run when the module is loaded. + +Running Tests (without KUnit Wrapper) +------------------------------------- +Build and run your kernel. In the kernel log, the test output is printed +out in the TAP format. This will only happen by default if KUnit/tests +are built-in. Otherwise the module will need to be loaded. + +.. note :: + Some lines and/or data may get interspersed in the TAP output. + +Writing Your First Test +======================= +In your kernel repository, let's add some code that we can test. + +1. Create a file ``drivers/misc/example.h``, which includes: + +.. code-block:: c + + int misc_example_add(int left, int right); + +2. Create a file ``drivers/misc/example.c``, which includes: + +.. code-block:: c + + #include <linux/errno.h> + + #include "example.h" + + int misc_example_add(int left, int right) + { + return left + right; + } + +3. Add the following lines to ``drivers/misc/Kconfig``: + +.. code-block:: kconfig + + config MISC_EXAMPLE + bool "My example" + +4. Add the following lines to ``drivers/misc/Makefile``: + +.. code-block:: make + + obj-$(CONFIG_MISC_EXAMPLE) += example.o + +Now we are ready to write the test cases. + +1. Add the below test case in ``drivers/misc/example_test.c``: + +.. code-block:: c + + #include <kunit/test.h> + #include "example.h" + + /* Define the test cases. */ + + static void misc_example_add_test_basic(struct kunit *test) + { + KUNIT_EXPECT_EQ(test, 1, misc_example_add(1, 0)); + KUNIT_EXPECT_EQ(test, 2, misc_example_add(1, 1)); + KUNIT_EXPECT_EQ(test, 0, misc_example_add(-1, 1)); + KUNIT_EXPECT_EQ(test, INT_MAX, misc_example_add(0, INT_MAX)); + KUNIT_EXPECT_EQ(test, -1, misc_example_add(INT_MAX, INT_MIN)); + } + + static void misc_example_test_failure(struct kunit *test) + { + KUNIT_FAIL(test, "This test never passes."); + } + + static struct kunit_case misc_example_test_cases[] = { + KUNIT_CASE(misc_example_add_test_basic), + KUNIT_CASE(misc_example_test_failure), + {} + }; + + static struct kunit_suite misc_example_test_suite = { + .name = "misc-example", + .test_cases = misc_example_test_cases, + }; + kunit_test_suite(misc_example_test_suite); + +2. Add the following lines to ``drivers/misc/Kconfig``: + +.. code-block:: kconfig + + config MISC_EXAMPLE_TEST + tristate "Test for my example" if !KUNIT_ALL_TESTS + depends on MISC_EXAMPLE && KUNIT=y + default KUNIT_ALL_TESTS + +3. Add the following lines to ``drivers/misc/Makefile``: + +.. code-block:: make + + obj-$(CONFIG_MISC_EXAMPLE_TEST) += example_test.o + +4. Add the following lines to ``.kunit/.kunitconfig``: + +.. code-block:: none + + CONFIG_MISC_EXAMPLE=y + CONFIG_MISC_EXAMPLE_TEST=y + +5. Run the test: + +.. code-block:: bash + + ./tools/testing/kunit/kunit.py run + +You should see the following failure: + +.. code-block:: none + + ... + [16:08:57] [PASSED] misc-example:misc_example_add_test_basic + [16:08:57] [FAILED] misc-example:misc_example_test_failure + [16:08:57] EXPECTATION FAILED at drivers/misc/example-test.c:17 + [16:08:57] This test never passes. + ... + +Congrats! You just wrote your first KUnit test. + +Next Steps +========== + +* Documentation/dev-tools/kunit/architecture.rst - KUnit architecture. +* Documentation/dev-tools/kunit/run_wrapper.rst - run kunit_tool. +* Documentation/dev-tools/kunit/run_manual.rst - run tests without kunit_tool. +* Documentation/dev-tools/kunit/usage.rst - write tests. +* Documentation/dev-tools/kunit/tips.rst - best practices with + examples. +* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs + used for testing. +* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and + answers. diff --git a/Documentation/dev-tools/kunit/style.rst b/Documentation/dev-tools/kunit/style.rst new file mode 100644 index 000000000..b6d0d7359 --- /dev/null +++ b/Documentation/dev-tools/kunit/style.rst @@ -0,0 +1,202 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========================== +Test Style and Nomenclature +=========================== + +To make finding, writing, and using KUnit tests as simple as possible, it is +strongly encouraged that they are named and written according to the guidelines +below. While it is possible to write KUnit tests which do not follow these rules, +they may break some tooling, may conflict with other tests, and may not be run +automatically by testing systems. + +It is recommended that you only deviate from these guidelines when: + +1. Porting tests to KUnit which are already known with an existing name. +2. Writing tests which would cause serious problems if automatically run. For + example, non-deterministically producing false positives or negatives, or + taking a long time to run. + +Subsystems, Suites, and Tests +============================= + +To make tests easy to find, they are grouped into suites and subsystems. A test +suite is a group of tests which test a related area of the kernel. A subsystem +is a set of test suites which test different parts of a kernel subsystem +or a driver. + +Subsystems +---------- + +Every test suite must belong to a subsystem. A subsystem is a collection of one +or more KUnit test suites which test the same driver or part of the kernel. A +test subsystem should match a single kernel module. If the code being tested +cannot be compiled as a module, in many cases the subsystem should correspond to +a directory in the source tree or an entry in the ``MAINTAINERS`` file. If +unsure, follow the conventions set by tests in similar areas. + +Test subsystems should be named after the code being tested, either after the +module (wherever possible), or after the directory or files being tested. Test +subsystems should be named to avoid ambiguity where necessary. + +If a test subsystem name has multiple components, they should be separated by +underscores. *Do not* include "test" or "kunit" directly in the subsystem name +unless we are actually testing other tests or the kunit framework itself. For +example, subsystems could be called: + +``ext4`` + Matches the module and filesystem name. +``apparmor`` + Matches the module name and LSM name. +``kasan`` + Common name for the tool, prominent part of the path ``mm/kasan`` +``snd_hda_codec_hdmi`` + Has several components (``snd``, ``hda``, ``codec``, ``hdmi``) separated by + underscores. Matches the module name. + +Avoid names as shown in examples below: + +``linear-ranges`` + Names should use underscores, not dashes, to separate words. Prefer + ``linear_ranges``. +``qos-kunit-test`` + This name should use underscores, and not have "kunit-test" as a + suffix. ``qos`` is also ambiguous as a subsystem name, because several parts + of the kernel have a ``qos`` subsystem. ``power_qos`` would be a better name. +``pc_parallel_port`` + The corresponding module name is ``parport_pc``, so this subsystem should also + be named ``parport_pc``. + +.. note:: + The KUnit API and tools do not explicitly know about subsystems. They are + a way of categorizing test suites and naming modules which provides a + simple, consistent way for humans to find and run tests. This may change + in the future. + +Suites +------ + +KUnit tests are grouped into test suites, which cover a specific area of +functionality being tested. Test suites can have shared initialization and +shutdown code which is run for all tests in the suite. Not all subsystems need +to be split into multiple test suites (for example, simple drivers). + +Test suites are named after the subsystem they are part of. If a subsystem +contains several suites, the specific area under test should be appended to the +subsystem name, separated by an underscore. + +In the event that there are multiple types of test using KUnit within a +subsystem (for example, both unit tests and integration tests), they should be +put into separate suites, with the type of test as the last element in the suite +name. Unless these tests are actually present, avoid using ``_test``, ``_unittest`` +or similar in the suite name. + +The full test suite name (including the subsystem name) should be specified as +the ``.name`` member of the ``kunit_suite`` struct, and forms the base for the +module name. For example, test suites could include: + +``ext4_inode`` + Part of the ``ext4`` subsystem, testing the ``inode`` area. +``kunit_try_catch`` + Part of the ``kunit`` implementation itself, testing the ``try_catch`` area. +``apparmor_property_entry`` + Part of the ``apparmor`` subsystem, testing the ``property_entry`` area. +``kasan`` + The ``kasan`` subsystem has only one suite, so the suite name is the same as + the subsystem name. + +Avoid names, for example: + +``ext4_ext4_inode`` + There is no reason to state the subsystem twice. +``property_entry`` + The suite name is ambiguous without the subsystem name. +``kasan_integration_test`` + Because there is only one suite in the ``kasan`` subsystem, the suite should + just be called as ``kasan``. Do not redundantly add + ``integration_test``. It should be a separate test suite. For example, if the + unit tests are added, then that suite could be named as ``kasan_unittest`` or + similar. + +Test Cases +---------- + +Individual tests consist of a single function which tests a constrained +codepath, property, or function. In the test output, an individual test's +results will show up as subtests of the suite's results. + +Tests should be named after what they are testing. This is often the name of the +function being tested, with a description of the input or codepath being tested. +As tests are C functions, they should be named and written in accordance with +the kernel coding style. + +.. note:: + As tests are themselves functions, their names cannot conflict with + other C identifiers in the kernel. This may require some creative + naming. It is a good idea to make your test functions `static` to avoid + polluting the global namespace. + +Example test names include: + +``unpack_u32_with_null_name`` + Tests the ``unpack_u32`` function when a NULL name is passed in. +``test_list_splice`` + Tests the ``list_splice`` macro. It has the prefix ``test_`` to avoid a + name conflict with the macro itself. + + +Should it be necessary to refer to a test outside the context of its test suite, +the *fully-qualified* name of a test should be the suite name followed by the +test name, separated by a colon (i.e. ``suite:test``). + +Test Kconfig Entries +==================== + +Every test suite should be tied to a Kconfig entry. + +This Kconfig entry must: + +* be named ``CONFIG_<name>_KUNIT_TEST``: where <name> is the name of the test + suite. +* be listed either alongside the config entries for the driver/subsystem being + tested, or be under [Kernel Hacking]->[Kernel Testing and Coverage] +* depend on ``CONFIG_KUNIT``. +* be visible only if ``CONFIG_KUNIT_ALL_TESTS`` is not enabled. +* have a default value of ``CONFIG_KUNIT_ALL_TESTS``. +* have a brief description of KUnit in the help text. + +If we are not able to meet above conditions (for example, the test is unable to +be built as a module), Kconfig entries for tests should be tristate. + +For example, a Kconfig entry might look like: + +.. code-block:: none + + config FOO_KUNIT_TEST + tristate "KUnit test for foo" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + This builds unit tests for foo. + + For more information on KUnit and unit tests in general, + please refer to the KUnit documentation in Documentation/dev-tools/kunit/. + + If unsure, say N. + + +Test File and Module Names +========================== + +KUnit tests can often be compiled as a module. These modules should be named +after the test suite, followed by ``_test``. If this is likely to conflict with +non-KUnit tests, the suffix ``_kunit`` can also be used. + +The easiest way of achieving this is to name the file containing the test suite +``<suite>_test.c`` (or, as above, ``<suite>_kunit.c``). This file should be +placed next to the code under test. + +If the suite name contains some or all of the name of the test's parent +directory, it may make sense to modify the source filename to reduce redundancy. +For example, a ``foo_firmware`` suite could be in the ``foo/firmware_test.c`` +file. diff --git a/Documentation/dev-tools/kunit/tips.rst b/Documentation/dev-tools/kunit/tips.rst new file mode 100644 index 000000000..492d2ded2 --- /dev/null +++ b/Documentation/dev-tools/kunit/tips.rst @@ -0,0 +1,190 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================ +Tips For Writing KUnit Tests +============================ + +Exiting early on failed expectations +------------------------------------ + +``KUNIT_EXPECT_EQ`` and friends will mark the test as failed and continue +execution. In some cases, it's unsafe to continue and you can use the +``KUNIT_ASSERT`` variant to exit on failure. + +.. code-block:: c + + void example_test_user_alloc_function(struct kunit *test) + { + void *object = alloc_some_object_for_me(); + + /* Make sure we got a valid pointer back. */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, object); + do_something_with_object(object); + } + +Allocating memory +----------------- + +Where you would use ``kzalloc``, you should prefer ``kunit_kzalloc`` instead. +KUnit will ensure the memory is freed once the test completes. + +This is particularly useful since it lets you use the ``KUNIT_ASSERT_EQ`` +macros to exit early from a test without having to worry about remembering to +call ``kfree``. + +Example: + +.. code-block:: c + + void example_test_allocation(struct kunit *test) + { + char *buffer = kunit_kzalloc(test, 16, GFP_KERNEL); + /* Ensure allocation succeeded. */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buffer); + + KUNIT_ASSERT_STREQ(test, buffer, ""); + } + + +Testing static functions +------------------------ + +If you don't want to expose functions or variables just for testing, one option +is to conditionally ``#include`` the test file at the end of your .c file, e.g. + +.. code-block:: c + + /* In my_file.c */ + + static int do_interesting_thing(); + + #ifdef CONFIG_MY_KUNIT_TEST + #include "my_kunit_test.c" + #endif + +Injecting test-only code +------------------------ + +Similarly to the above, it can be useful to add test-specific logic. + +.. code-block:: c + + /* In my_file.h */ + + #ifdef CONFIG_MY_KUNIT_TEST + /* Defined in my_kunit_test.c */ + void test_only_hook(void); + #else + void test_only_hook(void) { } + #endif + +This test-only code can be made more useful by accessing the current kunit +test, see below. + +Accessing the current test +-------------------------- + +In some cases, you need to call test-only code from outside the test file, e.g. +like in the example above or if you're providing a fake implementation of an +ops struct. +There is a ``kunit_test`` field in ``task_struct``, so you can access it via +``current->kunit_test``. + +Here's a slightly in-depth example of how one could implement "mocking": + +.. code-block:: c + + #include <linux/sched.h> /* for current */ + + struct test_data { + int foo_result; + int want_foo_called_with; + }; + + static int fake_foo(int arg) + { + struct kunit *test = current->kunit_test; + struct test_data *test_data = test->priv; + + KUNIT_EXPECT_EQ(test, test_data->want_foo_called_with, arg); + return test_data->foo_result; + } + + static void example_simple_test(struct kunit *test) + { + /* Assume priv is allocated in the suite's .init */ + struct test_data *test_data = test->priv; + + test_data->foo_result = 42; + test_data->want_foo_called_with = 1; + + /* In a real test, we'd probably pass a pointer to fake_foo somewhere + * like an ops struct, etc. instead of calling it directly. */ + KUNIT_EXPECT_EQ(test, fake_foo(1), 42); + } + + +Note: here we're able to get away with using ``test->priv``, but if you wanted +something more flexible you could use a named ``kunit_resource``, see +Documentation/dev-tools/kunit/api/test.rst. + +Failing the current test +------------------------ + +But sometimes, you might just want to fail the current test. In that case, we +have ``kunit_fail_current_test(fmt, args...)`` which is defined in ``<kunit/test-bug.h>`` and +doesn't require pulling in ``<kunit/test.h>``. + +E.g. say we had an option to enable some extra debug checks on some data structure: + +.. code-block:: c + + #include <kunit/test-bug.h> + + #ifdef CONFIG_EXTRA_DEBUG_CHECKS + static void validate_my_data(struct data *data) + { + if (is_valid(data)) + return; + + kunit_fail_current_test("data %p is invalid", data); + + /* Normal, non-KUnit, error reporting code here. */ + } + #else + static void my_debug_function(void) { } + #endif + + +Customizing error messages +-------------------------- + +Each of the ``KUNIT_EXPECT`` and ``KUNIT_ASSERT`` macros have a ``_MSG`` variant. +These take a format string and arguments to provide additional context to the automatically generated error messages. + +.. code-block:: c + + char some_str[41]; + generate_sha1_hex_string(some_str); + + /* Before. Not easy to tell why the test failed. */ + KUNIT_EXPECT_EQ(test, strlen(some_str), 40); + + /* After. Now we see the offending string. */ + KUNIT_EXPECT_EQ_MSG(test, strlen(some_str), 40, "some_str='%s'", some_str); + +Alternatively, one can take full control over the error message by using ``KUNIT_FAIL()``, e.g. + +.. code-block:: c + + /* Before */ + KUNIT_EXPECT_EQ(test, some_setup_function(), 0); + + /* After: full control over the failure message. */ + if (some_setup_function()) + KUNIT_FAIL(test, "Failed to setup thing for testing"); + +Next Steps +========== +* Optional: see the Documentation/dev-tools/kunit/usage.rst page for a more + in-depth explanation of KUnit. diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst new file mode 100644 index 000000000..2737863ef --- /dev/null +++ b/Documentation/dev-tools/kunit/usage.rst @@ -0,0 +1,705 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Writing Tests +============= + +Test Cases +---------- + +The fundamental unit in KUnit is the test case. A test case is a function with +the signature ``void (*)(struct kunit *test)``. It calls the function under test +and then sets *expectations* for what should happen. For example: + +.. code-block:: c + + void example_test_success(struct kunit *test) + { + } + + void example_test_failure(struct kunit *test) + { + KUNIT_FAIL(test, "This test never passes."); + } + +In the above example, ``example_test_success`` always passes because it does +nothing; no expectations are set, and therefore all expectations pass. On the +other hand ``example_test_failure`` always fails because it calls ``KUNIT_FAIL``, +which is a special expectation that logs a message and causes the test case to +fail. + +Expectations +~~~~~~~~~~~~ +An *expectation* specifies that we expect a piece of code to do something in a +test. An expectation is called like a function. A test is made by setting +expectations about the behavior of a piece of code under test. When one or more +expectations fail, the test case fails and information about the failure is +logged. For example: + +.. code-block:: c + + void add_test_basic(struct kunit *test) + { + KUNIT_EXPECT_EQ(test, 1, add(1, 0)); + KUNIT_EXPECT_EQ(test, 2, add(1, 1)); + } + +In the above example, ``add_test_basic`` makes a number of assertions about the +behavior of a function called ``add``. The first parameter is always of type +``struct kunit *``, which contains information about the current test context. +The second parameter, in this case, is what the value is expected to be. The +last value is what the value actually is. If ``add`` passes all of these +expectations, the test case, ``add_test_basic`` will pass; if any one of these +expectations fails, the test case will fail. + +A test case *fails* when any expectation is violated; however, the test will +continue to run, and try other expectations until the test case ends or is +otherwise terminated. This is as opposed to *assertions* which are discussed +later. + +To learn about more KUnit expectations, see Documentation/dev-tools/kunit/api/test.rst. + +.. note:: + A single test case should be short, easy to understand, and focused on a + single behavior. + +For example, if we want to rigorously test the ``add`` function above, create +additional tests cases which would test each property that an ``add`` function +should have as shown below: + +.. code-block:: c + + void add_test_basic(struct kunit *test) + { + KUNIT_EXPECT_EQ(test, 1, add(1, 0)); + KUNIT_EXPECT_EQ(test, 2, add(1, 1)); + } + + void add_test_negative(struct kunit *test) + { + KUNIT_EXPECT_EQ(test, 0, add(-1, 1)); + } + + void add_test_max(struct kunit *test) + { + KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX)); + KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN)); + } + + void add_test_overflow(struct kunit *test) + { + KUNIT_EXPECT_EQ(test, INT_MIN, add(INT_MAX, 1)); + } + +Assertions +~~~~~~~~~~ + +An assertion is like an expectation, except that the assertion immediately +terminates the test case if the condition is not satisfied. For example: + +.. code-block:: c + + static void test_sort(struct kunit *test) + { + int *a, i, r = 1; + a = kunit_kmalloc_array(test, TEST_LEN, sizeof(*a), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, a); + for (i = 0; i < TEST_LEN; i++) { + r = (r * 725861) % 6599; + a[i] = r; + } + sort(a, TEST_LEN, sizeof(*a), cmpint, NULL); + for (i = 0; i < TEST_LEN-1; i++) + KUNIT_EXPECT_LE(test, a[i], a[i + 1]); + } + +In this example, the method under test should return pointer to a value. If the +pointer returns null or an errno, we want to stop the test since the following +expectation could crash the test case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us +to bail out of the test case if the appropriate conditions are not satisfied to +complete the test. + +Test Suites +~~~~~~~~~~~ + +We need many test cases covering all the unit's behaviors. It is common to have +many similar tests. In order to reduce duplication in these closely related +tests, most unit testing frameworks (including KUnit) provide the concept of a +*test suite*. A test suite is a collection of test cases for a unit of code +with optional setup and teardown functions that run before/after the whole +suite and/or every test case. For example: + +.. code-block:: c + + static struct kunit_case example_test_cases[] = { + KUNIT_CASE(example_test_foo), + KUNIT_CASE(example_test_bar), + KUNIT_CASE(example_test_baz), + {} + }; + + static struct kunit_suite example_test_suite = { + .name = "example", + .init = example_test_init, + .exit = example_test_exit, + .suite_init = example_suite_init, + .suite_exit = example_suite_exit, + .test_cases = example_test_cases, + }; + kunit_test_suite(example_test_suite); + +In the above example, the test suite ``example_test_suite`` would first run +``example_suite_init``, then run the test cases ``example_test_foo``, +``example_test_bar``, and ``example_test_baz``. Each would have +``example_test_init`` called immediately before it and ``example_test_exit`` +called immediately after it. Finally, ``example_suite_exit`` would be called +after everything else. ``kunit_test_suite(example_test_suite)`` registers the +test suite with the KUnit test framework. + +.. note:: + A test case will only run if it is associated with a test suite. + +``kunit_test_suite(...)`` is a macro which tells the linker to put the +specified test suite in a special linker section so that it can be run by KUnit +either after ``late_init``, or when the test module is loaded (if the test was +built as a module). + +For more information, see Documentation/dev-tools/kunit/api/test.rst. + +.. _kunit-on-non-uml: + +Writing Tests For Other Architectures +------------------------------------- + +It is better to write tests that run on UML to tests that only run under a +particular architecture. It is better to write tests that run under QEMU or +another easy to obtain (and monetarily free) software environment to a specific +piece of hardware. + +Nevertheless, there are still valid reasons to write a test that is architecture +or hardware specific. For example, we might want to test code that really +belongs in ``arch/some-arch/*``. Even so, try to write the test so that it does +not depend on physical hardware. Some of our test cases may not need hardware, +only few tests actually require the hardware to test it. When hardware is not +available, instead of disabling tests, we can skip them. + +Now that we have narrowed down exactly what bits are hardware specific, the +actual procedure for writing and running the tests is same as writing normal +KUnit tests. + +.. important:: + We may have to reset hardware state. If this is not possible, we may only + be able to run one test case per invocation. + +.. TODO(brendanhiggins@google.com): Add an actual example of an architecture- + dependent KUnit test. + +Common Patterns +=============== + +Isolating Behavior +------------------ + +Unit testing limits the amount of code under test to a single unit. It controls +what code gets run when the unit under test calls a function. Where a function +is exposed as part of an API such that the definition of that function can be +changed without affecting the rest of the code base. In the kernel, this comes +from two constructs: classes, which are structs that contain function pointers +provided by the implementer, and architecture-specific functions, which have +definitions selected at compile time. + +Classes +~~~~~~~ + +Classes are not a construct that is built into the C programming language; +however, it is an easily derived concept. Accordingly, in most cases, every +project that does not use a standardized object oriented library (like GNOME's +GObject) has their own slightly different way of doing object oriented +programming; the Linux kernel is no exception. + +The central concept in kernel object oriented programming is the class. In the +kernel, a *class* is a struct that contains function pointers. This creates a +contract between *implementers* and *users* since it forces them to use the +same function signature without having to call the function directly. To be a +class, the function pointers must specify that a pointer to the class, known as +a *class handle*, be one of the parameters. Thus the member functions (also +known as *methods*) have access to member variables (also known as *fields*) +allowing the same implementation to have multiple *instances*. + +A class can be *overridden* by *child classes* by embedding the *parent class* +in the child class. Then when the child class *method* is called, the child +implementation knows that the pointer passed to it is of a parent contained +within the child. Thus, the child can compute the pointer to itself because the +pointer to the parent is always a fixed offset from the pointer to the child. +This offset is the offset of the parent contained in the child struct. For +example: + +.. code-block:: c + + struct shape { + int (*area)(struct shape *this); + }; + + struct rectangle { + struct shape parent; + int length; + int width; + }; + + int rectangle_area(struct shape *this) + { + struct rectangle *self = container_of(this, struct rectangle, parent); + + return self->length * self->width; + }; + + void rectangle_new(struct rectangle *self, int length, int width) + { + self->parent.area = rectangle_area; + self->length = length; + self->width = width; + } + +In this example, computing the pointer to the child from the pointer to the +parent is done by ``container_of``. + +Faking Classes +~~~~~~~~~~~~~~ + +In order to unit test a piece of code that calls a method in a class, the +behavior of the method must be controllable, otherwise the test ceases to be a +unit test and becomes an integration test. + +A fake class implements a piece of code that is different than what runs in a +production instance, but behaves identical from the standpoint of the callers. +This is done to replace a dependency that is hard to deal with, or is slow. For +example, implementing a fake EEPROM that stores the "contents" in an +internal buffer. Assume we have a class that represents an EEPROM: + +.. code-block:: c + + struct eeprom { + ssize_t (*read)(struct eeprom *this, size_t offset, char *buffer, size_t count); + ssize_t (*write)(struct eeprom *this, size_t offset, const char *buffer, size_t count); + }; + +And we want to test code that buffers writes to the EEPROM: + +.. code-block:: c + + struct eeprom_buffer { + ssize_t (*write)(struct eeprom_buffer *this, const char *buffer, size_t count); + int flush(struct eeprom_buffer *this); + size_t flush_count; /* Flushes when buffer exceeds flush_count. */ + }; + + struct eeprom_buffer *new_eeprom_buffer(struct eeprom *eeprom); + void destroy_eeprom_buffer(struct eeprom *eeprom); + +We can test this code by *faking out* the underlying EEPROM: + +.. code-block:: c + + struct fake_eeprom { + struct eeprom parent; + char contents[FAKE_EEPROM_CONTENTS_SIZE]; + }; + + ssize_t fake_eeprom_read(struct eeprom *parent, size_t offset, char *buffer, size_t count) + { + struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent); + + count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset); + memcpy(buffer, this->contents + offset, count); + + return count; + } + + ssize_t fake_eeprom_write(struct eeprom *parent, size_t offset, const char *buffer, size_t count) + { + struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent); + + count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset); + memcpy(this->contents + offset, buffer, count); + + return count; + } + + void fake_eeprom_init(struct fake_eeprom *this) + { + this->parent.read = fake_eeprom_read; + this->parent.write = fake_eeprom_write; + memset(this->contents, 0, FAKE_EEPROM_CONTENTS_SIZE); + } + +We can now use it to test ``struct eeprom_buffer``: + +.. code-block:: c + + struct eeprom_buffer_test { + struct fake_eeprom *fake_eeprom; + struct eeprom_buffer *eeprom_buffer; + }; + + static void eeprom_buffer_test_does_not_write_until_flush(struct kunit *test) + { + struct eeprom_buffer_test *ctx = test->priv; + struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; + struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; + char buffer[] = {0xff}; + + eeprom_buffer->flush_count = SIZE_MAX; + + eeprom_buffer->write(eeprom_buffer, buffer, 1); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); + + eeprom_buffer->write(eeprom_buffer, buffer, 1); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0); + + eeprom_buffer->flush(eeprom_buffer); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); + } + + static void eeprom_buffer_test_flushes_after_flush_count_met(struct kunit *test) + { + struct eeprom_buffer_test *ctx = test->priv; + struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; + struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; + char buffer[] = {0xff}; + + eeprom_buffer->flush_count = 2; + + eeprom_buffer->write(eeprom_buffer, buffer, 1); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); + + eeprom_buffer->write(eeprom_buffer, buffer, 1); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); + } + + static void eeprom_buffer_test_flushes_increments_of_flush_count(struct kunit *test) + { + struct eeprom_buffer_test *ctx = test->priv; + struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; + struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; + char buffer[] = {0xff, 0xff}; + + eeprom_buffer->flush_count = 2; + + eeprom_buffer->write(eeprom_buffer, buffer, 1); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); + + eeprom_buffer->write(eeprom_buffer, buffer, 2); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); + /* Should have only flushed the first two bytes. */ + KUNIT_EXPECT_EQ(test, fake_eeprom->contents[2], 0); + } + + static int eeprom_buffer_test_init(struct kunit *test) + { + struct eeprom_buffer_test *ctx; + + ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx); + + ctx->fake_eeprom = kunit_kzalloc(test, sizeof(*ctx->fake_eeprom), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->fake_eeprom); + fake_eeprom_init(ctx->fake_eeprom); + + ctx->eeprom_buffer = new_eeprom_buffer(&ctx->fake_eeprom->parent); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->eeprom_buffer); + + test->priv = ctx; + + return 0; + } + + static void eeprom_buffer_test_exit(struct kunit *test) + { + struct eeprom_buffer_test *ctx = test->priv; + + destroy_eeprom_buffer(ctx->eeprom_buffer); + } + +Testing Against Multiple Inputs +------------------------------- + +Testing just a few inputs is not enough to ensure that the code works correctly, +for example: testing a hash function. + +We can write a helper macro or function. The function is called for each input. +For example, to test ``sha1sum(1)``, we can write: + +.. code-block:: c + + #define TEST_SHA1(in, want) \ + sha1sum(in, out); \ + KUNIT_EXPECT_STREQ_MSG(test, out, want, "sha1sum(%s)", in); + + char out[40]; + TEST_SHA1("hello world", "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed"); + TEST_SHA1("hello world!", "430ce34d020724ed75a196dfc2ad67c77772d169"); + +Note the use of the ``_MSG`` version of ``KUNIT_EXPECT_STREQ`` to print a more +detailed error and make the assertions clearer within the helper macros. + +The ``_MSG`` variants are useful when the same expectation is called multiple +times (in a loop or helper function) and thus the line number is not enough to +identify what failed, as shown below. + +In complicated cases, we recommend using a *table-driven test* compared to the +helper macro variation, for example: + +.. code-block:: c + + int i; + char out[40]; + + struct sha1_test_case { + const char *str; + const char *sha1; + }; + + struct sha1_test_case cases[] = { + { + .str = "hello world", + .sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", + }, + { + .str = "hello world!", + .sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169", + }, + }; + for (i = 0; i < ARRAY_SIZE(cases); ++i) { + sha1sum(cases[i].str, out); + KUNIT_EXPECT_STREQ_MSG(test, out, cases[i].sha1, + "sha1sum(%s)", cases[i].str); + } + + +There is more boilerplate code involved, but it can: + +* be more readable when there are multiple inputs/outputs (due to field names). + + * For example, see ``fs/ext4/inode-test.c``. + +* reduce duplication if test cases are shared across multiple tests. + + * For example: if we want to test ``sha256sum``, we could add a ``sha256`` + field and reuse ``cases``. + +* be converted to a "parameterized test". + +Parameterized Testing +~~~~~~~~~~~~~~~~~~~~~ + +The table-driven testing pattern is common enough that KUnit has special +support for it. + +By reusing the same ``cases`` array from above, we can write the test as a +"parameterized test" with the following. + +.. code-block:: c + + // This is copy-pasted from above. + struct sha1_test_case { + const char *str; + const char *sha1; + }; + const struct sha1_test_case cases[] = { + { + .str = "hello world", + .sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", + }, + { + .str = "hello world!", + .sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169", + }, + }; + + // Need a helper function to generate a name for each test case. + static void case_to_desc(const struct sha1_test_case *t, char *desc) + { + strcpy(desc, t->str); + } + // Creates `sha1_gen_params()` to iterate over `cases`. + KUNIT_ARRAY_PARAM(sha1, cases, case_to_desc); + + // Looks no different from a normal test. + static void sha1_test(struct kunit *test) + { + // This function can just contain the body of the for-loop. + // The former `cases[i]` is accessible under test->param_value. + char out[40]; + struct sha1_test_case *test_param = (struct sha1_test_case *)(test->param_value); + + sha1sum(test_param->str, out); + KUNIT_EXPECT_STREQ_MSG(test, out, test_param->sha1, + "sha1sum(%s)", test_param->str); + } + + // Instead of KUNIT_CASE, we use KUNIT_CASE_PARAM and pass in the + // function declared by KUNIT_ARRAY_PARAM. + static struct kunit_case sha1_test_cases[] = { + KUNIT_CASE_PARAM(sha1_test, sha1_gen_params), + {} + }; + +Exiting Early on Failed Expectations +------------------------------------ + +We can use ``KUNIT_EXPECT_EQ`` to mark the test as failed and continue +execution. In some cases, it is unsafe to continue. We can use the +``KUNIT_ASSERT`` variant to exit on failure. + +.. code-block:: c + + void example_test_user_alloc_function(struct kunit *test) + { + void *object = alloc_some_object_for_me(); + + /* Make sure we got a valid pointer back. */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, object); + do_something_with_object(object); + } + +Allocating Memory +----------------- + +Where you might use ``kzalloc``, you can instead use ``kunit_kzalloc`` as KUnit +will then ensure that the memory is freed once the test completes. + +This is useful because it lets us use the ``KUNIT_ASSERT_EQ`` macros to exit +early from a test without having to worry about remembering to call ``kfree``. +For example: + +.. code-block:: c + + void example_test_allocation(struct kunit *test) + { + char *buffer = kunit_kzalloc(test, 16, GFP_KERNEL); + /* Ensure allocation succeeded. */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buffer); + + KUNIT_ASSERT_STREQ(test, buffer, ""); + } + + +Testing Static Functions +------------------------ + +If we do not want to expose functions or variables for testing, one option is to +conditionally ``#include`` the test file at the end of your .c file. For +example: + +.. code-block:: c + + /* In my_file.c */ + + static int do_interesting_thing(); + + #ifdef CONFIG_MY_KUNIT_TEST + #include "my_kunit_test.c" + #endif + +Injecting Test-Only Code +------------------------ + +Similar to as shown above, we can add test-specific logic. For example: + +.. code-block:: c + + /* In my_file.h */ + + #ifdef CONFIG_MY_KUNIT_TEST + /* Defined in my_kunit_test.c */ + void test_only_hook(void); + #else + void test_only_hook(void) { } + #endif + +This test-only code can be made more useful by accessing the current ``kunit_test`` +as shown in next section: *Accessing The Current Test*. + +Accessing The Current Test +-------------------------- + +In some cases, we need to call test-only code from outside the test file. +For example, see example in section *Injecting Test-Only Code* or if +we are providing a fake implementation of an ops struct. Using +``kunit_test`` field in ``task_struct``, we can access it via +``current->kunit_test``. + +The example below includes how to implement "mocking": + +.. code-block:: c + + #include <linux/sched.h> /* for current */ + + struct test_data { + int foo_result; + int want_foo_called_with; + }; + + static int fake_foo(int arg) + { + struct kunit *test = current->kunit_test; + struct test_data *test_data = test->priv; + + KUNIT_EXPECT_EQ(test, test_data->want_foo_called_with, arg); + return test_data->foo_result; + } + + static void example_simple_test(struct kunit *test) + { + /* Assume priv (private, a member used to pass test data from + * the init function) is allocated in the suite's .init */ + struct test_data *test_data = test->priv; + + test_data->foo_result = 42; + test_data->want_foo_called_with = 1; + + /* In a real test, we'd probably pass a pointer to fake_foo somewhere + * like an ops struct, etc. instead of calling it directly. */ + KUNIT_EXPECT_EQ(test, fake_foo(1), 42); + } + +In this example, we are using the ``priv`` member of ``struct kunit`` as a way +of passing data to the test from the init function. In general ``priv`` is +pointer that can be used for any user data. This is preferred over static +variables, as it avoids concurrency issues. + +Had we wanted something more flexible, we could have used a named ``kunit_resource``. +Each test can have multiple resources which have string names providing the same +flexibility as a ``priv`` member, but also, for example, allowing helper +functions to create resources without conflicting with each other. It is also +possible to define a clean up function for each resource, making it easy to +avoid resource leaks. For more information, see Documentation/dev-tools/kunit/api/test.rst. + +Failing The Current Test +------------------------ + +If we want to fail the current test, we can use ``kunit_fail_current_test(fmt, args...)`` +which is defined in ``<kunit/test-bug.h>`` and does not require pulling in ``<kunit/test.h>``. +For example, we have an option to enable some extra debug checks on some data +structures as shown below: + +.. code-block:: c + + #include <kunit/test-bug.h> + + #ifdef CONFIG_EXTRA_DEBUG_CHECKS + static void validate_my_data(struct data *data) + { + if (is_valid(data)) + return; + + kunit_fail_current_test("data %p is invalid", data); + + /* Normal, non-KUnit, error reporting code here. */ + } + #else + static void my_debug_function(void) { } + #endif + |
