horok/qemu
1
0
Fork 0
Code Activity

Adding upstream version 1:10.0.2+ds.

Browse source 
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
This commit is contained in:
Daniel Baumann 2025-06-22 14:27:05 +02:00
parent bf2768bd0f
commit ea34ddeea6
Signed by: daniel.baumann
GPG key ID: BCC918A2ABD66424
37998 changed files with 9510514 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

155
docs/devel/testing/acpi-bits.rst Normal file
View file

@ -0,0 +1,155 @@
==================================
ACPI/SMBIOS testing using biosbits
==================================
************
Introduction
************
Biosbits is a software written by Josh Triplett that can be downloaded
from https://biosbits.org/. The github codebase can be found
`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that
executes the bios components such as acpi and smbios tables directly through
acpica bios interpreter (a freely available C based library written by Intel,
downloadable from https://acpica.org/ and is included with biosbits) without an
operating system getting involved in between. Bios-bits has python integration
with grub so actual routines that executes bios components can be written in
python instead of bash-ish (grub's native scripting language).
There are several advantages to directly testing the bios in a real physical
machine or in a VM as opposed to indirectly discovering bios issues through the
operating system (the OS). Operating systems tend to bypass bios problems and
hide them from the end user. We have more control of what we wanted to test and
how by being as close to the bios on a running system as possible without a
complicated software component such as an operating system coming in between.
Another issue is that we cannot exercise bios components such as ACPI and
SMBIOS without being in the highest hardware privilege level, ring 0 for
example in case of x86. Since the OS executes from ring 0 whereas normal user
land software resides in unprivileged ring 3, operating system must be modified
in order to write our test routines that exercise and test the bios. This is
not possible in all cases. Lastly, test frameworks and routines are preferably
written using a high level scripting language such as python. OSes and
OS modules are generally written using low level languages such as C and
low level assembly machine language. Writing test routines in a low level
language makes things more cumbersome. These and other reasons makes using
bios-bits very attractive for testing bioses. More details on the inspiration
for developing biosbits and its real life uses were presented `at Plumbers
in 2011 <Plumbers_>`__ and `at Linux.conf.au in 2012 <Linux.conf.au_>`__.
For QEMU, we maintain a fork of bios bits in `gitlab`_, along with all
the dependent submodules. This fork contains numerous fixes, a newer
acpica and changes specific to running these functional QEMU tests using
bits. The author of this document is the current maintainer of the QEMU
fork of bios bits repository. For more information, please see `the
author's FOSDEM presentation <FOSDEM_>`__ on this bios-bits based test framework.
.. _Plumbers: https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
.. _Linux.conf.au: https://www.youtube.com/watch?v=36QIepyUuhg
.. _gitlab: https://gitlab.com/qemu-project/biosbits-bits
.. _FOSDEM: https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/
*********************************
Description of the test framework
*********************************
Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU
functional test that drives all this.
A brief description of the various test files follows.
Under ``tests/functional/`` as the root we have:
::
├── acpi-bits
│ ├── bits-config
│ │ └── bits-cfg.txt
│ ├── bits-tests
│ ├── smbios.py2
│ ├── testacpi.py2
│ └── testcpuid.py2
├── test_acpi_bits.py
* ``tests/functional``:
``test_acpi_bits.py``:
This is the main python functional test script that generates a
biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
test failures. This is the script one would be interested in if they wanted
to add or change some component of the log parsing, add a new command line
to alter how QEMU is spawned etc. Test writers typically would not need to
modify this script unless they wanted to enhance or change the log parsing
for their tests. In order to enable debugging, you can set **V=1**
environment variable. This enables verbose mode for the test and also dumps
the entire log from bios bits and more information in case failure happens.
You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable
verbose logs and also retain the temporary work directory the test used for
you to inspect and run the specific commands manually.
In order to run this test, please perform the following steps from the QEMU
build directory (assuming that the sources are in ".."):
::
$ export PYTHONPATH=../python:../tests/functional
$ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
$ python3 ../tests/functional/test_acpi_bits.py
The above will run all acpi-bits functional tests (producing output in
tap format).
You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/
for more information about the run or in order to diagnoze issues.
If you pass V=1 in the environment, more diagnostic logs will be put into
the test log.
* ``tests/functional/acpi-bits/bits-config``:
This location contains biosbits configuration files that determine how the
software runs the tests.
``bits-config.txt``:
This is the biosbits config file that determines what tests
or actions are performed by bits. The description of the config options are
provided in the file itself.
* ``tests/functional/acpi-bits/bits-tests``:
This directory contains biosbits python based tests that are run from within
the biosbits environment in the spawned VM. New additions of test cases can
be made in the appropriate test file. For example, new acpi tests can go
into testacpi.py2 and one would call testsuite.add_test() to register the new
test so that it gets executed as a part of the ACPI tests.
It might be occasionally necessary to disable some subtests or add a new
test that belongs to a test suite not already present in this directory. To
do this, please clone the bits source from
https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
Note that this is the "qemu-bits" branch and not the "bits" branch of the
repository. "qemu-bits" is the branch where we have made all the QEMU
specific enhancements and we must use the source from this branch only.
Copy the test suite/script that needs modification (addition of new tests
or disabling them) from python directory into this directory. For
example, in order to change cpuid related tests, copy the following
file into this directory and rename it with .py2 extension:
https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
Then make your additions and changes here. Therefore, the steps are:
(a) Copy unmodified test script to this directory from bits source.
(b) Add a SPDX license header.
(c) Perform modifications to the test.
Commits (a), (b) and (c) preferably should go under separate commits so that
the original test script and the changes we have made are separated and
clear. (a) and (b) can sometimes be combined into a single step.
The test framework will then use your modified test script to run the test.
No further changes would be needed. Please check the logs to make sure that
appropriate changes have taken effect.
The tests have an extension .py2 in order to indicate that:
(a) They are python2.7 based scripts and not python 3 scripts.
(b) They are run from within the bios bits VM and is not subjected to QEMU
build/test python script maintenance and dependency resolutions.
(c) They need not be loaded by the test framework by accident when running
tests.
Author: Ani Sinha <anisinha@redhat.com>

581
docs/devel/testing/avocado.rst Normal file
View file

@ -0,0 +1,581 @@
.. _checkavocado-ref:
Integration testing with Avocado
================================
The ``tests/avocado`` directory hosts integration tests. They're usually
higher level tests, and may interact with external resources and with
various guest operating systems.
These tests are written using the Avocado Testing Framework (which must be
installed separately) in conjunction with a the ``avocado_qemu.QemuSystemTest``
class, implemented at ``tests/avocado/avocado_qemu``.
Tests based on ``avocado_qemu.QemuSystemTest`` can easily:
* Customize the command line arguments given to the convenience
``self.vm`` attribute (a QEMUMachine instance)
* Interact with the QEMU monitor, send QMP commands and check
their results
* Interact with the guest OS, using the convenience console device
(which may be useful to assert the effectiveness and correctness of
command line arguments or QMP commands)
* Interact with external data files that accompany the test itself
(see ``self.get_data()``)
* Download (and cache) remote data files, such as firmware and kernel
images
* Have access to a library of guest OS images (by means of the
``avocado.utils.vmimage`` library)
* Make use of various other test related utilities available at the
test class itself and at the utility library:
- http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
- http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
Running tests
-------------
You can run the avocado tests simply by executing:
.. code::
make check-avocado
This involves the automatic installation, from PyPI, of all the
necessary avocado-framework dependencies into the QEMU venv within the
build tree (at ``./pyvenv``). Test results are also saved within the
build tree (at ``tests/results``).
Note: the build environment must be using a Python 3 stack, and have
the ``venv`` and ``pip`` packages installed. If necessary, make sure
``configure`` is called with ``--python=`` and that those modules are
available. On Debian and Ubuntu based systems, depending on the
specific version, they may be on packages named ``python3-venv`` and
``python3-pip``.
It is also possible to run tests based on tags using the
``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
variable:
.. code::
make check-avocado AVOCADO_TAGS=quick
Note that tags separated with commas have an AND behavior, while tags
separated by spaces have an OR behavior. For more information on Avocado
tags, see:
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
To run a single test file, a couple of them, or a test within a file
using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
environment variable with the test files or test names. To run all
tests from a single file, use:
.. code::
make check-avocado AVOCADO_TESTS=$FILEPATH
The same is valid to run tests from multiple test files:
.. code::
make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
To run a single test within a file, use:
.. code::
make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
The same is valid to run single tests from multiple test files:
.. code::
make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
The scripts installed inside the virtual environment may be used
without an "activation". For instance, the Avocado test runner
may be invoked by running:
.. code::
pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
Note that if ``make check-avocado`` was not executed before, it is
possible to create the Python virtual environment with the dependencies
needed running:
.. code::
make check-venv
It is also possible to run tests from a single file or a single test within
a test file. To run tests from a single file within the build tree, use:
.. code::
pyvenv/bin/avocado run tests/avocado/$TESTFILE
To run a single test within a test file, use:
.. code::
pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
Valid test names are visible in the output from any previous execution
of Avocado or ``make check-avocado``, and can also be queried using:
.. code::
pyvenv/bin/avocado list tests/avocado
Manual Installation
-------------------
To manually install Avocado and its dependencies, run:
.. code::
pip install --user avocado-framework
Alternatively, follow the instructions on this link:
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
Overview
--------
The ``tests/avocado/avocado_qemu`` directory provides the
``avocado_qemu`` Python module, containing the ``avocado_qemu.QemuSystemTest``
class. Here's a simple usage example:
.. code::
from avocado_qemu import QemuSystemTest
class Version(QemuSystemTest):
"""
:avocado: tags=quick
"""
def test_qmp_human_info_version(self):
self.vm.launch()
res = self.vm.cmd('human-monitor-command',
command_line='info version')
self.assertRegex(res, r'^(\d+\.\d+\.\d)')
To execute your test, run:
.. code::
avocado run version.py
Tests may be classified according to a convention by using docstring
directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests
in the current directory, tagged as "quick", run:
.. code::
avocado run -t quick .
The ``avocado_qemu.QemuSystemTest`` base test class
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``avocado_qemu.QemuSystemTest`` class has a number of characteristics
that are worth being mentioned right away.
First of all, it attempts to give each test a ready to use QEMUMachine
instance, available at ``self.vm``. Because many tests will tweak the
QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
is left to the test writer.
The base test class has also support for tests with more than one
QEMUMachine. The way to get machines is through the ``self.get_vm()``
method which will return a QEMUMachine instance. The ``self.get_vm()``
method accepts arguments that will be passed to the QEMUMachine creation
and also an optional ``name`` attribute so you can identify a specific
machine and get it more than once through the tests methods. A simple
and hypothetical example follows:
.. code::
from avocado_qemu import QemuSystemTest
class MultipleMachines(QemuSystemTest):
def test_multiple_machines(self):
first_machine = self.get_vm()
second_machine = self.get_vm()
self.get_vm(name='third_machine').launch()
first_machine.launch()
second_machine.launch()
first_res = first_machine.cmd(
'human-monitor-command',
command_line='info version')
second_res = second_machine.cmd(
'human-monitor-command',
command_line='info version')
third_res = self.get_vm(name='third_machine').cmd(
'human-monitor-command',
command_line='info version')
self.assertEqual(first_res, second_res, third_res)
At test "tear down", ``avocado_qemu.QemuSystemTest`` handles all the
QEMUMachines shutdown.
The ``avocado_qemu.LinuxTest`` base test class
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``avocado_qemu.LinuxTest`` is further specialization of the
``avocado_qemu.QemuSystemTest`` class, so it contains all the characteristics
of the later plus some extra features.
First of all, this base class is intended for tests that need to
interact with a fully booted and operational Linux guest. At this
time, it uses a Fedora 31 guest image. The most basic example looks
like this:
.. code::
from avocado_qemu import LinuxTest
class SomeTest(LinuxTest):
def test(self):
self.launch_and_wait()
self.ssh_command('some_command_to_be_run_in_the_guest')
Please refer to tests that use ``avocado_qemu.LinuxTest`` under
``tests/avocado`` for more examples.
QEMUMachine
-----------
The QEMUMachine API is already widely used in the Python iotests,
device-crash-test and other Python scripts. It's a wrapper around the
execution of a QEMU binary, giving its users:
* the ability to set command line arguments to be given to the QEMU
binary
* a ready to use QMP connection and interface, which can be used to
send commands and inspect its results, as well as asynchronous
events
* convenience methods to set commonly used command line arguments in
a more succinct and intuitive way
QEMU binary selection
^^^^^^^^^^^^^^^^^^^^^
The QEMU binary used for the ``self.vm`` QEMUMachine instance will
primarily depend on the value of the ``qemu_bin`` parameter. If it's
not explicitly set, its default value will be the result of a dynamic
probe in the same source tree. A suitable binary will be one that
targets the architecture matching host machine.
Based on this description, test writers will usually rely on one of
the following approaches:
1) Set ``qemu_bin``, and use the given binary
2) Do not set ``qemu_bin``, and use a QEMU binary named like
"qemu-system-${arch}", either in the current
working directory, or in the current source tree.
The resulting ``qemu_bin`` value will be preserved in the
``avocado_qemu.QemuSystemTest`` as an attribute with the same name.
Attribute reference
-------------------
Test
^^^^
Besides the attributes and methods that are part of the base
``avocado.Test`` class, the following attributes are available on any
``avocado_qemu.QemuSystemTest`` instance.
vm
""
A QEMUMachine instance, initially configured according to the given
``qemu_bin`` parameter.
arch
""""
The architecture can be used on different levels of the stack, e.g. by
the framework or by the test itself. At the framework level, it will
currently influence the selection of a QEMU binary (when one is not
explicitly given).
Tests are also free to use this attribute value, for their own needs.
A test may, for instance, use the same value when selecting the
architecture of a kernel or disk image to boot a VM with.
The ``arch`` attribute will be set to the test parameter of the same
name. If one is not given explicitly, it will either be set to
``None``, or, if the test is tagged with one (and only one)
``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
cpu
"""
The cpu model that will be set to all QEMUMachine instances created
by the test.
The ``cpu`` attribute will be set to the test parameter of the same
name. If one is not given explicitly, it will either be set to
``None ``, or, if the test is tagged with one (and only one)
``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
machine
"""""""
The machine type that will be set to all QEMUMachine instances created
by the test.
The ``machine`` attribute will be set to the test parameter of the same
name. If one is not given explicitly, it will either be set to
``None``, or, if the test is tagged with one (and only one)
``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
qemu_bin
""""""""
The preserved value of the ``qemu_bin`` parameter or the result of the
dynamic probe for a QEMU binary in the current working directory or
source tree.
LinuxTest
^^^^^^^^^
Besides the attributes present on the ``avocado_qemu.QemuSystemTest`` base
class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
distro
""""""
The name of the Linux distribution used as the guest image for the
test. The name should match the **Provider** column on the list
of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_version
""""""""""""""
The version of the Linux distribution as the guest image for the
test. The name should match the **Version** column on the list
of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_checksum
"""""""""""""""
The sha256 hash of the guest image file used for the test.
If this value is not set in the code or by a test parameter (with the
same name), no validation on the integrity of the image will be
performed.
Parameter reference
-------------------
To understand how Avocado parameters are accessed by tests, and how
they can be passed to tests, please refer to::
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
Parameter values can be easily seen in the log files, and will look
like the following:
.. code::
PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
Test
^^^^
arch
""""
The architecture that will influence the selection of a QEMU binary
(when one is not explicitly given).
Tests are also free to use this parameter value, for their own needs.
A test may, for instance, use the same value when selecting the
architecture of a kernel or disk image to boot a VM with.
This parameter has a direct relation with the ``arch`` attribute. If
not given, it will default to None.
cpu
"""
The cpu model that will be set to all QEMUMachine instances created
by the test.
machine
"""""""
The machine type that will be set to all QEMUMachine instances created
by the test.
qemu_bin
""""""""
The exact QEMU binary to be used on QEMUMachine.
LinuxTest
^^^^^^^^^
Besides the parameters present on the ``avocado_qemu.QemuSystemTest`` base
class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
distro
""""""
The name of the Linux distribution used as the guest image for the
test. The name should match the **Provider** column on the list
of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_version
""""""""""""""
The version of the Linux distribution as the guest image for the
test. The name should match the **Version** column on the list
of images supported by the avocado.utils.vmimage library:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
distro_checksum
"""""""""""""""
The sha256 hash of the guest image file used for the test.
If this value is not set in the code or by this parameter no
validation on the integrity of the image will be performed.
Skipping tests
--------------
The Avocado framework provides Python decorators which allow for easily skip
tests running under certain conditions. For example, on the lack of a binary
on the test system or when the running environment is a CI system. For further
information about those decorators, please refer to::
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
While the conditions for skipping tests are often specifics of each one, there
are recurring scenarios identified by the QEMU developers and the use of
environment variables became a kind of standard way to enable/disable tests.
Here is a list of the most used variables:
AVOCADO_ALLOW_LARGE_STORAGE
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tests which are going to fetch or produce assets considered *large* are not
going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
the environment.
The definition of *large* is a bit arbitrary here, but it usually means an
asset which occupies at least 1GB of size on disk when uncompressed.
SPEED
^^^^^
Tests which have a long runtime will not be run unless ``SPEED=slow`` is
exported on the environment.
The definition of *long* is a bit arbitrary here, and it depends on the
usefulness of the test too. A unique test is worth spending more time on,
small variations on existing tests perhaps less so. As a rough guide,
a test or set of similar tests which take more than 100 seconds to
complete.
AVOCADO_ALLOW_UNTRUSTED_CODE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are tests which will boot a kernel image or firmware that can be
considered not safe to run on the developer's workstation, thus they are
skipped by default. The definition of *not safe* is also arbitrary but
usually it means a blob which either its source or build process aren't
public available.
You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
order to allow tests which make use of those kind of assets.
AVOCADO_TIMEOUT_EXPECTED
^^^^^^^^^^^^^^^^^^^^^^^^
The Avocado framework has a timeout mechanism which interrupts tests to avoid the
test suite of getting stuck. The timeout value can be set via test parameter or
property defined in the test class, for further details::
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
Even though the timeout can be set by the test developer, there are some tests
that may not have a well-defined limit of time to finish under certain
conditions. For example, tests that take longer to execute when QEMU is
compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
has been used to determine whether those tests should run or not.
QEMU_TEST_FLAKY_TESTS
^^^^^^^^^^^^^^^^^^^^^
Some tests are not working reliably and thus are disabled by default.
This includes tests that don't run reliably on GitLab's CI which
usually expose real issues that are rarely seen on developer machines
due to the constraints of the CI environment. If you encounter a
similar situation then raise a bug and then mark the test as shown on
the code snippet below:
.. code::
# See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test(self):
do_something()
You can also add ``:avocado: tags=flaky`` to the test meta-data so
only the flaky tests can be run as a group:
.. code::
env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \
run tests/avocado -filter-by-tags=flaky
Tests should not live in this state forever and should either be fixed
or eventually removed.
Uninstalling Avocado
--------------------
If you've followed the manual installation instructions above, you can
easily uninstall Avocado. Start by listing the packages you have
installed::
pip list --user
And remove any package you want with::
pip uninstall <package_name>
If you've used ``make check-avocado``, the Python virtual environment where
Avocado is installed will be cleaned up as part of ``make check-clean``.

177
docs/devel/testing/blkdebug.rst Normal file
View file

@ -0,0 +1,177 @@
Block I/O error injection using ``blkdebug``
============================================
..
Copyright (C) 2014-2015 Red Hat Inc
This work is licensed under the terms of the GNU GPL, version 2 or later. See
the COPYING file in the top-level directory.
The ``blkdebug`` block driver is a rule-based error injection engine. It can be
used to exercise error code paths in block drivers including ``ENOSPC`` (out of
space) and ``EIO``.
This document gives an overview of the features available in ``blkdebug``.
Background
----------
Block drivers have many error code paths that handle I/O errors. Image formats
are especially complex since metadata I/O errors during cluster allocation or
while updating tables happen halfway through request processing and require
discipline to keep image files consistent.
Error injection allows test cases to trigger I/O errors at specific points.
This way, all error paths can be tested to make sure they are correct.
Rules
-----
The ``blkdebug`` block driver takes a list of "rules" that tell the error injection
engine when to fail an I/O request.
Each I/O request is evaluated against the rules. If a rule matches the request
then its "action" is executed.
Rules can be placed in a configuration file; the configuration file
follows the same .ini-like format used by QEMU's ``-readconfig`` option, and
each section of the file represents a rule.
The following configuration file defines a single rule::
$ cat blkdebug.conf
[inject-error]
event = "read_aio"
errno = "28"
This rule fails all aio read requests with ``ENOSPC`` (28). Note that the errno
value depends on the host. On Linux, see
``/usr/include/asm-generic/errno-base.h`` for errno values.
Invoke QEMU as follows::
$ qemu-system-x86_64
-drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \
-device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0
Rules support the following attributes:
``event``
which type of operation to match (e.g. ``read_aio``, ``write_aio``,
``flush_to_os``, ``flush_to_disk``). See `Events`_ for
information on events.
``state``
(optional) the engine must be in this state number in order for this
rule to match. See `State transitions`_ for information
on states.
``errno``
the numeric errno value to return when a request matches this rule.
The errno values depend on the host since the numeric values are not
standardized in the POSIX specification.
``sector``
(optional) a sector number that the request must overlap in order to
match this rule
``once``
(optional, default ``off``) only execute this action on the first
matching request
``immediately``
(optional, default ``off``) return a NULL ``BlockAIOCB``
pointer and fail without an errno instead. This
exercises the code path where ``BlockAIOCB`` fails and the
caller's ``BlockCompletionFunc`` is not invoked.
Events
------
Block drivers provide information about the type of I/O request they are about
to make so rules can match specific types of requests. For example, the ``qcow2``
block driver tells ``blkdebug`` when it accesses the L1 table so rules can match
only L1 table accesses and not other metadata or guest data requests.
The core events are:
``read_aio``
guest data read
``write_aio``
guest data write
``flush_to_os``
write out unwritten block driver state (e.g. cached metadata)
``flush_to_disk``
flush the host block device's disk cache
See ``qapi/block-core.json:BlkdebugEvent`` for the full list of events.
You may need to grep block driver source code to understand the
meaning of specific events.
State transitions
-----------------
There are cases where more power is needed to match a particular I/O request in
a longer sequence of requests. For example::
write_aio
flush_to_disk
write_aio
How do we match the 2nd ``write_aio`` but not the first? This is where state
transitions come in.
The error injection engine has an integer called the "state" that always starts
initialized to 1. The state integer is internal to ``blkdebug`` and cannot be
observed from outside but rules can interact with it for powerful matching
behavior.
Rules can be conditional on the current state and they can transition to a new
state.
When a rule's "state" attribute is non-zero then the current state must equal
the attribute in order for the rule to match.
For example, to match the 2nd write_aio::
[set-state]
event = "write_aio"
state = "1"
new_state = "2"
[inject-error]
event = "write_aio"
state = "2"
errno = "5"
The first ``write_aio`` request matches the ``set-state`` rule and transitions from
state 1 to state 2. Once state 2 has been entered, the ``set-state`` rule no
longer matches since it requires state 1. But the ``inject-error`` rule now
matches the next ``write_aio`` request and injects ``EIO`` (5).
State transition rules support the following attributes:
``event``
which type of operation to match (e.g. ``read_aio``, ``write_aio``,
``flush_to_os`, ``flush_to_disk``). See `Events`_ for
information on events.
``state``
(optional) the engine must be in this state number in order for this
rule to match
``new_state``
transition to this state number
Suspend and resume
------------------
Exercising code paths in block drivers may require specific ordering amongst
concurrent requests. The "breakpoint" feature allows requests to be halted on
a ``blkdebug`` event and resumed later. This makes it possible to achieve
deterministic ordering when multiple requests are in flight.
Breakpoints on ``blkdebug`` events are associated with a user-defined ``tag`` string.
This tag serves as an identifier by which the request can be resumed at a later
point.
See the ``qemu-io(1)`` ``break``, ``resume``, ``remove_break``, and ``wait_break``
commands for details.

73
docs/devel/testing/blkverify.rst Normal file
View file

@ -0,0 +1,73 @@
Block driver correctness testing with ``blkverify``
===================================================
Introduction
------------
This document describes how to use the ``blkverify`` protocol to test that a block
driver is operating correctly.
It is difficult to test and debug block drivers against real guests. Often
processes inside the guest will crash because corrupt sectors were read as part
of the executable. Other times obscure errors are raised by a program inside
the guest. These issues are extremely hard to trace back to bugs in the block
driver.
``blkverify`` solves this problem by catching data corruption inside QEMU the first
time bad data is read and reporting the disk sector that is corrupted.
How it works
------------
The ``blkverify`` protocol has two child block devices, the "test" device and the
"raw" device. Read/write operations are mirrored to both devices so their
state should always be in sync.
The "raw" device is a raw image, a flat file, that has identical starting
contents to the "test" image. The idea is that the "raw" device will handle
read/write operations correctly and not corrupt data. It can be used as a
reference for comparison against the "test" device.
After a mirrored read operation completes, ``blkverify`` will compare the data and
raise an error if it is not identical. This makes it possible to catch the
first instance where corrupt data is read.
Example
-------
Imagine raw.img has 0xcd repeated throughout its first sector::
$ ./qemu-io -c 'read -v 0 512' raw.img
00000000: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
00000010: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
[...]
000001e0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
000001f0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................
read 512/512 bytes at offset 0
512.000000 bytes, 1 ops; 0.0000 sec (97.656 MiB/sec and 200000.0000 ops/sec)
And test.img is corrupt, its first sector is zeroed when it shouldn't be::
$ ./qemu-io -c 'read -v 0 512' test.img
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[...]
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
read 512/512 bytes at offset 0
512.000000 bytes, 1 ops; 0.0000 sec (81.380 MiB/sec and 166666.6667 ops/sec)
This error is caught by ``blkverify``::
$ ./qemu-io -c 'read 0 512' blkverify:a.img:b.img
blkverify: read sector_num=0 nb_sectors=4 contents mismatch in sector 0
A more realistic scenario is verifying the installation of a guest OS::
$ ./qemu-img create raw.img 16G
$ ./qemu-img create -f qcow2 test.qcow2 16G
$ ./qemu-system-x86_64 -cdrom debian.iso \
-drive file=blkverify:raw.img:test.qcow2
If the installation is aborted when ``blkverify`` detects corruption, use ``qemu-io``
to explore the contents of the disk image at the sector in question.

121
docs/devel/testing/ci-definitions.rst.inc Normal file
View file

@ -0,0 +1,121 @@
Definition of terms
===================
This section defines the terms used in this document and correlates them with
what is currently used on QEMU.
Automated tests
---------------
An automated test is written on a test framework using its generic test
functions/classes. The test framework can run the tests and report their
success or failure [1]_.
An automated test has essentially three parts:
1. The test initialization of the parameters, where the expected parameters,
like inputs and expected results, are set up;
2. The call to the code that should be tested;
3. An assertion, comparing the result from the previous call with the expected
result set during the initialization of the parameters. If the result
matches the expected result, the test has been successful; otherwise, it has
failed.
Unit testing
------------
A unit test is responsible for exercising individual software components as a
unit, like interfaces, data structures, and functionality, uncovering errors
within the boundaries of a component. The verification effort is in the
smallest software unit and focuses on the internal processing logic and data
structures. A test case of unit tests should be designed to uncover errors due
to erroneous computations, incorrect comparisons, or improper control flow [2]_.
On QEMU, unit testing is represented by the 'check-unit' target from 'make'.
Functional testing
------------------
A functional test focuses on the functional requirement of the software.
Deriving sets of input conditions, the functional tests should fully exercise
all the functional requirements for a program. Functional testing is
complementary to other testing techniques, attempting to find errors like
incorrect or missing functions, interface errors, behavior errors, and
initialization and termination errors [3]_.
On QEMU, functional testing is represented by the 'check-qtest' target from
'make'.
System testing
--------------
System tests ensure all application elements mesh properly while the overall
functionality and performance are achieved [4]_. Some or all system components
are integrated to create a complete system to be tested as a whole. System
testing ensures that components are compatible, interact correctly, and
transfer the right data at the right time across their interfaces. As system
testing focuses on interactions, use case-based testing is a practical approach
to system testing [5]_. Note that, in some cases, system testing may require
interaction with third-party software, like operating system images, databases,
networks, and so on.
On QEMU, system testing is represented by the 'check-avocado' target from
'make'.
Flaky tests
-----------
A flaky test is defined as a test that exhibits both a passing and a failing
result with the same code on different runs. Some usual reasons for an
intermittent/flaky test are async wait, concurrency, and test order dependency
[6]_.
Gating
------
A gate restricts the move of code from one stage to another on a
test/deployment pipeline. The step move is granted with approval. The approval
can be a manual intervention or a set of tests succeeding [7]_.
On QEMU, the gating process happens during the pull request. The approval is
done by the project leader running its own set of tests. The pull request gets
merged when the tests succeed.
Continuous Integration (CI)
---------------------------
Continuous integration (CI) requires the builds of the entire application and
the execution of a comprehensive set of automated tests every time there is a
need to commit any set of changes [8]_. The automated tests can be composed of
the unit, functional, system, and other tests.
Keynotes about continuous integration (CI) [9]_:
1. System tests may depend on external software (operating system images,
firmware, database, network).
2. It may take a long time to build and test. It may be impractical to build
the system being developed several times per day.
3. If the development platform is different from the target platform, it may
not be possible to run system tests in the developer’s private workspace.
There may be differences in hardware, operating system, or installed
software. Therefore, more time is required for testing the system.
References
----------
.. [1] Sommerville, Ian (2016). Software Engineering. p. 233.
.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
A Practitioner’s Approach. p. 48, 376, 378, 381.
.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
A Practitioner’s Approach. p. 388.
.. [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
A Practitioner’s Approach. Software Engineering, p. 377.
.. [5] Sommerville, Ian (2016). Software Engineering. p. 59, 232, 240.
.. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests.
Proceedings of the 22nd ACM SIGSOFT International Symposium on
Foundations of Software Engineering. 2014.
.. [7] Humble, Jez & Farley, David (2010). Continuous Delivery:
Reliable Software Releases Through Build, Test, and Deployment, p. 122.
.. [8] Humble, Jez & Farley, David (2010). Continuous Delivery:
Reliable Software Releases Through Build, Test, and Deployment, p. 55.
.. [9] Sommerville, Ian (2016). Software Engineering. p. 743.

190
docs/devel/testing/ci-jobs.rst.inc Normal file
View file

@ -0,0 +1,190 @@
.. _ci_var:
Custom CI/CD variables
======================
QEMU CI pipelines can be tuned by setting some CI environment variables.
Set variable globally in the user's CI namespace
------------------------------------------------
Variables can be set globally in the user's CI namespace setting.
For further information about how to set these variables, please refer to::
https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
Set variable manually when pushing a branch or tag to the user's repository
---------------------------------------------------------------------------
Variables can be set manually when pushing a branch or tag, using
git-push command line arguments.
Example setting the QEMU_CI_EXAMPLE_VAR variable:
.. code::
git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
For further information about how to set these variables, please refer to::
https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
Setting aliases in your git config
----------------------------------
You can use aliases to make it easier to push branches with different
CI configurations. For example define an alias for triggering CI:
.. code::
git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
Which lets you run:
.. code::
git push-ci
to create the pipeline, or:
.. code::
git push-ci-now
to create and run the pipeline
Variable naming and grouping
----------------------------
The variables used by QEMU's CI configuration are grouped together
in a handful of namespaces
* QEMU_JOB_nnnn - variables to be defined in individual jobs
or templates, to influence the shared rules defined in the
.base_job_template.
* QEMU_CI_nnn - variables to be set by contributors in their
repository CI settings, or as git push variables, to influence
which jobs get run in a pipeline
* QEMU_CI_CONTAINER_TAG - the tag used to publish containers
in stage 1, for use by build jobs in stage 2. Defaults to
'latest', but if running pipelines for different branches
concurrently, it should be overridden per pipeline.
* QEMU_CI_UPSTREAM - gitlab namespace that is considered to be
the 'upstream'. This defaults to 'qemu-project'. Contributors
may choose to override this if they are modifying rules in
base.yml and need to validate how they will operate when in
an upstream context, as opposed to their fork context.
* nnn - other misc variables not falling into the above
categories, or using different names for historical reasons
and not yet converted.
Maintainer controlled job variables
-----------------------------------
The following variables may be set when defining a job in the
CI configuration file.
QEMU_JOB_CIRRUS
~~~~~~~~~~~~~~~
The job makes use of Cirrus CI infrastructure, requiring the
configuration setup for cirrus-run to be present in the repository
QEMU_JOB_OPTIONAL
~~~~~~~~~~~~~~~~~
The job is expected to be successful in general, but is not run
by default due to need to conserve limited CI resources. It is
available to be started manually by the contributor in the CI
pipelines UI.
QEMU_JOB_ONLY_FORKS
~~~~~~~~~~~~~~~~~~~
The job results are only of interest to contributors prior to
submitting code. They are not required as part of the gating
CI pipeline.
QEMU_JOB_SKIPPED
~~~~~~~~~~~~~~~~
The job is not reliably successful in general, so is not
currently suitable to be run by default. Ideally this should
be a temporary marker until the problems can be addressed, or
the job permanently removed.
QEMU_JOB_PUBLISH
~~~~~~~~~~~~~~~~
The job is for publishing content after a branch has been
merged into the upstream default branch.
QEMU_JOB_AVOCADO
~~~~~~~~~~~~~~~~
The job runs the Avocado integration test suite
Contributor controlled runtime variables
----------------------------------------
The following variables may be set by contributors to control
job execution
QEMU_CI
~~~~~~~
By default, no pipelines will be created on contributor forks
in order to preserve CI credits
Set this variable to 1 to create the pipelines, but leave all
the jobs to be manually started from the UI
Set this variable to 2 to create the pipelines and run all
the jobs immediately, as was the historical behaviour
QEMU_CI_AVOCADO_TESTING
~~~~~~~~~~~~~~~~~~~~~~~
By default, tests using the Avocado framework are not run automatically in
the pipelines (because multiple artifacts have to be downloaded, and if
these artifacts are not already cached, downloading them make the jobs
reach the timeout limit). Set this variable to have the tests using the
Avocado framework run automatically.
Other misc variables
--------------------
These variables are primarily to control execution of jobs on
private runners
AARCH64_RUNNER_AVAILABLE
~~~~~~~~~~~~~~~~~~~~~~~~
If you've got access to an aarch64 host that can be used as a gitlab-CI
runner, you can set this variable to enable the tests that require this
kind of host. The runner should be tagged with "aarch64".
AARCH32_RUNNER_AVAILABLE
~~~~~~~~~~~~~~~~~~~~~~~~
If you've got access to an armhf host or an arch64 host that can run
aarch32 EL0 code to be used as a gitlab-CI runner, you can set this
variable to enable the tests that require this kind of host. The
runner should be tagged with "aarch32".
S390X_RUNNER_AVAILABLE
~~~~~~~~~~~~~~~~~~~~~~
If you've got access to an IBM Z host that can be used as a gitlab-CI
runner, you can set this variable to enable the tests that require this
kind of host. The runner should be tagged with "s390x".
CCACHE_DISABLE
~~~~~~~~~~~~~~
The jobs are configured to use "ccache" by default since this typically
reduces compilation time, at the cost of increased storage. If the
use of "ccache" is suspected to be hurting the overall job execution
time, setting the "CCACHE_DISABLE=1" env variable to disable it.

116
docs/devel/testing/ci-runners.rst.inc Normal file
View file

@ -0,0 +1,116 @@
Jobs on Custom Runners
======================
Besides the jobs run under the various CI systems listed before, there
are a number additional jobs that will run before an actual merge.
These use the same GitLab CI's service/framework already used for all
other GitLab based CI jobs, but rely on additional systems, not the
ones provided by GitLab as "shared runners".
The architecture of GitLab's CI service allows different machines to
be set up with GitLab's "agent", called gitlab-runner, which will take
care of running jobs created by events such as a push to a branch.
Here, the combination of a machine, properly configured with GitLab's
gitlab-runner, is called a "custom runner".
The GitLab CI jobs definition for the custom runners are located under::
.gitlab-ci.d/custom-runners.yml
Custom runners entail custom machines. To see a list of the machines
currently deployed in the QEMU GitLab CI and their maintainers, please
refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
Machine Setup Howto
-------------------
For all Linux based systems, the setup can be mostly automated by the
execution of two Ansible playbooks. Create an ``inventory`` file
under ``scripts/ci/setup``, such as this::
fully.qualified.domain
other.machine.hostname
You may need to set some variables in the inventory file itself. One
very common need is to tell Ansible to use a Python 3 interpreter on
those hosts. This would look like::
fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
other.machine.hostname ansible_python_interpreter=/usr/bin/python3
Build environment
~~~~~~~~~~~~~~~~~
The ``scripts/ci/setup/$DISTRO/build-environment.yml`` Ansible
playbook will set up machines with the environment needed to perform
builds and run QEMU tests. This playbook consists on the installation
of various required packages (and a general package update while at
it).
The minimum required version of Ansible successfully tested in this
playbook is 2.8.0 (a version check is embedded within the playbook
itself). To run the playbook, execute::
cd scripts/ci/setup
ansible-playbook -i inventory $DISTRO/build-environment.yml
Please note that most of the tasks in the playbook require superuser
privileges, such as those from the ``root`` account or those obtained
by ``sudo``. If necessary, please refer to ``ansible-playbook``
options such as ``--become``, ``--become-method``, ``--become-user``
and ``--ask-become-pass``.
gitlab-runner setup and registration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The gitlab-runner agent needs to be installed on each machine that
will run jobs. The association between a machine and a GitLab project
happens with a registration token. To find the registration token for
your repository/project, navigate on GitLab's web UI to:
* Settings (the gears-like icon at the bottom of the left hand side
vertical toolbar), then
* CI/CD, then
* Runners, and click on the "Expand" button, then
* Under "Set up a specific Runner manually", look for the value under
"And this registration token:"
Copy the ``scripts/ci/setup/vars.yml.template`` file to
``scripts/ci/setup/vars.yml``. Then, set the
``gitlab_runner_registration_token`` variable to the value obtained
earlier.
To run the playbook, execute::
cd scripts/ci/setup
ansible-playbook -i inventory gitlab-runner.yml
Following the registration, it's necessary to configure the runner tags,
and optionally other configurations on the GitLab UI. Navigate to:
* Settings (the gears like icon), then
* CI/CD, then
* Runners, and click on the "Expand" button, then
* "Runners activated for this project", then
* Click on the "Edit" icon (next to the "Lock" Icon)
Tags are very important as they are used to route specific jobs to
specific types of runners, so it's a good idea to double check that
the automatically created tags are consistent with the OS and
architecture. For instance, an Ubuntu 20.04 aarch64 system should
have tags set as::
ubuntu_20.04,aarch64
Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
would contain::
ubuntu-20.04-aarch64-all:
tags:
- ubuntu_20.04
- aarch64
It's also recommended to:
* increase the "Maximum job timeout" to something like ``2h``
* give it a better Description

14
docs/devel/testing/ci.rst Normal file
View file

@ -0,0 +1,14 @@
.. _ci:
==
CI
==
Most of QEMU's CI is run on GitLab's infrastructure although a number
of other CI services are used for specialised purposes. The most up to
date information about them and their status can be found on the
`project wiki testing page <https://wiki.qemu.org/Testing/CI>`_.
.. include:: ci-definitions.rst.inc
.. include:: ci-jobs.rst.inc
.. include:: ci-runners.rst.inc

383
docs/devel/testing/functional.rst Normal file
View file

@ -0,0 +1,383 @@
.. _checkfunctional-ref:
Functional testing with Python
==============================
The ``tests/functional`` directory hosts functional tests written in
Python. They are usually higher level tests, and may interact with
external resources and with various guest operating systems.
The functional tests have initially evolved from the Avocado tests, so there
is a lot of similarity to those tests here (see :ref:`checkavocado-ref` for
details about the Avocado tests).
The tests should be written in the style of the Python `unittest`_ framework,
using stdio for the TAP protocol. The folder ``tests/functional/qemu_test``
provides classes (e.g. the ``QemuBaseTest``, ``QemuUserTest`` and the
``QemuSystemTest`` classes) and utility functions that help to get your test
into the right shape, e.g. by replacing the 'stdout' python object to redirect
the normal output of your test to stderr instead.
Note that if you don't use one of the QemuBaseTest based classes for your
test, or if you spawn subprocesses from your test, you have to make sure
that there is no TAP-incompatible output written to stdio, e.g. either by
prefixing every line with a "# " to mark the output as a TAP comment, or
e.g. by capturing the stdout output of subprocesses (redirecting it to
stderr is OK).
Tests based on ``qemu_test.QemuSystemTest`` can easily:
* Customize the command line arguments given to the convenience
``self.vm`` attribute (a QEMUMachine instance)
* Interact with the QEMU monitor, send QMP commands and check
their results
* Interact with the guest OS, using the convenience console device
(which may be useful to assert the effectiveness and correctness of
command line arguments or QMP commands)
* Download (and cache) remote data files, such as firmware and kernel
images
Running tests
-------------
You can run the functional tests simply by executing:
.. code::
make check-functional
It is also possible to run tests for a certain target only, for example
the following line will only run the tests for the x86_64 target:
.. code::
make check-functional-x86_64
To run a single test file without the meson test runner, you can also
execute the file directly by specifying two environment variables first,
the PYTHONPATH that has to include the python folder and the tests/functional
folder of the source tree, and QEMU_TEST_QEMU_BINARY that has to point
to the QEMU binary that should be used for the test. The current working
directory should be your build folder. For example::
$ export PYTHONPATH=../python:../tests/functional
$ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
$ pyvenv/bin/python3 ../tests/functional/test_file.py
The test framework will automatically purge any scratch files created during
the tests. If needing to debug a failed test, it is possible to keep these
files around on disk by setting ```QEMU_TEST_KEEP_SCRATCH=1``` as an env
variable. Any preserved files will be deleted the next time the test is run
without this variable set.
Logging
-------
The framework collects log files for each test in the build directory
in the following subfolder::
<builddir>/tests/functional/<arch>/<fileid>.<classid>.<testname>/
There are usually three log files:
* ``base.log`` contains the generic logging information that is written
by the calls to the logging functions in the test code (e.g. by calling
the ``self.log.info()`` or ``self.log.debug()`` functions).
* ``console.log`` contains the output of the serial console of the guest.
* ``default.log`` contains the output of QEMU. This file could be named
differently if the test chooses to use a different identifier for
the guest VM (e.g. when the test spins up multiple VMs).
Introduction to writing tests
-----------------------------
The ``tests/functional/qemu_test`` directory provides the ``qemu_test``
Python module, containing the ``qemu_test.QemuSystemTest`` class.
Here is a simple usage example:
.. code::
#!/usr/bin/env python3
from qemu_test import QemuSystemTest
class Version(QemuSystemTest):
def test_qmp_human_info_version(self):
self.vm.launch()
res = self.vm.cmd('human-monitor-command',
command_line='info version')
self.assertRegex(res, r'^(\d+\.\d+\.\d)')
if __name__ == '__main__':
QemuSystemTest.main()
By providing the "hash bang" line at the beginning of the script, marking
the file as executable and by calling into QemuSystemTest.main(), the test
can also be run stand-alone, without a test runner. OTOH when run via a test
runner, the QemuSystemTest.main() function takes care of running the test
functions in the right fassion (e.g. with TAP output that is required by the
meson test runner).
The ``qemu_test.QemuSystemTest`` base test class
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``qemu_test.QemuSystemTest`` class has a number of characteristics
that are worth being mentioned.
First of all, it attempts to give each test a ready to use QEMUMachine
instance, available at ``self.vm``. Because many tests will tweak the
QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
is left to the test writer.
The base test class has also support for tests with more than one
QEMUMachine. The way to get machines is through the ``self.get_vm()``
method which will return a QEMUMachine instance. The ``self.get_vm()``
method accepts arguments that will be passed to the QEMUMachine creation
and also an optional ``name`` attribute so you can identify a specific
machine and get it more than once through the tests methods. A simple
and hypothetical example follows:
.. code::
from qemu_test import QemuSystemTest
class MultipleMachines(QemuSystemTest):
def test_multiple_machines(self):
first_machine = self.get_vm()
second_machine = self.get_vm()
self.get_vm(name='third_machine').launch()
first_machine.launch()
second_machine.launch()
first_res = first_machine.cmd(
'human-monitor-command',
command_line='info version')
second_res = second_machine.cmd(
'human-monitor-command',
command_line='info version')
third_res = self.get_vm(name='third_machine').cmd(
'human-monitor-command',
command_line='info version')
self.assertEqual(first_res, second_res, third_res)
At test "tear down", ``qemu_test.QemuSystemTest`` handles all the QEMUMachines
shutdown.
QEMUMachine
-----------
The QEMUMachine API is already widely used in the Python iotests,
device-crash-test and other Python scripts. It's a wrapper around the
execution of a QEMU binary, giving its users:
* the ability to set command line arguments to be given to the QEMU
binary
* a ready to use QMP connection and interface, which can be used to
send commands and inspect its results, as well as asynchronous
events
* convenience methods to set commonly used command line arguments in
a more succinct and intuitive way
QEMU binary selection
^^^^^^^^^^^^^^^^^^^^^
The QEMU binary used for the ``self.vm`` QEMUMachine instance will
primarily depend on the value of the ``qemu_bin`` instance attribute.
If it is not explicitly set by the test code, its default value will
be the result the QEMU_TEST_QEMU_BINARY environment variable.
Debugging hung QEMU
^^^^^^^^^^^^^^^^^^^
When test cases go wrong it may be helpful to debug a stalled QEMU
process. While the QEMUMachine class owns the primary QMP monitor
socket, it is possible to request a second QMP monitor be created
by setting the ``QEMU_TEST_QMP_BACKDOOR`` env variable to refer
to a UNIX socket name. The ``qmp-shell`` command can then be
attached to the stalled QEMU to examine its live state.
Attribute reference
-------------------
QemuBaseTest
^^^^^^^^^^^^
The following attributes are available on any ``qemu_test.QemuBaseTest``
instance.
arch
""""
The target architecture of the QEMU binary.
Tests are also free to use this attribute value, for their own needs.
A test may, for instance, use this value when selecting the architecture
of a kernel or disk image to boot a VM with.
qemu_bin
""""""""
The preserved value of the ``QEMU_TEST_QEMU_BINARY`` environment
variable.
QemuUserTest
^^^^^^^^^^^^
The QemuUserTest class can be used for running an executable via the
usermode emulation binaries.
QemuSystemTest
^^^^^^^^^^^^^^
The QemuSystemTest class can be used for running tests via one of the
qemu-system-* binaries.
vm
""
A QEMUMachine instance, initially configured according to the given
``qemu_bin`` parameter.
cpu
"""
The cpu model that will be set to all QEMUMachine instances created
by the test.
machine
"""""""
The machine type that will be set to all QEMUMachine instances created
by the test. By using the set_machine() function of the QemuSystemTest
class to set this attribute, you can automatically check whether the
machine is available to skip the test in case it is not built into the
QEMU binary.
Asset handling
--------------
Many functional tests download assets (e.g. Linux kernels, initrds,
firmware images, etc.) from the internet to be able to run tests with
them. This imposes additional challenges to the test framework.
First there is the problem that some people might not have an
unconstrained internet connection, so such tests should not be run by
default when running ``make check``. To accomplish this situation,
the tests that download files should only be added to the "thorough"
speed mode in the meson.build file, while the "quick" speed mode is
fine for functional tests that can be run without downloading files.
``make check`` then only runs the quick functional tests along with
the other quick tests from the other test suites. If you choose to
run only run ``make check-functional``, the "thorough" tests will be
executed, too. And to run all functional tests along with the others,
you can use something like::
make -j$(nproc) check SPEED=thorough
The second problem with downloading files from the internet are time
constraints. The time for downloading files should not be taken into
account when the test is running and the timeout of the test is ticking
(since downloading can be very slow, depending on the network bandwidth).
This problem is solved by downloading the assets ahead of time, before
the tests are run. This pre-caching is done with the qemu_test.Asset
class. To use it in your test, declare an asset in your test class with
its URL and SHA256 checksum like this::
from qemu_test import Asset
ASSET_somename = Asset(
('https://www.qemu.org/assets/images/qemu_head_200.png'),
'34b74cad46ea28a2966c1d04e102510daf1fd73e6582b6b74523940d5da029dd')
In your test function, you can then get the file name of the cached
asset like this::
def test_function(self):
file_path = self.ASSET_somename.fetch()
The pre-caching will be done automatically when running
``make check-functional`` (but not when running e.g.
``make check-functional-<target>``). In case you just want to download
the assets without running the tests, you can do so by running::
make precache-functional
The cache is populated in the ``~/.cache/qemu/download`` directory by
default, but the location can be changed by setting the
``QEMU_TEST_CACHE_DIR`` environment variable.
Skipping tests
--------------
Since the test framework is based on the common Python unittest framework,
you can use the usual Python decorators which allow for easily skipping
tests running under certain conditions, for example, on the lack of a binary
on the test system or when the running environment is a CI system. For further
information about those decorators, please refer to:
https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures
While the conditions for skipping tests are often specifics of each one, there
are recurring scenarios identified by the QEMU developers and the use of
environment variables became a kind of standard way to enable/disable tests.
Here is a list of the most used variables:
QEMU_TEST_ALLOW_LARGE_STORAGE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tests which are going to fetch or produce assets considered *large* are not
going to run unless that ``QEMU_TEST_ALLOW_LARGE_STORAGE=1`` is exported on
the environment.
The definition of *large* is a bit arbitrary here, but it usually means an
asset which occupies at least 1GB of size on disk when uncompressed.
QEMU_TEST_ALLOW_UNTRUSTED_CODE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are tests which will boot a kernel image or firmware that can be
considered not safe to run on the developer's workstation, thus they are
skipped by default. The definition of *not safe* is also arbitrary but
usually it means a blob which either its source or build process aren't
public available.
You should export ``QEMU_TEST_ALLOW_UNTRUSTED_CODE=1`` on the environment in
order to allow tests which make use of those kind of assets.
QEMU_TEST_FLAKY_TESTS
^^^^^^^^^^^^^^^^^^^^^
Some tests are not working reliably and thus are disabled by default.
This includes tests that don't run reliably on GitLab's CI which
usually expose real issues that are rarely seen on developer machines
due to the constraints of the CI environment. If you encounter a
similar situation then raise a bug and then mark the test as shown on
the code snippet below:
.. code::
# See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test(self):
do_something()
Tests should not live in this state forever and should either be fixed
or eventually removed.
QEMU_TEST_ALLOW_SLOW
^^^^^^^^^^^^^^^^^^^^
Tests that have a very long runtime and might run into timeout issues
e.g. if the QEMU binary has been compiled with debugging options enabled.
To avoid these timeout issues by default and to save some precious CPU
cycles during normal testing, such tests are disabled by default unless
the QEMU_TEST_ALLOW_SLOW environment variable has been set.
.. _unittest: https://docs.python.org/3/library/unittest.html

305
docs/devel/testing/fuzzing.rst Normal file
View file

@ -0,0 +1,305 @@
========
Fuzzing
========
This document describes the virtual-device fuzzing infrastructure in QEMU and
how to use it to implement additional fuzzers.
Basics
------
Fuzzing operates by passing inputs to an entry point/target function. The
fuzzer tracks the code coverage triggered by the input. Based on these
findings, the fuzzer mutates the input and repeats the fuzzing.
To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer
is an *in-process* fuzzer. For the developer, this means that it is their
responsibility to ensure that state is reset between fuzzing-runs.
Building the fuzzers
--------------------
To build the fuzzers, install a recent version of clang:
Configure with (substitute the clang binaries with the version you installed).
Here, enable-asan and enable-ubsan are optional but they allow us to reliably
detect bugs such as out-of-bounds accesses, uses-after-free, double-frees
etc.::
CC=clang-8 CXX=clang++-8 /path/to/configure \
--enable-fuzzing --enable-asan --enable-ubsan
Fuzz targets are built similarly to system targets::
make qemu-fuzz-i386
This builds ``./qemu-fuzz-i386``
The first option to this command is: ``--fuzz-target=FUZZ_NAME``
To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments.
For example::
./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz
Internally, libfuzzer parses all arguments that do not begin with ``"--"``.
Information about these is available by passing ``-help=1``
Now the only thing left to do is wait for the fuzzer to trigger potential
crashes.
Useful libFuzzer flags
----------------------
As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will
list the available arguments. In particular, these arguments might be helpful:
* ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer.
libFuzzer stores each "interesting" input in this corpus directory. The next
time you run libFuzzer, it will read all of the inputs from the corpus, and
continue fuzzing from there. You can also specify multiple directories.
libFuzzer loads existing inputs from all specified directories, but will only
write new ones to the first one specified.
* ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer
will generate.
* ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that
trigger many debug/error messages, or create output on the serial console.
* ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in
parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only
``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal
to half the available CPU cores. Replace 4 with a number appropriate for your
machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel
fuzzers to share information about the interesting inputs they find.
* ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes
``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the
coverage table. Useful for targets with "magic" constants. If Arg1 came from
the fuzzer's input and Arg2 is a magic constant, then each time the Hamming
distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the
corpus.
* ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to
better coverage performance, depending on the target.
Note that libFuzzer's exact behavior will depend on the version of
clang and libFuzzer used to build the device fuzzers.
Generating Coverage Reports
---------------------------
Code coverage is a crucial metric for evaluating a fuzzer's performance.
libFuzzer's output provides a "cov: " column that provides a total number of
unique blocks/edges covered. To examine coverage on a line-by-line basis we
can use Clang coverage:
1. Configure libFuzzer to store a corpus of all interesting inputs (see
CORPUS_DIR above)
2. ``./configure`` the QEMU build with ::
--enable-fuzzing \
--extra-cflags="-fprofile-instr-generate -fcoverage-mapping"
3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer
to execute all of the inputs in $CORPUS_DIR and exit. Once the process
exits, you should find a file, "default.profraw" in the working directory.
4. Execute these commands to generate a detailed HTML coverage-report::
llvm-profdata merge -output=default.profdata default.profraw
llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \
--format html -output-dir=/path/to/output/report
Adding a new fuzzer
-------------------
Coverage over virtual devices can be improved by adding additional fuzzers.
Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to
``tests/qtest/fuzz/meson.build``
Fuzzers can rely on both qtest and libqos to communicate with virtual devices.
1. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``.
2. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers
for reference.
3. Add the fuzzer to ``tests/qtest/fuzz/meson.build``.
Fuzzers can be more-or-less thought of as special qtest programs which can
modify the qtest commands and/or qtest command arguments based on inputs
provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the
fuzzer loops over the byte-array interpreting it as a list of qtest commands,
addresses, or values.
The Generic Fuzzer
------------------
Writing a fuzz target can be a lot of effort (especially if a device driver has
not be built-out within libqos). Many devices can be fuzzed to some degree,
without any device-specific code, using the generic-fuzz target.
The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO,
and DMA input-spaces. To apply the generic-fuzz to a device, we need to define
two env-variables, at minimum:
* ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with
the device attached. For example, if we want to fuzz the virtio-net device
attached to a pc-i440fx machine, we can specify::
QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \
-device virtio-net,netdev=user0"
* ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify
the MemoryRegions that will be fuzzed. These strings are compared against
MemoryRegion names and MemoryRegion owner names, to decide whether each
MemoryRegion should be fuzzed. These strings support globbing. For the
virtio-net example, we could use one of ::
QEMU_FUZZ_OBJECTS='virtio-net'
QEMU_FUZZ_OBJECTS='virtio*'
QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker
QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine``
The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially
useful for identifying the ``MemoryRegion`` and ``Object`` names used for
matching.
As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the
greater the input-space, and the smaller the probability of finding crashing
inputs for individual devices. As such, it is usually a good idea to limit the
fuzzer to only a few ``MemoryRegions``.
To ensure that these env variables have been configured correctly, we can use::
./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0
The output should contain a complete list of matched MemoryRegions.
OSS-Fuzz
--------
QEMU is continuously fuzzed on `OSS-Fuzz
<https://github.com/google/oss-fuzz>`_. By default, the OSS-Fuzz build
will try to fuzz every fuzz-target. Since the generic-fuzz target
requires additional information provided in environment variables, we
pre-define some generic-fuzz configs in
``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify:
- ``.name``: To identify the fuzzer config
- ``.args`` OR ``.argfunc``: A string or pointer to a function returning a
string. These strings are used to specify the ``QEMU_FUZZ_ARGS``
environment variable. ``argfunc`` is useful when the config relies on e.g.
a dynamically created temp directory, or a free tcp/udp port.
- ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment
variable.
To fuzz additional devices/device configuration on OSS-Fuzz, send patches for
either a new device-specific fuzzer or a new generic-fuzz config.
Build details:
- The Dockerfile that sets up the environment for building QEMU's
fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository
__(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile)
- The script responsible for building the fuzzers can be found in the
QEMU source tree at ``scripts/oss-fuzz/build.sh``
Building Crash Reproducers
-----------------------------------------
When we find a crash, we should try to create an independent reproducer, that
can be used on a non-fuzzer build of QEMU. This filters out any potential
false-positives, and improves the debugging experience for developers.
Here are the steps for building a reproducer for a crash found by the
generic-fuzz target.
- Ensure the crash reproduces::
qemu-fuzz-i386 --fuzz-target... ./crash-...
- Gather the QTest output for the crash::
QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \
qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace
- Reorder and clean-up the resulting trace::
scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer
- Get the arguments needed to start qemu, and provide a path to qemu::
less /tmp/trace # The args should be logged at the top of this file
export QEMU_ARGS="-machine ..."
export QEMU_PATH="path/to/qemu-system"
- Ensure the crash reproduces in qemu-system::
$QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer
- From the crash output, obtain some string that identifies the crash. This
can be a line in the stack-trace, for example::
export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865"
- Minimize the reproducer::
scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \
/tmp/reproducer /tmp/reproducer-minimized
- Confirm that the minimized reproducer still crashes::
$QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized
- Create a one-liner reproducer that can be sent over email::
./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized
- Output the C source code for a test case that will reproduce the bug::
./scripts/oss-fuzz/output_reproducer.py -owner "John Smith <john@smith.com>"\
-name "test_function_name" /tmp/reproducer-minimized
- Report the bug and send a patch with the C reproducer upstream
Implementation Details / Fuzzer Lifecycle
-----------------------------------------
The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's
own ``main()``, which performs some setup, and calls the entrypoints:
``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the
necessary state
``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and
resets the state at the end of each run.
In more detail:
``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two
dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments
select the fuzz target. Then, the qtest client is initialized. If the target
requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized.
Then the QGraph is walked and the QEMU cmd_line is determined and saved.
After this, the ``vl.c:main`` is called to set up the guest. There are
target-specific hooks that can be called before and after main, for
additional setup(e.g. PCI setup, or VM snapshotting).
``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz
input. It is also responsible for manually calling ``main_loop_wait`` to ensure
that bottom halves are executed and any cleanup required before the next input.
Since the same process is reused for many fuzzing runs, QEMU state needs to
be reset at the end of each run. For example, this can be done by rebooting the
VM, after each run.
- *Pros*: Straightforward and fast for simple fuzz targets.
- *Cons*: Depending on the device, does not reset all device state. If the
device requires some initialization prior to being ready for fuzzing (common
for QOS-based targets), this initialization needs to be done after each
reboot.
- *Example target*: ``i440fx-qtest-reboot-fuzz``

18
docs/devel/testing/index.rst Normal file
View file

@ -0,0 +1,18 @@
Testing QEMU
------------
Details about how to test QEMU and how it is integrated into our CI
testing infrastructure.
.. toctree::
:maxdepth: 3
main
qtest
functional
avocado
acpi-bits
ci
fuzzing
blkdebug
blkverify

1025
docs/devel/testing/main.rst Normal file
View file

File diff suppressed because it is too large Load diff

628
docs/devel/testing/qgraph.rst Normal file
View file

@ -0,0 +1,628 @@
.. _qgraph:
Qtest Driver Framework
======================
In order to test a specific driver, plain libqos tests need to
take care of booting QEMU with the right machine and devices.
This makes each test "hardcoded" for a specific configuration, reducing
the possible coverage that it can reach.
For example, the sdhci device is supported on both x86_64 and ARM boards,
therefore a generic sdhci test should test all machines and drivers that
support that device.
Using only libqos APIs, the test has to manually take care of
covering all the setups, and build the correct command line.
This also introduces backward compatibility issues: if a device/driver command
line name is changed, all tests that use that will not work
properly anymore and need to be adjusted.
The aim of qgraph is to create a graph of drivers, machines and tests such that
a test aimed to a certain driver does not have to care of
booting the right QEMU machine, pick the right device, build the command line
and so on. Instead, it only defines what type of device it is testing
(interface in qgraph terms) and the framework takes care of
covering all supported types of devices and machine architectures.
Following the above example, an interface would be ``sdhci``,
so the sdhci-test should only care of linking its qgraph node with
that interface. In this way, if the command line of a sdhci driver
is changed, only the respective qgraph driver node has to be adjusted.
QGraph concepts
---------------
The graph is composed by nodes that represent machines, drivers, tests
and edges that define the relationships between them (``CONSUMES``, ``PRODUCES``, and
``CONTAINS``).
Nodes
~~~~~
A node can be of four types:
- **QNODE_MACHINE**: for example ``arm/raspi2b``
- **QNODE_DRIVER**: for example ``generic-sdhci``
- **QNODE_INTERFACE**: for example ``sdhci`` (interface for all ``-sdhci``
drivers).
An interface is not explicitly created, it will be automatically
instantiated when a node consumes or produces it.
An interface is simply a struct that abstracts the various drivers
for the same type of device, and offers an API to the nodes that
use it ("consume" relation in qgraph terms) that is implemented/backed up by the drivers that implement it ("produce" relation in qgraph terms).
- **QNODE_TEST**: for example ``sdhci-test``. A test consumes an interface
and tests the functions provided by it.
Notes for the nodes:
- QNODE_MACHINE: each machine struct must have a ``QGuestAllocator`` and
implement ``get_driver()`` to return the allocator mapped to the interface
"memory". The function can also return ``NULL`` if the allocator
is not set.
- QNODE_DRIVER: driver names must be unique, and machines and nodes
planned to be "consumed" by other nodes must match QEMU
drivers name, otherwise they won't be discovered
Edges
~~~~~
An edge relation between two nodes (drivers or machines) ``X`` and ``Y`` can be:
- ``X CONSUMES Y``: ``Y`` can be plugged into ``X``
- ``X PRODUCES Y``: ``X`` provides the interface ``Y``
- ``X CONTAINS Y``: ``Y`` is part of ``X`` component
Execution steps
~~~~~~~~~~~~~~~
The basic framework steps are the following:
- All nodes and edges are created in their respective
machine/driver/test files
- The framework starts QEMU and asks for a list of available devices
and machines (note that only machines and "consumed" nodes are mapped
1:1 with QEMU devices)
- The framework walks the graph starting from the available machines and
performs a Depth First Search for tests
- Once a test is found, the path is walked again and all drivers are
allocated accordingly and the final interface is passed to the test
- The test is executed
- Unused objects are cleaned and the path discovery is continued
Depending on the QEMU binary used, only some drivers/machines will be
available and only test that are reached by them will be executed.
Command line
~~~~~~~~~~~~
Command line is built by using node names and optional arguments
passed by the user when building the edges.
There are three types of command line arguments:
- ``in node`` : created from the node name. For example, machines will
have ``-M <machine>`` to its command line, while devices
``-device <device>``. It is automatically done by the framework.
- ``after node`` : added as additional argument to the node name.
This argument is added optionally when creating edges,
by setting the parameter ``after_cmd_line`` and
``extra_edge_opts`` in ``QOSGraphEdgeOptions``.
The framework automatically adds
a comma before ``extra_edge_opts``,
because it is going to add attributes
after the destination node pointed by
the edge containing these options, and automatically
adds a space before ``after_cmd_line``, because it
adds an additional device, not an attribute.
- ``before node`` : added as additional argument to the node name.
This argument is added optionally when creating edges,
by setting the parameter ``before_cmd_line`` in
``QOSGraphEdgeOptions``. This attribute
is going to add attributes before the destination node
pointed by the edge containing these options. It is
helpful to commands that are not node-representable,
such as ``-fdsev`` or ``-netdev``.
While adding command line in edges is always used, not all nodes names are
used in every path walk: this is because the contained or produced ones
are already added by QEMU, so only nodes that "consumes" will be used to
build the command line. Also, nodes that will have ``{ "abstract" : true }``
as QMP attribute will loose their command line, since they are not proper
devices to be added in QEMU.
Example::
QOSGraphEdgeOptions opts = {
.before_cmd_line = "-drive id=drv0,if=none,file=null-co://,"
"file.read-zeroes=on,format=raw",
.after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0",
opts.extra_device_opts = "id=vs0";
};
qos_node_create_driver("virtio-scsi-device",
virtio_scsi_device_create);
qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts);
Will produce the following command line:
``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0``
Troubleshooting unavailable tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If there is no path from an available machine to a test then that test will be
unavailable and won't execute. This can happen if a test or driver did not set
up its qgraph node correctly. It can also happen if the necessary machine type
or device is missing from the QEMU binary because it was compiled out or
otherwise.
It is possible to troubleshoot unavailable tests by running::
$ QTEST_QEMU_BINARY=build/qemu-system-x86_64 build/tests/qtest/qos-test --verbose
# ALL QGRAPH EDGES: {
# src='virtio-net'
# |-> dest='virtio-net-tests/vhost-user/multiqueue' type=2 (node=0x559142109e30)
# |-> dest='virtio-net-tests/vhost-user/migrate' type=2 (node=0x559142109d00)
# src='virtio-net-pci'
# |-> dest='virtio-net' type=1 (node=0x55914210d740)
# src='pci-bus'
# |-> dest='virtio-net-pci' type=2 (node=0x55914210d880)
# src='pci-bus-pc'
# |-> dest='pci-bus' type=1 (node=0x559142103f40)
# src='i440FX-pcihost'
# |-> dest='pci-bus-pc' type=0 (node=0x55914210ac70)
# src='x86_64/pc'
# |-> dest='i440FX-pcihost' type=0 (node=0x5591421117f0)
# src=''
# |-> dest='x86_64/pc' type=0 (node=0x559142111600)
# |-> dest='arm/raspi2b' type=0 (node=0x559142110740)
...
# }
# ALL QGRAPH NODES: {
# name='virtio-net-tests/announce-self' type=3 cmd_line='(null)' [available]
# name='arm/raspi2b' type=0 cmd_line='-M raspi2b ' [UNAVAILABLE]
...
# }
The ``virtio-net-tests/announce-self`` test is listed as "available" in the
"ALL QGRAPH NODES" output. This means the test will execute. We can follow the
qgraph path in the "ALL QGRAPH EDGES" output as follows: '' -> 'x86_64/pc' ->
'i440FX-pcihost' -> 'pci-bus-pc' -> 'pci-bus' -> 'virtio-net-pci' ->
'virtio-net'. The root of the qgraph is '' and the depth first search begins
there.
The ``arm/raspi2b`` machine node is listed as "UNAVAILABLE". Although it is
reachable from the root via '' -> 'arm/raspi2b' the node is unavailable because
the QEMU binary did not list it when queried by the framework. This is expected
because we used the ``qemu-system-x86_64`` binary which does not support ARM
machine types.
If a test is unexpectedly listed as "UNAVAILABLE", first check that the "ALL
QGRAPH EDGES" output reports edge connectivity from the root ('') to the test.
If there is no connectivity then the qgraph nodes were not set up correctly and
the driver or test code is incorrect. If there is connectivity, check the
availability of each node in the path in the "ALL QGRAPH NODES" output. The
first unavailable node in the path is the reason why the test is unavailable.
Typically this is because the QEMU binary lacks support for the necessary
machine type or device.
Creating a new driver and its interface
---------------------------------------
Here we continue the ``sdhci`` use case, with the following scenario:
- ``sdhci-test`` aims to test the ``read[q,w], writeq`` functions
offered by the ``sdhci`` drivers.
- The current ``sdhci`` device is supported by both ``x86_64/pc`` and ``ARM``
(in this example we focus on the ``arm-raspi2b``) machines.
- QEMU offers 2 types of drivers: ``QSDHCI_MemoryMapped`` for ``ARM`` and
``QSDHCI_PCI`` for ``x86_64/pc``. Both implement the
``read[q,w], writeq`` functions.
In order to implement such scenario in qgraph, the test developer needs to:
- Create the ``x86_64/pc`` machine node. This machine uses the
``pci-bus`` architecture so it ``contains`` a PCI driver,
``pci-bus-pc``. The actual path is
``x86_64/pc --contains--> 1440FX-pcihost --contains-->
pci-bus-pc --produces--> pci-bus``.
For the sake of this example,
we do not focus on the PCI interface implementation.
- Create the ``sdhci-pci`` driver node, representing ``QSDHCI_PCI``.
The driver uses the PCI bus (and its API),
so it must ``consume`` the ``pci-bus`` generic interface (which abstracts
all the pci drivers available)
``sdhci-pci --consumes--> pci-bus``
- Create an ``arm/raspi2b`` machine node. This machine ``contains``
a ``generic-sdhci`` memory mapped ``sdhci`` driver node, representing
``QSDHCI_MemoryMapped``.
``arm/raspi2b --contains--> generic-sdhci``
- Create the ``sdhci`` interface node. This interface offers the
functions that are shared by all ``sdhci`` devices.
The interface is produced by ``sdhci-pci`` and ``generic-sdhci``,
the available architecture-specific drivers.
``sdhci-pci --produces--> sdhci``
``generic-sdhci --produces--> sdhci``
- Create the ``sdhci-test`` test node. The test ``consumes`` the
``sdhci`` interface, using its API. It doesn't need to look at
the supported machines or drivers.
``sdhci-test --consumes--> sdhci``
``arm-raspi2b`` machine, simplified from
``tests/qtest/libqos/arm-raspi2-machine.c``::
#include "qgraph.h"
struct QRaspi2Machine {
QOSGraphObject obj;
QGuestAllocator alloc;
QSDHCI_MemoryMapped sdhci;
};
static void *raspi2_get_driver(void *object, const char *interface)
{
QRaspi2Machine *machine = object;
if (!g_strcmp0(interface, "memory")) {
return &machine->alloc;
}
fprintf(stderr, "%s not present in arm/raspi2b\n", interface);
g_assert_not_reached();
}
static QOSGraphObject *raspi2_get_device(void *obj,
const char *device)
{
QRaspi2Machine *machine = obj;
if (!g_strcmp0(device, "generic-sdhci")) {
return &machine->sdhci.obj;
}
fprintf(stderr, "%s not present in arm/raspi2b\n", device);
g_assert_not_reached();
}
static void *qos_create_machine_arm_raspi2(QTestState *qts)
{
QRaspi2Machine *machine = g_new0(QRaspi2Machine, 1);
alloc_init(&machine->alloc, ...);
/* Get node(s) contained inside (CONTAINS) */
machine->obj.get_device = raspi2_get_device;
/* Get node(s) produced (PRODUCES) */
machine->obj.get_driver = raspi2_get_driver;
/* free the object */
machine->obj.destructor = raspi2_destructor;
qos_init_sdhci_mm(&machine->sdhci, ...);
return &machine->obj;
}
static void raspi2_register_nodes(void)
{
/* arm/raspi2b --contains--> generic-sdhci */
qos_node_create_machine("arm/raspi2b",
qos_create_machine_arm_raspi2);
qos_node_contains("arm/raspi2b", "generic-sdhci", NULL);
}
libqos_init(raspi2_register_nodes);
``x86_64/pc`` machine, simplified from
``tests/qtest/libqos/x86_64_pc-machine.c``::
#include "qgraph.h"
struct i440FX_pcihost {
QOSGraphObject obj;
QPCIBusPC pci;
};
struct QX86PCMachine {
QOSGraphObject obj;
QGuestAllocator alloc;
i440FX_pcihost bridge;
};
/* i440FX_pcihost */
static QOSGraphObject *i440FX_host_get_device(void *obj,
const char *device)
{
i440FX_pcihost *host = obj;
if (!g_strcmp0(device, "pci-bus-pc")) {
return &host->pci.obj;
}
fprintf(stderr, "%s not present in i440FX-pcihost\n", device);
g_assert_not_reached();
}
/* x86_64/pc machine */
static void *pc_get_driver(void *object, const char *interface)
{
QX86PCMachine *machine = object;
if (!g_strcmp0(interface, "memory")) {
return &machine->alloc;
}
fprintf(stderr, "%s not present in x86_64/pc\n", interface);
g_assert_not_reached();
}
static QOSGraphObject *pc_get_device(void *obj, const char *device)
{
QX86PCMachine *machine = obj;
if (!g_strcmp0(device, "i440FX-pcihost")) {
return &machine->bridge.obj;
}
fprintf(stderr, "%s not present in x86_64/pc\n", device);
g_assert_not_reached();
}
static void *qos_create_machine_pc(QTestState *qts)
{
QX86PCMachine *machine = g_new0(QX86PCMachine, 1);
/* Get node(s) contained inside (CONTAINS) */
machine->obj.get_device = pc_get_device;
/* Get node(s) produced (PRODUCES) */
machine->obj.get_driver = pc_get_driver;
/* free the object */
machine->obj.destructor = pc_destructor;
pc_alloc_init(&machine->alloc, qts, ALLOC_NO_FLAGS);
/* Get node(s) contained inside (CONTAINS) */
machine->bridge.obj.get_device = i440FX_host_get_device;
return &machine->obj;
}
static void pc_machine_register_nodes(void)
{
/* x86_64/pc --contains--> 1440FX-pcihost --contains-->
* pci-bus-pc [--produces--> pci-bus (in pci.h)] */
qos_node_create_machine("x86_64/pc", qos_create_machine_pc);
qos_node_contains("x86_64/pc", "i440FX-pcihost", NULL);
/* contained drivers don't need a constructor,
* they will be init by the parent */
qos_node_create_driver("i440FX-pcihost", NULL);
qos_node_contains("i440FX-pcihost", "pci-bus-pc", NULL);
}
libqos_init(pc_machine_register_nodes);
``sdhci`` taken from ``tests/qtest/libqos/sdhci.c``::
/* Interface node, offers the sdhci API */
struct QSDHCI {
uint16_t (*readw)(QSDHCI *s, uint32_t reg);
uint64_t (*readq)(QSDHCI *s, uint32_t reg);
void (*writeq)(QSDHCI *s, uint32_t reg, uint64_t val);
/* other fields */
};
/* Memory Mapped implementation of QSDHCI */
struct QSDHCI_MemoryMapped {
QOSGraphObject obj;
QSDHCI sdhci;
/* other driver-specific fields */
};
/* PCI implementation of QSDHCI */
struct QSDHCI_PCI {
QOSGraphObject obj;
QSDHCI sdhci;
/* other driver-specific fields */
};
/* Memory mapped implementation of QSDHCI */
static void *sdhci_mm_get_driver(void *obj, const char *interface)
{
QSDHCI_MemoryMapped *smm = obj;
if (!g_strcmp0(interface, "sdhci")) {
return &smm->sdhci;
}
fprintf(stderr, "%s not present in generic-sdhci\n", interface);
g_assert_not_reached();
}
void qos_init_sdhci_mm(QSDHCI_MemoryMapped *sdhci, QTestState *qts,
uint32_t addr, QSDHCIProperties *common)
{
/* Get node contained inside (CONTAINS) */
sdhci->obj.get_driver = sdhci_mm_get_driver;
/* SDHCI interface API */
sdhci->sdhci.readw = sdhci_mm_readw;
sdhci->sdhci.readq = sdhci_mm_readq;
sdhci->sdhci.writeq = sdhci_mm_writeq;
sdhci->qts = qts;
}
/* PCI implementation of QSDHCI */
static void *sdhci_pci_get_driver(void *object,
const char *interface)
{
QSDHCI_PCI *spci = object;
if (!g_strcmp0(interface, "sdhci")) {
return &spci->sdhci;
}
fprintf(stderr, "%s not present in sdhci-pci\n", interface);
g_assert_not_reached();
}
static void *sdhci_pci_create(void *pci_bus,
QGuestAllocator *alloc,
void *addr)
{
QSDHCI_PCI *spci = g_new0(QSDHCI_PCI, 1);
QPCIBus *bus = pci_bus;
uint64_t barsize;
qpci_device_init(&spci->dev, bus, addr);
/* SDHCI interface API */
spci->sdhci.readw = sdhci_pci_readw;
spci->sdhci.readq = sdhci_pci_readq;
spci->sdhci.writeq = sdhci_pci_writeq;
/* Get node(s) produced (PRODUCES) */
spci->obj.get_driver = sdhci_pci_get_driver;
spci->obj.start_hw = sdhci_pci_start_hw;
spci->obj.destructor = sdhci_destructor;
return &spci->obj;
}
static void qsdhci_register_nodes(void)
{
QOSGraphEdgeOptions opts = {
.extra_device_opts = "addr=04.0",
};
/* generic-sdhci */
/* generic-sdhci --produces--> sdhci */
qos_node_create_driver("generic-sdhci", NULL);
qos_node_produces("generic-sdhci", "sdhci");
/* sdhci-pci */
/* sdhci-pci --produces--> sdhci
* sdhci-pci --consumes--> pci-bus */
qos_node_create_driver("sdhci-pci", sdhci_pci_create);
qos_node_produces("sdhci-pci", "sdhci");
qos_node_consumes("sdhci-pci", "pci-bus", &opts);
}
libqos_init(qsdhci_register_nodes);
In the above example, all possible types of relations are created::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci --consumes--> pci-bus <--produces--+
|
+--produces--+
|
v
sdhci
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
or inverting the consumes edge in consumed_by::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci <--consumed by-- pci-bus <--produces--+
|
+--produces--+
|
v
sdhci
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
Adding a new test
-----------------
Given the above setup, adding a new test is very simple.
``sdhci-test``, taken from ``tests/qtest/sdhci-test.c``::
static void check_capab_sdma(QSDHCI *s, bool supported)
{
uint64_t capab, capab_sdma;
capab = s->readq(s, SDHC_CAPAB);
capab_sdma = FIELD_EX64(capab, SDHC_CAPAB, SDMA);
g_assert_cmpuint(capab_sdma, ==, supported);
}
static void test_registers(void *obj, void *data,
QGuestAllocator *alloc)
{
QSDHCI *s = obj;
/* example test */
check_capab_sdma(s, s->props.capab.sdma);
}
static void register_sdhci_test(void)
{
/* sdhci-test --consumes--> sdhci */
qos_add_test("registers", "sdhci", test_registers, NULL);
}
libqos_init(register_sdhci_test);
Here a new test is created, consuming ``sdhci`` interface node
and creating a valid path from both machines to a test.
Final graph will be like this::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci --consumes--> pci-bus <--produces--+
|
+--produces--+
|
v
sdhci <--consumes-- sdhci-test
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
or inverting the consumes edge in consumed_by::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci <--consumed by-- pci-bus <--produces--+
|
+--produces--+
|
v
sdhci --consumed by--> sdhci-test
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
Assuming there the binary is
``QTEST_QEMU_BINARY=./qemu-system-x86_64``
a valid test path will be:
``/x86_64/pc/1440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pc/sdhci/sdhci-test``
and for the binary ``QTEST_QEMU_BINARY=./qemu-system-arm``:
``/arm/raspi2b/generic-sdhci/sdhci/sdhci-test``
Additional examples are also in ``test-qgraph.c``
Qgraph API reference
--------------------
.. kernel-doc:: tests/qtest/libqos/qgraph.h

93
docs/devel/testing/qtest.rst Normal file
View file

@ -0,0 +1,93 @@
.. _qtest:
========================================
QTest Device Emulation Testing Framework
========================================
.. toctree::
qgraph
QTest is a device emulation testing framework. It can be very useful to test
device models; it could also control certain aspects of QEMU (such as virtual
clock stepping), with a special purpose "qtest" protocol. Refer to
:ref:`qtest-protocol` for more details of the protocol.
QTest cases can be executed with
.. code::
make check-qtest
The QTest library is implemented by ``tests/qtest/libqtest.c`` and the API is
defined in ``tests/qtest/libqtest.h``.
Consider adding a new QTest case when you are introducing a new virtual
hardware, or extending one if you are adding functionalities to an existing
virtual device.
On top of libqtest, a higher level library, ``libqos``, was created to
encapsulate common tasks of device drivers, such as memory management and
communicating with system buses or devices. Many virtual device tests use
libqos instead of directly calling into libqtest.
Libqos also offers the Qgraph API to increase each test coverage and
automate QEMU command line arguments and devices setup.
Refer to :ref:`qgraph` for Qgraph explanation and API.
Steps to add a new QTest case are:
1. Create a new source file for the test. (More than one file can be added as
necessary.) For example, ``tests/qtest/foo-test.c``.
2. Write the test code with the glib and libqtest/libqos API. See also existing
tests and the library headers for reference.
3. Register the new test in ``tests/qtest/meson.build``. Add the test
executable name to an appropriate ``qtests_*`` variable. There is
one variable per architecture, plus ``qtests_generic`` for tests
that can be run for all architectures. For example::
qtests_generic = [
...
'foo-test',
...
]
4. If the test has more than one source file or needs to be linked with any
dependency other than ``qemuutil`` and ``qos``, list them in the ``qtests``
dictionary. For example a test that needs to use the ``QIO`` library
will have an entry like::
{
...
'foo-test': [io],
...
}
Debugging a QTest failure is slightly harder than the unit test because the
tests look up QEMU program names in the environment variables, such as
``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy
to attach gdb to the QEMU process spawned from the test. But manual invoking
and using gdb on the test is still simple to do: find out the actual command
from the output of
.. code::
make check-qtest V=1
which you can run manually.
.. _qtest-protocol:
QTest Protocol
--------------
.. kernel-doc:: system/qtest.c
:doc: QTest Protocol
libqtest API reference
----------------------
.. kernel-doc:: tests/qtest/libqtest.h