diff options
Diffstat (limited to 'test/README.testsuite')
| -rw-r--r-- | test/README.testsuite | 310 |
1 files changed, 310 insertions, 0 deletions
diff --git a/test/README.testsuite b/test/README.testsuite new file mode 100644 index 0000000..9174e16 --- /dev/null +++ b/test/README.testsuite @@ -0,0 +1,310 @@ +The extended testsuite only works with UID=0. It consists of the subdirectories +named "test/TEST-??-*", each of which contains a description of an OS image and +a test which consists of systemd units and scripts to execute in this image. +The same image is used for execution under `systemd-nspawn` and `qemu`. + +To run the extended testsuite do the following: + +$ ninja -C build # Avoid building anything as root later +$ sudo test/run-integration-tests.sh +ninja: Entering directory `/home/zbyszek/src/systemd/build' +ninja: no work to do. +--x-- Running TEST-01-BASIC --x-- ++ make -C TEST-01-BASIC clean setup run +make: Entering directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC' +TEST-01-BASIC CLEANUP: Basic systemd setup +TEST-01-BASIC SETUP: Basic systemd setup +... +TEST-01-BASIC RUN: Basic systemd setup [OK] +make: Leaving directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC' +--x-- Result of TEST-01-BASIC: 0 --x-- +--x-- Running TEST-02-CRYPTSETUP --x-- ++ make -C TEST-02-CRYPTSETUP clean setup run + +If one of the tests fails, then $subdir/test.log contains the log file of +the test. + +To run just one of the cases: + +$ sudo make -C test/TEST-01-BASIC clean setup run + +Specifying the build directory +============================== + +If the build directory is not detected automatically, it can be specified +with BUILD_DIR=: + +$ sudo BUILD_DIR=some-other-build/ test/run-integration-tests + +or + +$ sudo make -C test/TEST-01-BASIC BUILD_DIR=../../some-other-build/ ... + +Note that in the second case, the path is relative to the test case directory. +An absolute path may also be used in both cases. + +Testing installed binaries instead of built +=========================================== + +To run the extended testsuite using the systemd installed on the system instead +of the systemd from a build, use the NO_BUILD=1: + +$ sudo NO_BUILD=1 test/run-integration-tests + +Configuration variables +======================= + +TEST_NO_QEMU=1 + Don't run tests under qemu + +TEST_QEMU_ONLY=1 + Run only tests that require qemu + +TEST_NO_NSPAWN=1 + Don't run tests under systemd-nspawn + +TEST_PREFER_NSPAWN=1 + Run all tests that do not require qemu under systemd-nspawn + +TEST_NO_KVM=1 + Disable qemu KVM auto-detection (may be necessary when you're trying to run the + *vanilla* qemu and have both qemu and qemu-kvm installed) + +TEST_NESTED_KVM=1 + Allow tests to run with nested KVM. By default, the testsuite disables + nested KVM if the host machine already runs under KVM. Setting this + variable disables such checks + +QEMU_MEM=512M + Configure amount of memory for qemu VMs (defaults to 512M) + +QEMU_SMP=1 + Configure number of CPUs for qemu VMs (defaults to 1) + +KERNEL_APPEND='...' + Append additional parameters to the kernel command line + +NSPAWN_ARGUMENTS='...' + Specify additional arguments for systemd-nspawn + +QEMU_TIMEOUT=infinity + Set a timeout for tests under qemu (defaults to 1800 sec) + +NSPAWN_TIMEOUT=infinity + Set a timeout for tests under systemd-nspawn (defaults to 1800 sec) + +INTERACTIVE_DEBUG=1 + Configure the machine to be more *user-friendly* for interactive debuggung + (e.g. by setting a usable default terminal, suppressing the shutdown after + the test, etc.) + +TEST_MATCH_SUBTEST=subtest + If the test makes use of `run_subtests` use this variable to provide + a POSIX extended regex to run only subtests matching the expression + +TEST_MATCH_TESTCASE=testcase + Same as $TEST_MATCH_SUBTEST but for subtests that make use of `run_testcases` + +The kernel and initrd can be specified with $KERNEL_BIN and $INITRD. (Fedora's +or Debian's default kernel path and initrd are used by default.) + +A script will try to find your qemu binary. If you want to specify a different +one with $QEMU_BIN. + +Debugging the qemu image +======================== + +If you want to log in the testsuite virtual machine, use INTERACTIVE_DEBUG=1 +and log in as root: + +$ sudo make -C test/TEST-01-BASIC INTERACTIVE_DEBUG=1 run + +The root password is empty. + +Ubuntu CI +========= + +New PR submitted to the project are run through regression tests, and one set +of those is the 'autopkgtest' runs for several different architectures, called +'Ubuntu CI'. Part of that testing is to run all these tests. Sometimes these +tests are temporarily deny-listed from running in the 'autopkgtest' tests while +debugging a flaky test; that is done by creating a file in the test directory +named 'deny-list-ubuntu-ci', for example to prevent the TEST-01-BASIC test from +running in the 'autopkgtest' runs, create the file +'TEST-01-BASIC/deny-list-ubuntu-ci'. + +The tests may be disabled only for specific archs, by creating a deny-list file +with the arch name at the end, e.g. +'TEST-01-BASIC/deny-list-ubuntu-ci-arm64' to disable the TEST-01-BASIC test +only on test runs for the 'arm64' architecture. + +Note the arch naming is not from 'uname -m', it is Debian arch names: +https://wiki.debian.org/ArchitectureSpecificsMemo + +For PRs that fix a currently deny-listed test, the PR should include removal +of the deny-list file. + +In case a test fails, the full set of artifacts, including the journal of the +failed run, can be downloaded from the artifacts.tar.gz archive which will be +reachable in the same URL parent directory as the logs.gz that gets linked on +the Github CI status. + +To add new dependencies or new binaries to the packages used during the tests, +a merge request can be sent to: https://salsa.debian.org/systemd-team/systemd +targeting the 'upstream-ci' branch. + +The cloud-side infrastructure, that is hooked into the Github interface, is +located at: + +https://git.launchpad.net/autopkgtest-cloud/ + +In case of infrastructure issues with this CI, things might go wrong in two +places: + +- starting a job: this is done via a Github webhook, so check if the HTTP POST + are failing on https://github.com/systemd/systemd/settings/hooks +- running a job: all currently running jobs are listed at + https://autopkgtest.ubuntu.com/running#pkg-systemd-upstream in case the PR + does not show the status for some reason +- reporting the job result: this is done on Canonical's cloud infrastructure, + if jobs are started and running but no status is visible on the PR, then it is + likely that reporting back is not working + +For infrastructure help, reaching out to Canonical via the #ubuntu-devel channel +on libera.chat is an effective way to receive support in general. + +Manually running a part of the Ubuntu CI test suite +=================================================== + +In some situations one may want/need to run one of the tests run by Ubuntu CI +locally for debugging purposes. For this, you need a machine (or a VM) with +the same Ubuntu release as is used by Ubuntu CI (Jammy ATTOW). + +First of all, clone the Debian systemd repository and sync it with the code of +the PR (set by the $UPSTREAM_PULL_REQUEST env variable) you'd like to debug: + +# git clone https://salsa.debian.org/systemd-team/systemd.git +# cd systemd +# git checkout upstream-ci +# TEST_UPSTREAM=1 UPSTREAM_PULL_REQUEST=12345 ./debian/extra/checkout-upstream + +Now install necessary build & test dependencies: + +## PPA with some newer Ubuntu packages required by upstream systemd +# add-apt-repository -y --enable-source ppa:upstream-systemd-ci/systemd-ci +# apt build-dep -y systemd +# apt install -y autopkgtest debhelper genisoimage git qemu-system-x86 \ + libcurl4-openssl-dev libfdisk-dev libtss2-dev libfido2-dev \ + libssl-dev python3-pefile + +Build systemd deb packages with debug info: + +# TEST_UPSTREAM=1 DEB_BUILD_OPTIONS="nocheck nostrip noopt" dpkg-buildpackage -us -uc +# cd .. + +Prepare a testbed image for autopkgtest (tweak the release as necessary): + +# autopkgtest-buildvm-ubuntu-cloud --ram-size 1024 -v -a amd64 -r jammy + +And finally run the autopkgtest itself: + +# autopkgtest -o logs *.deb systemd/ \ + --env=TEST_UPSTREAM=1 \ + --timeout-factor=3 \ + --test-name=boot-and-services \ + --shell-fail \ + -- autopkgtest-virt-qemu --cpus 4 --ram-size 2048 autopkgtest-jammy-amd64.img + +where --test-name= is the name of the test you want to run/debug. The +--shell-fail option will pause the execution in case the test fails and shows +you the information how to connect to the testbed for further debugging. + +Manually running CodeQL analysis +===================================== + +This is mostly useful for debugging various CodeQL quirks. + +Download the CodeQL Bundle from https://github.com/github/codeql-action/releases +and unpack it somewhere. From now the 'tutorial' assumes you have the `codeql` +binary from the unpacked archive in $PATH for brevity. + +Switch to the systemd repository if not already: + +$ cd <systemd-repo> + +Create an initial CodeQL database: + +$ CCACHE_DISABLE=1 codeql database create codeqldb --language=cpp -vvv + +Disabling ccache is important, otherwise you might see CodeQL complaining: + +No source code was seen and extracted to /home/mrc0mmand/repos/@ci-incubator/systemd/codeqldb. +This can occur if the specified build commands failed to compile or process any code. + - Confirm that there is some source code for the specified language in the project. + - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify + an explicit --command. + - For other languages, the --command must specify a "clean" build which compiles + all the source code files without reusing existing build artefacts. + +If you want to run all queries systemd uses in CodeQL, run: + +$ codeql database analyze codeqldb/ --format csv --output results.csv .github/codeql-custom.qls .github/codeql-queries/*.ql -vvv + +Note: this will take a while. + +If you're interested in a specific check, the easiest way (without hunting down +the specific CodeQL query file) is to create a custom query suite. For example: + +$ cat >test.qls <<EOF +- queries: . + from: codeql/cpp-queries +- include: + id: + - cpp/missing-return +EOF + +And then execute it in the same way as above: + +$ codeql database analyze codeqldb/ --format csv --output results.csv test.qls -vvv + +More about query suites here: https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/ + +The results are then located in the `results.csv` file as a comma separated +values list (obviously), which is the most human-friendly output format the +CodeQL utility provides (so far). + +Code coverage +============= + +We have a daily cron job in CentOS CI which runs all unit and integration tests, +collects coverage using gcov/lcov, and uploads the report to Coveralls[0]. In +order to collect the most accurate coverage information, some measures have +to be taken regarding sandboxing, namely: + + - ProtectSystem= and ProtectHome= need to be turned off + - the $BUILD_DIR with necessary .gcno files needs to be present in the image + and needs to be writable by all processes + +The first point is relatively easy to handle and is handled automagically by +our test "framework" by creating necessary dropins. + +Making the $BUILD_DIR accessible to _everything_ is slightly more complicated. +First, and foremost, the $BUILD_DIR has a POSIX ACL that makes it writable +to everyone. However, this is not enough in some cases, like for services +that use DynamicUser=yes, since that implies ProtectSystem=strict that can't +be turned off. A solution to this is to use ReadWritePaths=$BUILD_DIR, which +works for the majority of cases, but can't be turned on globally, since +ReadWritePaths= creates its own mount namespace which might break some +services. Hence, the ReadWritePaths=$BUILD_DIR is enabled for all services +with the `test-` prefix (i.e. test-foo.service or test-foo-bar.service), both +in the system and the user managers. + +So, if you're considering writing an integration test that makes use +of DynamicUser=yes, or other sandboxing stuff that implies it, please prefix +the test unit (be it a static one or a transient one created via systemd-run), +with `test-`, unless the test unit needs to be able to install mount points +in the main mount namespace - in that case use IGNORE_MISSING_COVERAGE=yes +in the test definition (i.e. TEST-*-NAME/test.sh), which will skip the post-test +check for missing coverage for the respective test. + +[0] https://coveralls.io/github/systemd/systemd |