summaryrefslogtreecommitdiffstats
path: root/Documentation/howto-tests.txt
blob: 15d088089d0518bce259f1a29374c585f295ed55 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
 util-linux regression tests
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

 It's expected that for each invasive change or important bugfix you will
 include a test to your patch.

 Compile binaries, libs, extra test programs and run the basic tests:

	$ make check

 Compile extra test programs only:

	$ make check-programs

 Note that the configure option --disable-static disables many of libmount and
 libblkid unit tests.

 Run all tests including tests that require root permissions:

	# cd tests
	# ./run.sh [options, see --help]

	Alternatively using sudo and make:

	$ make check-programs
	$ sudo -E make check TS_OPTS="--parallel=1"

 note that as root you have to manually remove output and diff directories

	# rm -rf output diff

 or run 'make clean' as root.


 Run subset of tests:

	$ cd tests
	$ ./run.sh <test_directory-name>

        for example:

              $ ./run.sh blkid
              $ ./run.sh libmount

        or individual test script, for example:

	      $ ./ts/cal/year

 The tests is possible to exclude by ./run.sh --exclude=<list>  where the
 <list> is blank separated test names in format "testdir/testname", for example:

	$ ./run.sh --exclude="mount/move"

 The --exclude is evaluated by the ./run.sh script only. See below

	TS_OPT_testdir_[testscript_]fake=

 environment variable which provides more powerful functionality to skip tests.


 *** WARNING for root users ***

 The tests touch your /etc/fstab, initialize loop devices or scsi_debug devices
 if executed with root permissions.

 Please, be careful and use these tests only for development and never on
 production system.

fuzz targets
------------

The fuzz targets can be built and run along with the other tests (after installing
clang):

	$ ./tools/config-gen fuzz
	$ make check

environment variables
---------------------

TS_COMMAND

    Evaluated by "make check" to override the default command (run.sh).
    Example:
      - build all test dependencies, but skip the actual test
        $ make check TS_COMMAND="true"

TS_OPTS

    Evaluated by "make check" to pass options.to run.sh (see ./run.sh --help).
    Examples:
      - run utmp tests only
        $ make check TS_OPTS="--parallel=1 utmp"

TS_OPT_testdir_[testscript_]fake="<yes|no>"

    Evaluated by any test script to skip certain tests.
    Examples:
      - skip all the tests within "fdisk" test-directory:
        $ make check TS_OPT_fdisk_fake="yes"

      - skip only "fdisk/bsd" test:
        $ make check TS_OPT_fdisk_bsd_fake="yes"

      - skip all "fdisk" tests except fdisk/bsd:
        $ make check TS_OPT_fdisk_fake="yes" TS_OPT_fdisk_bsd_fake="no"

TS_OPT_testdir_[testscript_]known_fail="<yes|no>"

    Similar usage like TS_OPT_*_fake above. "known_fail" means that the given
    test will run but (negative) results will be ignored. The build log and test
    diffs will still remind you about the issue.

TS_OPT_testdir_[testscript_]verbose="<yes|no>"

    Set verbosity for certain tests. Similar usage like TS_OPT_*_fake above.

TS_OPT_testdir_[testscript_]memcheck="<yes|no>"

    Run certain tests with valgrind. Similar usage like TS_OPT_*_fake above.


External services
-----------------

Coveralls coverage reports

    URL: https://coveralls.io/github/util-linux/util-linux

lgtm CI - automatically executed security code analysis

    URL: https://lgtm.com/projects/g/util-linux/util-linux/

Coverity Scan

    URL: https://scan.coverity.com/projects/karelzak-util-linux

Fossies codespell report

    URL: https://fossies.org/linux/test/util-linux-master.tar.gz/codespell.html

OSS-Fuzz

    URL: https://google.github.io/oss-fuzz/
    URL: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#util-linux
    URL: https://oss-fuzz.com/coverage-report/job/libfuzzer_asan_util-linux/latest

CIFuzz

    URL: https://google.github.io/oss-fuzz/getting-started/continuous-integration/
    URL: https://github.com/util-linux/util-linux/actions?query=workflow%3ACIFuzz