diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-05-05 18:37:14 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-05-05 18:37:14 +0000 |
commit | ea648e70a989cca190cd7403fe892fd2dcc290b4 (patch) | |
tree | e2b6b1c647da68b0d4d66082835e256eb30970e8 /unit/atf-src/doc | |
parent | Initial commit. (diff) | |
download | bind9-upstream.tar.xz bind9-upstream.zip |
Adding upstream version 1:9.11.5.P4+dfsg.upstream/1%9.11.5.P4+dfsgupstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'unit/atf-src/doc')
-rw-r--r-- | unit/atf-src/doc/Makefile.am.inc | 40 | ||||
-rw-r--r-- | unit/atf-src/doc/atf-test-case.4 | 321 | ||||
-rw-r--r-- | unit/atf-src/doc/atf-test-program.1 | 99 | ||||
-rw-r--r-- | unit/atf-src/doc/atf.7.in | 120 |
4 files changed, 580 insertions, 0 deletions
diff --git a/unit/atf-src/doc/Makefile.am.inc b/unit/atf-src/doc/Makefile.am.inc new file mode 100644 index 0000000..5f8a4a3 --- /dev/null +++ b/unit/atf-src/doc/Makefile.am.inc @@ -0,0 +1,40 @@ +# Copyright (c) 2007 The NetBSD Foundation, Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +# CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +# IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +# GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +# OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +# IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +dist_man_MANS += doc/atf-test-case.4 \ + doc/atf-test-program.1 + +man_MANS += doc/atf.7 +CLEANFILES += doc/atf.7 +EXTRA_DIST += doc/atf.7.in + +doc/atf.7: $(srcdir)/doc/atf.7.in + $(AM_V_GEN)test -d doc || mkdir -p doc; \ + sed -e 's#__DOCDIR__#$(docdir)#g' \ + -e 's#__TESTSDIR__#$(testsdir)#g' \ + <$(srcdir)/doc/atf.7.in >doc/atf.7.tmp; \ + mv doc/atf.7.tmp doc/atf.7 + +# vim: syntax=make:noexpandtab:shiftwidth=8:softtabstop=8 diff --git a/unit/atf-src/doc/atf-test-case.4 b/unit/atf-src/doc/atf-test-case.4 new file mode 100644 index 0000000..3025411 --- /dev/null +++ b/unit/atf-src/doc/atf-test-case.4 @@ -0,0 +1,321 @@ +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 5, 2014 +.Dt ATF-TEST-CASE 4 +.Os +.Sh NAME +.Nm atf-test-case +.Nd generic description of test cases +.Sh DESCRIPTION +A +.Em test case +is a piece of code that stress-tests a specific feature of the software. +This feature is typically self-contained enough, either in the amount of +code that implements it or in the general idea that describes it, to +warrant its independent testing. +Given this, test cases are very fine-grained, but they attempt to group +similar smaller tests which are semantically related. +.Pp +A test case is defined by three components regardless of the language it is +implemented in: a header, a body and a cleanup routine. +The +.Em header +is, basically, a declarative piece of code that defines several +properties to describe what the test case does and how it behaves. +In other words: it defines the test case's +.Em meta-data , +further described in the +.Sx Meta-data +section. +The +.Em body +is the test case itself. +It executes all actions needed to reproduce the test, and checks for +failures. +This body is only executed if the abstract conditions specified by the +header are met. +The +.Em cleanup +routine is a piece of code always executed after the body, regardless of +the exit status of the test case. +It can be used to undo side-effects of the test case. +Note that almost all side-effects of a test case are automatically cleaned +up by the library; this is explained in more detail in the rest of this +document. +.Pp +It is extremely important to keep the separation between a test case's +header and body well-defined, because the header is +.Em always +parsed, whereas the body is only executed when the conditions defined in +the header are met and when the user specifies that test case. +.Pp +At last, test cases are always contained into test programs. +The test programs act as a front-end to them, providing a consistent +interface to the user and several APIs to ease their implementation. +.Ss Results +Upon termination, a test case reports a status and, optionally, a textual +reason describing why the test reported such status. +The caller must ensure that the test case really performed the task that its +status describes, as the test program may be bogus and therefore providing a +misleading result (e.g. providing a result that indicates success but the +error code of the program says otherwise). +.Pp +The possible exit status of a test case are one of the following: +.Bl -tag -width expectedXfailureXX +.It expected_death +The test case expects to terminate abruptly. +.It expected_exit +The test case expects to exit cleanly. +.It expected_failure +The test case expects to exit with a controller fatal/non-fatal failure. +If this happens, the test program exits with a success error code. +.It expected_signal +The test case expects to receive a signal that makes it terminate. +.It expected_timeout +The test case expects to execute for longer than its timeout. +.It passed +The test case was executed successfully. +The test program exits with a success error code. +.It skipped +The test case could not be executed because some preconditions were not +met. +This is not a failure because it can typically be resolved by adjusting +the system to meet the necessary conditions. +This is always accompanied by a +.Em reason , +a message describing why the test was skipped. +The test program exits with a success error code. +.It failed +An error appeared during the execution of the test case. +This is always accompanied by a +.Em reason , +a message describing why the test failed. +The test program exits with a failure error code. +.El +.Pp +The usefulness of the +.Sq expected_* +results comes when writing test cases that verify known failures caused, +in general, due to programming errors (aka bugs). +Whenever the faulty condition that the +.Sq expected_* +result is trying to cover is fixed, then the test case will be reported as +.Sq failed +and the developer will have to adjust it to match its new condition. +.Pp +It is important to note that all +.Sq expected_* +results are only provided as a +.Em hint +to the caller; the caller must verify that the test case did actually terminate +as the expected condition says. +.Ss Input/output +Test cases are free to print whatever they want to their +.Xr stdout 4 +and +.Xr stderr 4 +file descriptors. +They are, in fact, encouraged to print status information as they execute +to keep the user informed of their actions. +This is specially important for long test cases. +.Pp +Test cases will log their results to an auxiliary file, which is then +collected by the test program they are contained in. +The developer need not care about this as long as he uses the correct +APIs to implement the test cases. +.Pp +The standard input of the test cases is unconditionally connected to +.Sq /dev/zero . +.Ss Meta-data +The following list describes all meta-data properties interpreted +internally by ATF. +You are free to define new properties in your test cases and use them as +you wish, but non-standard properties must be prefixed by +.Sq X- . +.Bl -tag -width requireXmachineXX +.It descr +Type: textual. +Required. +.Pp +A brief textual description of the test case's purpose. +Will be shown to the user in reports. +Also good for documentation purposes. +.It has.cleanup +Type: boolean. +Optional. +.Pp +If set to true, specifies that the test case has a cleanup routine that has +to be executed by the runtime engine during the cleanup phase of the execution. +This property is automatically set by the framework when defining a test case +with a cleanup routine, so it should never be set by hand. +.It ident +Type: textual. +Required. +.Pp +The test case's identifier. +Must be unique inside the test program and should be short but descriptive. +.It require.arch +Type: textual. +Optional. +.Pp +A whitespace separated list of architectures that the test case can be run +under without causing errors due to an architecture mismatch. +.It require.config +Type: textual. +Optional. +.Pp +A whitespace separated list of configuration variables that must be defined +to execute the test case. +If any of the required variables is not defined, the test case is +.Em skipped . +.It require.diskspace +Type: integer. +Optional. +Specifies the minimum amount of available disk space needed by the test. +The value can have a size suffix such as +.Sq K , +.Sq M , +.Sq G +or +.Sq T +to make the amount of bytes easier to type and read. +.It require.files +Type: textual. +Optional. +.Pp +A whitespace separated list of files that must be present to execute the +test case. +The names of these files must be absolute paths. +If any of the required files is not found, the test case is +.Em skipped . +.It require.machine +Type: textual. +Optional. +.Pp +A whitespace separated list of machine types that the test case can be run +under without causing errors due to a machine type mismatch. +.It require.memory +Type: integer. +Optional. +Specifies the minimum amount of physical memory needed by the test. +The value can have a size suffix such as +.Sq K , +.Sq M , +.Sq G +or +.Sq T +to make the amount of bytes easier to type and read. +.It require.progs +Type: textual. +Optional. +.Pp +A whitespace separated list of programs that must be present to execute +the test case. +These can be given as plain names, in which case they are looked in the +user's +.Ev PATH , +or as absolute paths. +If any of the required programs is not found, the test case is +.Em skipped . +.It require.user +Type: textual. +Optional. +.Pp +The required privileges to execute the test case. +Can be one of +.Sq root +or +.Sq unprivileged . +.Pp +If the test case is running as a regular user and this property is +.Sq root , +the test case is +.Em skipped . +.Pp +If the test case is running as root and this property is +.Sq unprivileged , +the runtime engine will automatically drop the privileges if the +.Sq unprivileged-user +configuration property is set; otherwise the test case is +.Em skipped . +.It timeout +Type: integral. +Optional; defaults to +.Sq 300 . +.Pp +Specifies the maximum amount of time the test case can run. +This is particularly useful because some tests can stall either because they +are incorrectly coded or because they trigger an anomalous behavior of the +program. +It is not acceptable for these tests to stall the whole execution of the +test program. +.Pp +Can optionally be set to zero, in which case the test case has no run-time +limit. +This is discouraged. +.El +.Ss Environment +Every time a test case is executed, several environment variables are +cleared or reseted to sane values to ensure they do not make the test fail +due to unexpected conditions. +These variables are: +.Bl -tag -width LCXMESSAGESXX +.It Ev HOME +Set to the work directory's path. +.It Ev LANG +Undefined. +.It Ev LC_ALL +Undefined. +.It Ev LC_COLLATE +Undefined. +.It Ev LC_CTYPE +Undefined. +.It Ev LC_MESSAGES +Undefined. +.It Ev LC_MONETARY +Undefined. +.It Ev LC_NUMERIC +Undefined. +.It Ev LC_TIME +Undefined. +.It Ev TZ +Hardcoded to +.Sq UTC . +.El +.Ss Work directories +The test program always creates a temporary directory +and switches to it before running the test case's body. +This way the test case is free to modify its current directory as it +wishes, and the runtime engine will be able to clean it up later on in a +safe way, removing any traces of its execution from the system. +To do so, the runtime engine will perform a recursive removal of the work +directory without crossing mount points; if a mount point is found, the +file system will be unmounted (if possible). +.Ss File creation mode mask (umask) +Test cases are always executed with a file creation mode mask (umask) of +.Sq 0022 . +The test case's code is free to change this during execution. +.Sh SEE ALSO +.Xr atf-test-program 1 diff --git a/unit/atf-src/doc/atf-test-program.1 b/unit/atf-src/doc/atf-test-program.1 new file mode 100644 index 0000000..0ca1963 --- /dev/null +++ b/unit/atf-src/doc/atf-test-program.1 @@ -0,0 +1,99 @@ +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd March 2, 2014 +.Dt ATF-TEST-PROGRAM 1 +.Os +.Sh NAME +.Nm atf-test-program +.Nd common interface to ATF test programs +.Sh SYNOPSIS +.Nm +.Op Fl r Ar resfile +.Op Fl s Ar srcdir +.Op Fl v Ar var1=value1 Op .. Fl v Ar varN=valueN +.Ar test_case +.Nm +.Fl l +.Sh DESCRIPTION +Test programs written using the ATF libraries all share a common user +interface, which is what this manual page describes. +.Em NOTE: There is no binary known as +.Nm ; +.Em what is described in this manual page is the command-line interface +.Em exposed by the atf-c, atf-c++ and atf-sh bindings . +.Pp +In the first synopsis form, the test program will execute the provided +test case and print its results to the standard output, unless otherwise +stated by the +.Fl r +flag. +Optionally, the test case name can be suffixed by +.Sq :cleanup , +in which case the cleanup routine of the test case will be executed +instead of the test case body; see +.Xr atf-test-case 4 . +Note that the test case is +.Em executed without isolation , +so it can and probably will create and modify files in the current directory. +To execute test cases in a controller manner, you need a runtime engine +that understands the ATF interface. +The recommended runtime engine is +.Xr kyua 1 . +You should only execute test cases by hand for debugging purposes. +.Pp +In the second synopsis form, the test program will list all available +test cases alongside their meta-data properties in a format that is +machine parseable. +This list is processed by +.Xr kyua 1 +to know how to execute the test cases of a given test program. +.Pp +The following options are available: +.Bl -tag -width XvXvarXvalueXX +.It Fl l +Lists available test cases alongside a brief description for each of them. +.It Fl r Ar resfile +Specifies the file that will receive the test case result. +If not specified, the test case prints its results to stdout. +If the result of a test case needs to be parsed by another program, you must +use this option to redirect the result to a file and then read the resulting +file from the other program. +Note: +.Em do not try to process the stdout of the test case +because your program may break in the future. +.It Fl s Ar srcdir +The path to the directory where the test program is located. +This is needed in all cases, except when the test program is being executed +from the current directory. +The test program will use this path to locate any helper data files or +utilities. +.It Fl v Ar var=value +Sets the configuration variable +.Ar var +to the value +.Ar value . +.El +.Sh SEE ALSO +.Xr kyua 1 diff --git a/unit/atf-src/doc/atf.7.in b/unit/atf-src/doc/atf.7.in new file mode 100644 index 0000000..ded00c1 --- /dev/null +++ b/unit/atf-src/doc/atf.7.in @@ -0,0 +1,120 @@ +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd September 14, 2014 +.Dt ATF 7 +.Os +.Sh NAME +.Nm ATF +.Nd introduction to the Automated Testing Framework +.Sh DESCRIPTION +The Automated Testing Framework +.Pf ( Nm ) +is a +.Em collection of libraries +to implement test programs in a variety of languages. +These libraries all offer similar functionality and any test program +written with them exposes a consistent user interface. +.Pp +Test programs using the +.Nm +libraries rely on a separate runtime engine to execute them in a +deterministic fashion. +The runtime engine isolates the test programs from the rest of the system +and ensures some common side-effects are cleaned up. +The runtime engine is also responsible for gathering the results of all +tests and composing reports. +The current runtime of choice is Kyua, described in +.Xr kyua 1 . +.Pp +If your operating systems distributes +.Nm , +it should also provide an introductory +.Xr tests 7 +manual page. +You are encouraged to read it now. +.Pp +The rest of this manual page serves as a cross-reference to all the other +documentation shipped with +.Nm . +.Ss Language bindings +.Bl -tag -width atfXtestXprogramXXXXX +.It Xr atf-c 3 +C programming interface. +.It Xr atf-c++ 3 +C++ programming interface. +.It Xr atf-sh 3 +.Xr sh 1 +programming interface. +.El +.Ss Miscellaneous pages +.Bl -tag -width atfXtestXprogramXXXXX +.It Xr atf-test-case 4 +Generic description of test cases, independent of the language they are +implemented in. +.It Xr atf-test-program 1 +Common interface provided by the test programs written using the +.Nm +libraries. +.El +.Sh SEE ALSO +.Xr kyua 1 , +.Xr tests 7 +.Sh HISTORY +.Nm +started as a Google Summer of Code 2007 project mentored by The NetBSD +Foundation. +Its original goal was to provide a testing framework for the +.Nx +operating system, but it grew as an independent project because the +framework itself did not need to be tied to a specific operating system. +.Pp +Originally, +.Nm +shipped the collection of libraries described in this manual page as well +as a runtime engine. +The runtime engine has since been replaced by Kyua and the old tools were +removed in +.Nm 0.20 , +which shipped in early 2014. +.Pp +As of late 2014, both +.Fx +and +.Nx +ship +.Nm +in their base systems and provide extensive test suites based on it. +.Pp +For more details on historical changes, refer to: +.Bd -literal -offset indent +.Pa __DOCDIR__/NEWS +.Ed +.Sh AUTHORS +For more details on the people that made +.Nm +possible, refer to: +.Bd -literal -offset indent +.Pa __DOCDIR__/AUTHORS +.Ed |