diff options
Diffstat (limited to 'unit/atf-src/doc/atf-test-case.4')
| -rw-r--r-- | unit/atf-src/doc/atf-test-case.4 | 321 |
1 files changed, 321 insertions, 0 deletions
diff --git a/unit/atf-src/doc/atf-test-case.4 b/unit/atf-src/doc/atf-test-case.4 new file mode 100644 index 0000000..3025411 --- /dev/null +++ b/unit/atf-src/doc/atf-test-case.4 @@ -0,0 +1,321 @@ +.\" Copyright (c) 2007 The NetBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND +.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, +.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY +.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER +.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 5, 2014 +.Dt ATF-TEST-CASE 4 +.Os +.Sh NAME +.Nm atf-test-case +.Nd generic description of test cases +.Sh DESCRIPTION +A +.Em test case +is a piece of code that stress-tests a specific feature of the software. +This feature is typically self-contained enough, either in the amount of +code that implements it or in the general idea that describes it, to +warrant its independent testing. +Given this, test cases are very fine-grained, but they attempt to group +similar smaller tests which are semantically related. +.Pp +A test case is defined by three components regardless of the language it is +implemented in: a header, a body and a cleanup routine. +The +.Em header +is, basically, a declarative piece of code that defines several +properties to describe what the test case does and how it behaves. +In other words: it defines the test case's +.Em meta-data , +further described in the +.Sx Meta-data +section. +The +.Em body +is the test case itself. +It executes all actions needed to reproduce the test, and checks for +failures. +This body is only executed if the abstract conditions specified by the +header are met. +The +.Em cleanup +routine is a piece of code always executed after the body, regardless of +the exit status of the test case. +It can be used to undo side-effects of the test case. +Note that almost all side-effects of a test case are automatically cleaned +up by the library; this is explained in more detail in the rest of this +document. +.Pp +It is extremely important to keep the separation between a test case's +header and body well-defined, because the header is +.Em always +parsed, whereas the body is only executed when the conditions defined in +the header are met and when the user specifies that test case. +.Pp +At last, test cases are always contained into test programs. +The test programs act as a front-end to them, providing a consistent +interface to the user and several APIs to ease their implementation. +.Ss Results +Upon termination, a test case reports a status and, optionally, a textual +reason describing why the test reported such status. +The caller must ensure that the test case really performed the task that its +status describes, as the test program may be bogus and therefore providing a +misleading result (e.g. providing a result that indicates success but the +error code of the program says otherwise). +.Pp +The possible exit status of a test case are one of the following: +.Bl -tag -width expectedXfailureXX +.It expected_death +The test case expects to terminate abruptly. +.It expected_exit +The test case expects to exit cleanly. +.It expected_failure +The test case expects to exit with a controller fatal/non-fatal failure. +If this happens, the test program exits with a success error code. +.It expected_signal +The test case expects to receive a signal that makes it terminate. +.It expected_timeout +The test case expects to execute for longer than its timeout. +.It passed +The test case was executed successfully. +The test program exits with a success error code. +.It skipped +The test case could not be executed because some preconditions were not +met. +This is not a failure because it can typically be resolved by adjusting +the system to meet the necessary conditions. +This is always accompanied by a +.Em reason , +a message describing why the test was skipped. +The test program exits with a success error code. +.It failed +An error appeared during the execution of the test case. +This is always accompanied by a +.Em reason , +a message describing why the test failed. +The test program exits with a failure error code. +.El +.Pp +The usefulness of the +.Sq expected_* +results comes when writing test cases that verify known failures caused, +in general, due to programming errors (aka bugs). +Whenever the faulty condition that the +.Sq expected_* +result is trying to cover is fixed, then the test case will be reported as +.Sq failed +and the developer will have to adjust it to match its new condition. +.Pp +It is important to note that all +.Sq expected_* +results are only provided as a +.Em hint +to the caller; the caller must verify that the test case did actually terminate +as the expected condition says. +.Ss Input/output +Test cases are free to print whatever they want to their +.Xr stdout 4 +and +.Xr stderr 4 +file descriptors. +They are, in fact, encouraged to print status information as they execute +to keep the user informed of their actions. +This is specially important for long test cases. +.Pp +Test cases will log their results to an auxiliary file, which is then +collected by the test program they are contained in. +The developer need not care about this as long as he uses the correct +APIs to implement the test cases. +.Pp +The standard input of the test cases is unconditionally connected to +.Sq /dev/zero . +.Ss Meta-data +The following list describes all meta-data properties interpreted +internally by ATF. +You are free to define new properties in your test cases and use them as +you wish, but non-standard properties must be prefixed by +.Sq X- . +.Bl -tag -width requireXmachineXX +.It descr +Type: textual. +Required. +.Pp +A brief textual description of the test case's purpose. +Will be shown to the user in reports. +Also good for documentation purposes. +.It has.cleanup +Type: boolean. +Optional. +.Pp +If set to true, specifies that the test case has a cleanup routine that has +to be executed by the runtime engine during the cleanup phase of the execution. +This property is automatically set by the framework when defining a test case +with a cleanup routine, so it should never be set by hand. +.It ident +Type: textual. +Required. +.Pp +The test case's identifier. +Must be unique inside the test program and should be short but descriptive. +.It require.arch +Type: textual. +Optional. +.Pp +A whitespace separated list of architectures that the test case can be run +under without causing errors due to an architecture mismatch. +.It require.config +Type: textual. +Optional. +.Pp +A whitespace separated list of configuration variables that must be defined +to execute the test case. +If any of the required variables is not defined, the test case is +.Em skipped . +.It require.diskspace +Type: integer. +Optional. +Specifies the minimum amount of available disk space needed by the test. +The value can have a size suffix such as +.Sq K , +.Sq M , +.Sq G +or +.Sq T +to make the amount of bytes easier to type and read. +.It require.files +Type: textual. +Optional. +.Pp +A whitespace separated list of files that must be present to execute the +test case. +The names of these files must be absolute paths. +If any of the required files is not found, the test case is +.Em skipped . +.It require.machine +Type: textual. +Optional. +.Pp +A whitespace separated list of machine types that the test case can be run +under without causing errors due to a machine type mismatch. +.It require.memory +Type: integer. +Optional. +Specifies the minimum amount of physical memory needed by the test. +The value can have a size suffix such as +.Sq K , +.Sq M , +.Sq G +or +.Sq T +to make the amount of bytes easier to type and read. +.It require.progs +Type: textual. +Optional. +.Pp +A whitespace separated list of programs that must be present to execute +the test case. +These can be given as plain names, in which case they are looked in the +user's +.Ev PATH , +or as absolute paths. +If any of the required programs is not found, the test case is +.Em skipped . +.It require.user +Type: textual. +Optional. +.Pp +The required privileges to execute the test case. +Can be one of +.Sq root +or +.Sq unprivileged . +.Pp +If the test case is running as a regular user and this property is +.Sq root , +the test case is +.Em skipped . +.Pp +If the test case is running as root and this property is +.Sq unprivileged , +the runtime engine will automatically drop the privileges if the +.Sq unprivileged-user +configuration property is set; otherwise the test case is +.Em skipped . +.It timeout +Type: integral. +Optional; defaults to +.Sq 300 . +.Pp +Specifies the maximum amount of time the test case can run. +This is particularly useful because some tests can stall either because they +are incorrectly coded or because they trigger an anomalous behavior of the +program. +It is not acceptable for these tests to stall the whole execution of the +test program. +.Pp +Can optionally be set to zero, in which case the test case has no run-time +limit. +This is discouraged. +.El +.Ss Environment +Every time a test case is executed, several environment variables are +cleared or reseted to sane values to ensure they do not make the test fail +due to unexpected conditions. +These variables are: +.Bl -tag -width LCXMESSAGESXX +.It Ev HOME +Set to the work directory's path. +.It Ev LANG +Undefined. +.It Ev LC_ALL +Undefined. +.It Ev LC_COLLATE +Undefined. +.It Ev LC_CTYPE +Undefined. +.It Ev LC_MESSAGES +Undefined. +.It Ev LC_MONETARY +Undefined. +.It Ev LC_NUMERIC +Undefined. +.It Ev LC_TIME +Undefined. +.It Ev TZ +Hardcoded to +.Sq UTC . +.El +.Ss Work directories +The test program always creates a temporary directory +and switches to it before running the test case's body. +This way the test case is free to modify its current directory as it +wishes, and the runtime engine will be able to clean it up later on in a +safe way, removing any traces of its execution from the system. +To do so, the runtime engine will perform a recursive removal of the work +directory without crossing mount points; if a mount point is found, the +file system will be unmounted (if possible). +.Ss File creation mode mask (umask) +Test cases are always executed with a file creation mode mask (umask) of +.Sq 0022 . +The test case's code is free to change this during execution. +.Sh SEE ALSO +.Xr atf-test-program 1 |