Adding upstream version 1:9.18.19.upstream/1%9.18.19

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 843 insertions, 0 deletions

diff --git a/bin/tests/system/README b/bin/tests/system/README
new file mode 100644
index 0000000..265a9ef
--- /dev/null
+++ b/bin/tests/system/README
@@ -0,0 +1,843 @@
+Copyright (C) Internet Systems Consortium, Inc. ("ISC")
+
+SPDX-License-Identifier: MPL-2.0
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0.  If a copy of the MPL was not distributed with this
+file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+See the COPYRIGHT file distributed with this work for additional
+information regarding copyright ownership.
+
+Introduction
+===
+This directory holds a simple test environment for running bind9 system tests
+involving multiple name servers.
+
+With the exception of "common" (which holds configuration information common to
+multiple tests), each directory holds a set of scripts and configuration
+files to test different parts of BIND.  The directories are named for the
+aspect of BIND they test, for example:
+
+  dnssec/       DNSSEC tests
+  forward/      Forwarding tests
+  glue/         Glue handling tests
+
+etc.
+
+Typically each set of tests sets up 2-5 name servers and then performs one or
+more tests against them.  Within the test subdirectory, each name server has a
+separate subdirectory containing its configuration data.  These subdirectories
+are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2
+etc.)
+
+The tests are completely self-contained and do not require access to the real
+DNS.  Generally, one of the test servers (usually ns1) is set up as a root
+nameserver and is listed in the hints file of the others.
+
+
+Preparing to Run the Tests
+===
+To enable all servers to run on the same machine, they bind to separate virtual
+IP addresses on the loopback interface.  ns1 runs on 10.53.0.1, ns2 on
+10.53.0.2, etc.  Before running any tests, you must set up these addresses by
+running the command
+
+    sh ifconfig.sh up
+
+as root.  The interfaces can be removed by executing the command:
+
+    sh ifconfig.sh down
+
+... also as root.
+
+The servers use unprivileged ports (above 1024) instead of the usual port 53,
+so they can be run without root privileges once the interfaces have been set
+up.
+
+
+Note for MacOS Users
+---
+If you wish to make the interfaces survive across reboots, copy
+org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
+then run
+
+    launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
+
+... as root.
+
+
+Running the System Tests with pytest
+===
+
+The pytest system test runner is currently in development, but it is the
+recommended way to run tests. Please report issues to QA.
+
+Running an Individual Test
+---
+
+pytest -k <test-name>
+
+Note that in comparison to the legacy test runner, some additional tests might
+be picked up when specifying just the system test directory name. To check
+which tests will be executed, you can use the `--collect-only` option. You
+might also be able to find a more specific test name to provide to ensure only
+your desired test is executed. See help for `-k` option in `pytest --help` for
+more info.
+
+It is also possible to run a single individual pytest test case. For example,
+you can use the name test_sslyze_dot to execute just the test_sslyze_dot()
+function from doth/tests_sslyze.py. The entire needed setup and teardown will
+be handled by the framework.
+
+Running All the System Tests
+---
+
+Issuing plain `pytest` command without any argument will execute all tests
+sequenatially. To execute them in parallel, ensure you have pytest-xdist
+installed and run:
+
+pytest -n <number-of-workers>
+
+
+Running the System Tests Using the Legacy Runner
+===
+
+!!! WARNING !!!
+---
+The legacy way to run system tests is currently being reworked into a pytest
+system test runner described in the previous section. The contents of this
+section might be out of date and no longer applicable. Please try and use the
+pytest runner if possible and report issues and missing features.
+
+Running an Individual Test
+---
+The tests can be run individually using the following command:
+
+    sh legacy.run.sh [flags] <test-name> [<test-arguments>]
+
+e.g.
+
+    sh legacy.run.sh [flags] notify
+
+Optional flags are:
+
+    -k              Keep servers running after the test completes.  Each test
+                    usually starts a number of nameservers, either instances
+                    of the "named" being tested, or custom servers (written in
+                    Python or Perl) that feature test-specific behavior.  The
+                    servers are automatically started before the test is run
+                    and stopped after it ends.  This flag leaves them running
+                    at the end of the test, so that additional queries can be
+                    sent by hand.  To stop the servers afterwards, use the
+                    command "sh stop.sh <test-name>".
+
+    -n              Noclean - do not remove the output files if the test
+                    completes successfully.  By default, files created by the
+                    test are deleted if it passes;  they are not deleted if the
+                    test fails.
+
+    -p <number>     Sets the range of ports used by the test.  A block of 100
+                    ports is available for each test, the number given to the
+                    "-p" switch being the number of the start of that block
+                    (e.g. "-p 7900" will mean that the test is able to use
+                    ports 7900 through 7999).  If not specified, the test will
+                    have ports 5000 to 5099 available to it.
+
+Arguments are:
+
+    test-name       Mandatory.  The name of the test, which is the name of the
+                    subdirectory in bin/tests/system holding the test files.
+
+    test-arguments  Optional arguments that are passed to each of the test's
+                    scripts.
+
+
+Running All The System Tests
+---
+To run all the system tests, enter the command:
+
+    sh runall.sh [-c] [-n] [numproc]
+
+The optional flag "-c" forces colored output (by default system test output is
+not printed in color due to legacy.run.sh being piped through "tee").
+
+The optional flag "-n" has the same effect as it does for "legacy.run.sh" - it
+causes the retention of all output files from all tests.
+
+The optional "numproc" argument specifies the maximum number of tests that can
+run in parallel.  The default is 1, which means that all of the tests run
+sequentially.  If greater than 1, up to "numproc" tests will run simultaneously,
+new tests being started as tests finish.  Each test will get a unique set of
+ports, so there is no danger of tests interfering with one another.  Parallel
+running will reduce the total time taken to run the BIND system tests, but will
+mean that the output from all the tests sent to the screen will be mixed up
+with one another.  However, the systests.output file produced at the end of the
+run (in the bin/tests/system directory) will contain the output from each test
+in sequential order.
+
+Note that it is not possible to pass arguments to tests though the "runall.sh"
+script.
+
+A run of all the system tests can also be initiated via make:
+
+    make [-j numproc] test
+
+In this case, retention of the output files after a test completes successfully
+is specified by setting the environment variable SYSTEMTEST_NO_CLEAN to 1 prior
+to running make, e.g.
+
+    SYSTEMTEST_NO_CLEAN=1 make [-j numproc] test
+
+while setting environment variable SYSTEMTEST_FORCE_COLOR to 1 forces system
+test output to be printed in color.
+
+
+Running Multiple System Test Suites Simultaneously
+---
+In some cases it may be desirable to have multiple instances of the system test
+suite running simultaneously (e.g. from different terminal windows).  To do
+this:
+
+1. Each installation must have its own directory tree.  The system tests create
+files in the test directories, so separate directory trees are required to
+avoid interference between the same test running in the different
+installations.
+
+2. For one of the test suites, the starting port number must be specified by
+setting the environment variable STARTPORT before starting the test suite.
+Each test suite comprises about 100 tests, each being allocated a set of 100
+ports.  The port ranges for each test are allocated sequentially, so each test
+suite requires about 10,000 ports to itself.  By default, the port allocation
+starts at 5,000.  So the following set of commands:
+
+    Terminal Window 1:
+        cd <installation-1>/bin/tests/system
+        sh runall.sh 4
+
+    Terminal Window 2:
+        cd <installation-2>/bin/tests/system
+        STARTPORT=20000 sh runall.sh 4
+
+... will start the test suite for installation-1 using the default base port
+of 5,000, so the test suite will use ports 5,000 through 15,000 (or there
+abouts).  The use of "STARTPORT=20000" to prefix the run of the test suite for
+installation-2 will mean the test suite uses ports 20,000 through 30,000 or so.
+
+
+Format of Test Output
+---
+All output from the system tests is in the form of lines with the following
+structure:
+
+    <letter>:<test-name>:<message> [(<number>)]
+
+e.g.
+
+    I:catz:checking that dom1.example is not served by primary (1)
+
+The meanings of the fields are as follows:
+
+<letter>
+This indicates the type of message.  This is one of:
+
+    S   Start of the test
+    A   Start of test (retained for backwards compatibility)
+    T   Start of test (retained for backwards compatibility)
+    E   End of the test
+    I   Information.  A test will typically output many of these messages
+        during its run, indicating test progress.  Note that such a message may
+        be of the form "I:testname:failed", indicating that a sub-test has
+        failed.
+    R   Result.  Each test will result in one such message, which is of the
+        form:
+
+                R:<test-name>:<result>
+
+        where <result> is one of:
+
+            PASS        The test passed
+            FAIL        The test failed
+            SKIPPED     The test was not run, usually because some
+                        prerequisites required to run the test are missing.
+
+<test-name>
+This is the name of the test from which the message emanated, which is also the
+name of the subdirectory holding the test files.
+
+<message>
+This is text output by the test during its execution.
+
+(<number>)
+If present, this will correlate with a file created by the test.  The tests
+execute commands and route the output of each command to a file.  The name of
+this file depends on the command and the test, but will usually be of the form:
+
+    <command>.out.<suffix><number>
+
+e.g. nsupdate.out.test28, dig.out.q3.  This aids diagnosis of problems by
+allowing the output that caused the problem message to be identified.
+
+
+Re-Running the Tests
+---
+If there is a requirement to re-run a test (or the entire test suite), the
+files produced by the tests should be deleted first.  Normally, these files are
+deleted if the test succeeds but are retained on error.  The legacy.run.sh
+script automatically calls a given test's clean.sh script before invoking its
+setup.sh script.
+
+Deletion of the files produced by the set of tests (e.g. after the execution
+of "runall.sh") can be carried out using the command:
+
+    sh cleanall.sh
+
+or
+
+    make testclean
+
+(Note that the Makefile has two other targets for cleaning up files: "clean"
+will delete all the files produced by the tests, as well as the object and
+executable files used by the tests.  "distclean" does all the work of "clean"
+as well as deleting configuration files produced by "configure".)
+
+
+Developer Notes
+===
+This section is intended for developers writing new tests.
+
+
+Overview
+---
+As noted above, each test is in a separate directory.  To interact with the
+test framework, the directories contain the following standard files:
+
+prereq.sh   Run at the beginning to determine whether the test can be run at
+            all; if not, we see a R:SKIPPED result.  This file is optional:
+            if not present, the test is assumed to have all its prerequisites
+            met.
+
+setup.sh    Run after prereq.sh, this sets up the preconditions for the tests.
+            Although optional, virtually all tests will require such a file to
+            set up the ports they should use for the test.
+
+tests.sh    Runs the actual tests.  This file is mandatory.
+
+clean.sh    Run at the end to clean up temporary files, but only if the test
+            was completed successfully and its running was not inhibited by the
+            "-n" switch being passed to "legacy.run.sh".  Otherwise the
+            temporary files are left in place for inspection.
+
+ns<N>       These subdirectories contain test name servers that can be queried
+	    or can interact with each other.  The value of N indicates the
+	    address the server listens on: for example, ns2 listens on
+	    10.53.0.2, and ns4 on 10.53.0.4.  All test servers use an
+	    unprivileged port, so they don't need to run as root.  These
+	    servers log at the highest debug level and the log is captured in
+	    the file "named.run".
+
+ans<N>      Like ns[X], but these are simple mock name servers implemented in
+            Perl or Python.  They are generally programmed to misbehave in ways
+            named would not so as to exercise named's ability to interoperate
+            with badly behaved name servers.
+
+
+Port Usage
+---
+In order for the tests to run in parallel, each test requires a unique set of
+ports.  These are specified by the "-p" option passed to "legacy.run.sh", which
+sets environment variables that the scripts listed above can reference.
+
+The convention used in the system tests is that the number passed is the start
+of a range of 100 ports.  The test is free to use the ports as required,
+although the first ten ports in the block are named and generally tests use the
+named ports for their intended purpose.  The names of the environment variables
+are:
+
+    PORT                     Number to be used for the query port.
+    CONTROLPORT              Number to be used as the RNDC control port.
+    EXTRAPORT1 - EXTRAPORT8  Eight port numbers that can be used as needed.
+
+Two other environment variables are defined:
+
+    LOWPORT                  The lowest port number in the range.
+    HIGHPORT                 The highest port number in the range.
+
+Since port ranges usually start on a boundary of 10, the variables are set such
+that the last digit of the port number corresponds to the number of the
+EXTRAPORTn variable.  For example, if the port range were to start at 5200, the
+port assignments would be:
+
+    PORT = 5200
+    EXTRAPORT1 = 5201
+        :
+    EXTRAPORT8 = 5208
+    CONTROLPORT = 5209
+    LOWPORT = 5200
+    HIGHPORT = 5299
+
+When running tests in parallel (i.e. giving a value of "numproc" greater than 1
+in the "make" or "runall.sh" commands listed above), it is guaranteed that each
+test will get a set of unique port numbers.
+
+
+Writing a Test
+---
+The test framework requires up to four shell scripts (listed above) as well as
+a number of nameserver instances to run.  Certain expectations are put on each
+script:
+
+
+General
+---
+1. Each of the four scripts will be invoked with the command
+
+    (cd <test-directory> ; sh <script> [<arguments>] )
+
+... so that working directory when the script starts executing is the test
+directory.
+
+2. Arguments can be only passed to the script if the test is being run as a
+one-off with "legacy.run.sh".  In this case, everything on the command line
+after the name of the test is passed to each script.  For example, the command:
+
+    sh legacy.run.sh -p 12300 mytest -D xyz
+
+... will run "mytest" with a port range of 12300 to 12399.  Each of the
+framework scripts provided by the test will be invoked using the remaining
+arguments, e.g.:
+
+   (cd mytest ; sh prereq.sh -D xyz)
+   (cd mytest ; sh setup.sh -D xyz)
+   (cd mytest ; sh tests.sh -D xyz)
+   (cd mytest ; sh clean.sh -D xyz)
+
+No arguments will be passed to the test scripts if the test is run as part of
+a run of the full test suite (e.g. the tests are started with "runall.sh").
+
+3.  Each script should start with the following lines:
+
+    . ../conf.sh
+
+"conf.sh" defines a series of environment variables together with functions
+useful for the test scripts.
+
+
+prereq.sh
+---
+As noted above, this is optional.  If present, it should check whether specific
+software needed to run the test is available and/or whether BIND has been
+configured with the appropriate options required.
+
+    * If the software required to run the test is present and the BIND
+      configure options are correct, prereq.sh should return with a status code
+      of 0.
+
+    * If the software required to run the test is not available and/or BIND
+      has not been configured with the appropriate options, prereq.sh should
+      return with a status code of 1.
+
+    * If there is some other problem (e.g. prerequisite software is available
+      but is not properly configured), a status code of 255 should be returned.
+
+
+setup.sh
+---
+This is responsible for setting up the configuration files used in the test.
+
+To cope with the varying port number, ports are not hard-coded into
+configuration files (or, for that matter, scripts that emulate nameservers).
+Instead, setup.sh is responsible for editing the configuration files to set the
+port numbers.
+
+To do this, configuration files should be supplied in the form of templates
+containing tokens identifying ports.  The tokens have the same name as the
+environment variables listed above, but are prefixed and suffixed by the "@"
+symbol.  For example, a fragment of a configuration file template might look
+like:
+
+    controls {
+        inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
+    };
+
+    options {
+        query-source address 10.53.0.1;
+        notify-source 10.53.0.1;
+        transfer-source 10.53.0.1;
+        port @PORT@;
+        allow-new-zones yes;
+    };
+
+setup.sh should copy the template to the desired filename using the
+"copy_setports" shell function defined in "conf.sh", i.e.
+
+    copy_setports ns1/named.conf.in ns1/named.conf
+
+This replaces the tokens @PORT@, @CONTROLPORT@, @EXTRAPORT1@ through
+@EXTRAPORT8@ with the contents of the environment variables listed above.
+setup.sh should do this for all configuration files required when the test
+starts.
+
+("setup.sh" should also use this method for replacing the tokens in any Perl or
+Python name servers used in the test.)
+
+
+tests.sh
+---
+This is the main test file and the contents depend on the test.  The contents
+are completely up to the developer, although most test scripts have a form
+similar to the following for each sub-test:
+
+    1. n=$((n + 1))
+    2. echo_i "prime cache nodata.example ($n)"
+    3. ret=0
+    4. $DIG -p ${PORT} @10.53.0.1 nodata.example TXT > dig.out.test$n
+    5. grep "status: NOERROR" dig.out.test$n > /dev/null || ret=1
+    6. grep "ANSWER: 0," dig.out.test$n > /dev/null || ret=1
+    7. if [ $ret != 0 ]; then echo_i "failed"; fi
+    8. status=$((status + ret))
+
+1.  Increment the test number "n" (initialized to zero at the start of the
+    script).
+
+2.  Indicate that the sub-test is about to begin.  Note that "echo_i" instead
+    of "echo" is used.  echo_i is a function defined in "conf.sh" which will
+    prefix the message with "I:<testname>:", so allowing the output from each
+    test to be identified within the output.  The test number is included in
+    the message in order to tie the sub-test with its output.
+
+3. Initialize return status.
+
+4 - 6. Carry out the sub-test.  In this case, a nameserver is queried (note
+    that the port used is given by the PORT environment variable, which was set
+    by the inclusion of the file "conf.sh" at the start of the script).  The
+    output is routed to a file whose suffix includes the test number.  The
+    response from the server is examined and, in this case, if the required
+    string is not found, an error is indicated by setting "ret" to 1.
+
+7.  If the sub-test failed, a message is printed. "echo_i" is used to print
+    the message to add the prefix "I:<test-name>:" before it is output.
+
+8.  "status", used to track how many of the sub-tests have failed, is
+    incremented accordingly.  The value of "status" determines the status
+    returned by "tests.sh", which in turn determines whether the framework
+    prints the PASS or FAIL message.
+
+Regardless of this, rules that should be followed are:
+
+a.  Use the environment variables set by conf.sh to determine the ports to use
+    for sending and receiving queries.
+
+b.  Use a counter to tag messages and to associate the messages with the output
+    files.
+
+c.  Store all output produced by queries/commands into files.  These files
+    should be named according to the command that produced them, e.g. "dig"
+    output should be stored in a file "dig.out.<suffix>", the suffix being
+    related to the value of the counter.
+
+d.  Use "echo_i" to output informational messages.
+
+e.  Retain a count of test failures and return this as the exit status from
+    the script.
+
+
+clean.sh
+---
+The inverse of "setup.sh", this is invoked by the framework to clean up the
+test directory.  It should delete all files that have been created by the test
+during its run.
+
+
+Starting Nameservers
+---
+As noted earlier, a system test will involve a number of nameservers.  These
+will be either instances of named, or special servers written in a language
+such as Perl or Python.
+
+For the former, the version of "named" being run is that in the "bin/named"
+directory in the tree holding the tests (i.e. if "make test" is being run
+immediately after "make", the version of "named" used is that just built).  The
+configuration files, zone files etc. for these servers are located in
+subdirectories of the test directory named "nsN", where N is a small integer.
+The latter are special nameservers, mostly used for generating deliberately bad
+responses, located in subdirectories named "ansN" (again, N is an integer).
+In addition to configuration files, these directories should hold the
+appropriate script files as well.
+
+Note that the "N" for a particular test forms a single number space, e.g. if
+there is an "ns2" directory, there cannot be an "ans2" directory as well.
+Ideally, the directory numbers should start at 1 and work upwards.
+
+When running a test, the servers are started using "start.sh" (which is nothing
+more than a wrapper for start.pl).  The options for "start.pl" are documented
+in the header for that file, so will not be repeated here.  In summary, when
+invoked by "legacy.run.sh", start.pl looks for directories named "nsN" or
+"ansN" in the test directory and starts the servers it finds there.
+
+
+"named" Command-Line Options
+---
+By default, start.pl starts a "named" server with the following options:
+
+    -c named.conf   Specifies the configuration file to use (so by implication,
+                    each "nsN" nameserver's configuration file must be called
+                    named.conf).
+
+    -d 99           Sets the maximum debugging level.
+
+    -D <name>       The "-D" option sets a string used to identify the
+                    nameserver in a process listing.  In this case, the string
+                    is the name of the subdirectory.
+
+    -g              Runs the server in the foreground and logs everything to
+                    stderr.
+
+    -m record
+                    Turns on these memory usage debugging flags.
+
+    -U 4            Uses four listeners.
+
+    -X named.lock   Acquires a lock on this file in the "nsN" directory, so
+                    preventing multiple instances of this named running in this
+                    directory (which could possibly interfere with the test).
+
+All output is sent to a file called "named.run" in the nameserver directory.
+
+The options used to start named can be altered.  There are three ways of doing
+this.  "start.pl" checks the methods in a specific order: if a check succeeds,
+the options are set and any other specification is ignored.  In order, these
+are:
+
+1. Specifying options to "start.sh"/"start.pl" after the name of the test
+directory, e.g.
+
+    sh start.sh reclimit ns1 -- "-c n.conf -d 43"
+
+(This is only really useful when running tests interactively.)
+
+2. Including a file called "named.args" in the "nsN" directory.  If present,
+the contents of the first non-commented, non-blank line of the file are used as
+the named command-line arguments.  The rest of the file is ignored.
+
+3. Tweaking the default command line arguments with "-T" options.  This flag is
+used to alter the behavior of BIND for testing and is not documented in the
+ARM.  The presence of certain files in the "nsN" directory adds flags to
+the default command line (the content of the files is irrelevant - it
+is only the presence that counts):
+
+    named.noaa       Appends "-T noaa" to the command line, which causes
+                     "named" to never set the AA bit in an answer.
+
+    named.dropedns   Adds "-T dropedns" to the command line, which causes
+                     "named" to recognise EDNS options in messages, but drop
+                     messages containing them.
+
+    named.maxudp1460 Adds "-T maxudp1460" to the command line, setting the
+                     maximum UDP size handled by named to 1460.
+
+    named.maxudp512  Adds "-T maxudp512" to the command line, setting the
+                     maximum UDP size handled by named to 512.
+
+    named.noedns     Appends "-T noedns" to the command line, which disables
+                     recognition of EDNS options in messages.
+
+    named.notcp      Adds "-T notcp", which disables TCP in "named".
+
+    named.soa        Appends "-T nosoa" to the command line, which disables
+                     the addition of SOA records to negative responses (or to
+                     the additional section if the response is triggered by RPZ
+                     rewriting).
+
+Starting Other Nameservers
+---
+In contrast to "named", nameservers written in Perl or Python (whose script
+file should have the name "ans.pl" or "ans.py" respectively) are started with a
+fixed command line.  In essence, the server is given the address and nothing
+else.
+
+(This is not strictly true: Python servers are provided with the number of the
+query port to use.  Altering the port used by Perl servers currently requires
+creating a template file containing the "@PORT@" token, and having "setup.sh"
+substitute the actual port being used before the test starts.)
+
+
+Stopping Nameservers
+---
+As might be expected, the test system stops nameservers with the script
+"stop.sh", which is little more than a wrapper for "stop.pl".  Like "start.pl",
+the options available are listed in the file's header and will not be repeated
+here.
+
+In summary though, the nameservers for a given test, if left running by
+specifying the "-k" flag to "legacy.run.sh" when the test is started, can be
+stopped by the command:
+
+    sh stop.sh <test-name> [server]
+
+... where if the server (e.g. "ns1", "ans3") is not specified, all servers
+associated with the test are stopped.
+
+
+Adding a Test to the System Test Suite
+---
+Once a test has been created, the following files should be edited:
+
+* conf.sh.in  The name of the test should be added to the PARALLELDIRS or
+SEQUENTIALDIRS variables as appropriate.  The former is used for tests that
+can run in parallel with other tests, the latter for tests that are unable to
+do so.
+
+* Makefile.in The name of the test should be added to one of the the PARALLEL
+or SEQUENTIAL variables.
+
+(It is likely that a future iteration of the system test suite will remove the
+need to edit multiple files to add a test.)
+
+
+Valgrind
+---
+When running system tests, named can be run under Valgrind.  The output from
+Valgrind are sent to per-process files that can be reviewed after the test has
+completed.  To enable this, set the USE_VALGRIND environment variable to
+"helgrind" to run the Helgrind tool, or any other value to run the Memcheck
+tool.  To use "helgrind" effectively, build BIND with --disable-atomic.
+
+Developer Notes for pytest runner
+===
+
+Test discovery and collection
+---
+There are two distinct types of system tests. The first is a shell script
+tests.sh containing individual test cases executed sequentially and the
+success/failure is determined by return code. The second type is a regular
+pytest file which contains test functions.
+
+Dealing with the regular pytest files doesn't require any special consideration
+as long as the naming conventions are met. Discovering the tests.sh tests is
+more complicated.
+
+The chosen solution is to add a bit of glue for each system test. For every
+tests.sh, there is an accompanying tests_sh_*.py file that contains a test
+function which utilizes a custom run_tests_sh fixture to call the tests.sh
+script. Other solutions were tried and eventually rejected. While this
+introduces a bit of extra glue, it is the most portable, compatible and least
+complex solution.
+
+Module scope
+---
+Pytest fixtures can have a scope. The "module" scope is the most important for
+our use. A module is a python file which contains test functions. Every system
+test directory may contain multiple modules (i.e. tests_*.py files)!
+
+The server setup/teardown is done for a module. Bundling test cases together
+inside a single module may save some resources. However, test cases inside a
+single module can't be executed in parallel.
+
+It is possible to execute different modules defined within a single system test
+directory in parallel. This is possible thanks to executing the tests inside a
+temporary directory and proper port assignment to ensure there won't be any
+conflicts.
+
+Test logging
+---
+Each module has a separate log which will be saved as pytest.log.txt in the
+temporary directory in which the test is executed. This log includes messages
+for this module setup/teardown as well as any logging from the tests using the
+`logger` fixture. Logging level DEBUG and above will be present in this log.
+
+In general, any log messages using INFO or above will also be printed out
+during pytest execution. In CI, the pytest output is also saved to
+pytest.out.txt in the bin/tests/system directory.
+
+Parallel execution
+---
+As mentioned in the previous section, test cases inside a single module can't
+be executed in parallel. To put it differently, all tests cases inside the same
+module must be performed by the same worker/thread. Otherwise, server
+setup/teardown fixtures won't be shared and runtime issues due to port
+collisions are likely to occur.
+
+Pytest-xdist is used for executing pytest test cases in parallel using the `-n
+N_WORKERS` option. By default, xdist will distribute any test case to any
+worker, which would lead to the issue described above. Therefore, conftest.py
+enforces equivalent of `--dist loadscope` option which ensures that test cases
+within the same (module) scope will be handled by the same worker. Parallelism
+is automatically disabled when xdist.scheduler.loadscope library is not
+available.
+
+$ pytest -n auto
+
+Test selection
+---
+It is possible to run just a single pytest test case from any module. Use
+standard pytest facility to select the desired test case(s), i.e. pass a
+sufficiently unique identifier for `-k` parameter. You can also check which
+tests will be executed by using the `--collect-only` flag to debug your `-k`
+expression.
+
+Compatibility with older pytest version
+---
+Keep in mind that the pytest runner must work with ancient versions of pytest.
+When implementing new features, it is advisable to check feature support in
+pytest and pytest-xdist in older distributions first.
+
+As a general rule, any changes to the pytest runner need to keep working on all
+platforms in CI that use the pytest runner. As of 2023-01-13, the oldest
+supported version is whatever is available in EL8.
+
+We may need to add more compat code eventually to handle breaking upstream
+changes. For example, using request.fspath attribute is already deprecatred in
+latest pytest.
+
+Maintenance Notes for legacy runner
+===
+This section is aimed at developers maintaining BIND's system test framework.
+
+Notes on Parallel Execution
+---
+Although execution of an individual test is controlled by "legacy.run.sh",
+which executes the above shell scripts (and starts the relevant servers) for
+each test, the running of all tests in the test suite is controlled by the
+Makefile. ("runall.sh" does little more than invoke "make" on the Makefile.)
+
+All system tests are capable of being run in parallel.  For this to work, each
+test needs to use a unique set of ports.  To avoid the need to define which
+tests use which ports (and so risk port clashes as further tests are added),
+the ports are assigned when the tests are run.  This is achieved by having the
+"test" target in the Makefile depend on "parallel.mk".  That file is created
+when "make check" is run, and contains a target for each test of the form:
+
+    <test-name>:
+        @$(SHELL) legacy.run.sh -p <baseport> <test-name>
+
+The <baseport> is unique and the values of <baseport> for each test are
+separated by at least 100 ports.
+
+
+Cleaning Up From Tests
+---
+When a test is run, up to three different types of files are created:
+
+1. Files generated by the test itself, e.g. output from "dig" and "rndc", are
+stored in the test directory.
+
+2. Files produced by named which may not be cleaned up if named exits
+abnormally, e.g. core files, PID files etc., are stored in the test directory.
+
+3. A file "test.output.<test-name>" containing the text written to stdout by the
+test is written to bin/tests/system/.  This file is only produced when the test
+is run as part of the entire test suite (e.g. via "runall.sh").
+
+If the test fails, all these files are retained.  But if the test succeeds,
+they are cleaned up at different times:
+
+1. Files generated by the test itself are cleaned up by the test's own
+"clean.sh", which is called from "legacy.run.sh".
+
+2. Files that may not be cleaned up if named exits abnormally can be removed
+using the "cleanall.sh" script.
+
+3. "test.output.*" files are deleted when the test suite ends.  At this point,
+the file "testsummary.sh" is called which concatenates all the "test.output.*"
+files into a single "systests.output" file before deleting them.