Adding upstream version 6.0.4-dfsg.upstream/6.0.4-dfsgupstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 1062 insertions, 0 deletions

diff --git a/src/VBox/ValidationKit/docs/AutomaticTestingRevamp.txt b/src/VBox/ValidationKit/docs/AutomaticTestingRevamp.txt
new file mode 100644
index 00000000..525384b6
--- /dev/null
+++ b/src/VBox/ValidationKit/docs/AutomaticTestingRevamp.txt
@@ -0,0 +1,1062 @@
+
+Revamp of Automatic VirtualBox Testing
+======================================
+
+
+Introduction
+------------
+
+This is the design document for a revamped automatic testing framework.
+The revamp aims at replacing the current tinderbox based testing by a new
+system that is written from scratch.
+
+The old system is not easy to work with and was never meant to be used for
+managing tests, after all it just a simple a build manager tailored for
+contiguous building.  Modifying the existing tinderbox system to do what
+we want would require fundamental changes that would render it useless as
+a build manager, it would therefore end up as a fork.  The amount of work
+required would probably be about the same as writing a new system from
+scratch.  Other considerations, such as the license of the tinderbox
+system (MPL) and language it is realized in (Perl), are also in favor of
+doing it from scratch.
+
+The language envisioned for the new automatic testing framework is Python.  This
+is for several reasons:
+
+  - The VirtualBox API has Python bindings.
+  - Python is used quite a bit inside Sun (dunno about Oracle).
+  - Works relatively well with Apache for the server side bits.
+  - It is more difficult to produce write-only code in Python (alias the
+    we-don't-like-perl argument).
+  - You don't need to compile stuff.
+
+Note that the author of this document has no special training as a test
+engineer and may therefore be using the wrong terms here and there.  The
+primary focus is to express what we need to do in order to improve
+testing.
+
+This document is written in reStructuredText (rst) which just happens to
+be used by Python, the primary language for this revamp.  For more
+information on reStructuredText: http://docutils.sourceforge.net/rst.html
+
+
+Definitions / Glossary
+======================
+
+sub-test driver
+  A set of test cases that can be used by more than one test driver.  Could
+  also be called a test unit, in the pascal sense of unit, if it wasn't so
+  easily confused with 'unit test'.
+
+test
+  This is somewhat ambiguous and this document try avoid using it where
+  possible.  When used it normally refers to doing testing by executing one or
+  more testcases.
+
+test case
+  A set of inputs, test programs and expected results. It validates system
+  requirements and generates a pass or failed status.  A basic unit of testing.
+  Note that we use the term in a rather broad sense.
+
+test driver
+  A program/script used to execute a test.  Also known as a test harness.
+  Generally abbreviated 'td'.  It can have sub-test drivers.
+
+test manager
+  Software managing the automatic testing.  This is a web application that runs
+  on a dedicated server (tindertux).
+
+test set
+  The output of testing activity.  Logs, results, ++.  Our usage of this should
+  probably be renamed to 'test run'.
+
+test group
+  A collection of related test cases.
+
+testbox
+  A computer that does testing.
+
+testbox script
+  Script executing orders from the test manager on a testbox.  Started
+  automatically upon bootup.
+
+testing
+  todo
+
+TODO: Check that we've got all this right and make them more exact
+      where possible.
+
+See also http://encyclopedia2.thefreedictionary.com/testing%20types
+and http://www.aptest.com/glossary.html .
+
+
+
+Objectives
+==========
+
+ - A scalable test manager (>200 testboxes).
+ - Optimize the web user interface (WUI) for typical workflows and analysis.
+ - Efficient and flexibile test configuration.
+ - Import test result from other test systems (logo testing, VDI, ++).
+ - Easy to add lots of new testscripts.
+ - Run tests locally without a manager.
+ - Revamp a bit at the time.
+
+
+
+The Testbox Side
+================
+
+Each testbox has a unique name corresponding to its DNS zone entry.  When booted
+a testbox script is started automatically.  This script will query the test
+manager for orders and execute them.  The core order downloads and executes a
+test driver with parameters (configuration) from the server.  The test driver
+does all the necessary work for executing the test.  In a typical VirtualBox
+test this means picking a build, installing it, configuring VMs, running the
+test VMs, collecting the results, submitting them to the server, and finally
+cleaning up afterwards.
+
+The testbox environment which the test drivers are executed in will have a
+number of environment variables for determining location of the source images
+and other test data, scratch space, test set id, server URL, and so on and so
+forth.
+
+On startup, the testbox script will look for crash dumps and similar on
+systems where this is possible.  If any sign of a crash is found, it will
+put any dumps and reports in the upload directory and inform the test
+manager before reporting for duty.  In order to generate the proper file
+names and report the crash in the right test set as well as prevent
+reporting crashes unrelated to automatic testing, the testbox script will
+keep information (test set id, ++) in a separate scratch directory
+(${TESTBOX_PATH_SCRATCH}/../testbox) and make sure it is synced to the
+disk (both files and directories).
+
+After checking for crashes, the testbox script will clean up any previous test
+which might be around.  This involves first invoking the test script in cleanup
+mode and the wiping the scratch space.
+
+When reporting for duty the script will submit information about the host: OS
+name, OS version, OS bitness, CPU vendor, total number of cores, VT-x support,
+AMD-V support, amount of memory, amount of scratch space, and anything else that
+can be found useful for scheduling tests or filtering test configurations.
+
+
+
+Testbox Script Orders
+---------------------
+
+The orders are kept in a queue on the server and the testbox script will fetch
+them one by one.  Orders that cannot be executed at the moment will be masked in
+the query from the testbox.
+
+Execute Test Driver
+  Downloads and executes the a specified test driver with the given
+  configuration (arguments).  Only one test driver can be executed at a time.
+  The server can specify more than one ZIP file to be downloaded and unpacked
+  before executing the test driver.  The testbox script may cache these zip
+  files using http time stamping.
+
+Abort Test Driver
+  Aborts the current test driver.  This will drop a hint to the driver and give
+  it 60 seconds to shut down the normal way.  If that fails, the testbox script
+  will kill the driver processes (SIGKILL or equivalent), invoke the
+  testdriver in cleanup mode, and finally wipe the scratch area.  Should either
+  of the last two steps fail in some way, the testbox will be rebooted.
+
+Idle
+  Ask again in X seconds, where X is specified by the server.
+
+Reboot
+  Reboot the testbox.  If a test driver is current running, an attempt at
+  aborting it (Abort Test Driver) will be made first.
+
+Update
+  Updates the testbox script.  The order includes a server relative path to the
+  new testbox script.  This can only be executed when no test driver is
+  currently being executed.
+
+
+Testbox Environment: Variables
+------------------------------
+
+COMSPEC
+  This will be set to C:\Windows\System32\cmd.exe on Windows.
+
+PATH
+  This will contain the kBuild binary directory for the host platform.
+
+SHELL
+  This will be set to point to kmk_ash(.exe) on all platforms.
+
+TESTBOX_NAME
+  The testbox name.
+  This is not required by the local reporter.
+
+TESTBOX_PATH_BUILDS
+  The absolute path to where the build repository can be found.  This should be
+  a read only mount when possible.
+
+TESTBOX_PATH_RESOURCES
+  The absolute path to where static test resources like ISOs and VDIs can be
+  found.  The test drivers knows the layout of this.  This should be a read only
+  mount when possible.
+
+TESTBOX_PATH_SCRATCH
+  The absolute path to the scratch space.  This is the current directory when
+  starting the test driver.  It will be wiped automatically after executing the
+  test.
+  (Envisioned as ${TESTBOX_PATH_SCRIPTS}/../scratch and that
+  ${TESTBOX_PATH_SCRATCH}/ will be automatically wiped by the testbox script.)
+
+TESTBOX_PATH_SCRIPTS
+  The absolute path to the test driver and the other files that was unzipped
+  together with it.  This is also where the test-driver-abort file will be put.
+  (Envisioned as ${TESTBOX_PATH_SCRATCH}/../driver, see above.)
+
+TESTBOX_PATH_UPLOAD
+  The absolute path to the upload directory for the testbox.  This is for
+  putting VOBs, PNGs, core dumps, crash dumps, and such on.  The files should be
+  bzipped or zipped if they aren't compress already.  The names should contain
+  the testbox and test set ID.
+
+TESTBOX_REPORTER
+  The name of the test reporter back end.  If not present, it will default to
+  the local reporter.
+
+TESTBOX_TEST_SET_ID
+  The test set ID if we're running.
+  This is not required by the local reporter.
+
+TESTBOX_MANAGER_URL
+  The URL to the test manager.
+  This is not required by the local reporter.
+
+TESTBOX_XYZ
+  There will probably be some more of these.
+
+
+Testbox Environment: Core Utilities
+-----------------------------------
+
+The testbox will not provide the typical unix /bin and /usr/bin utilities.  In
+other words, cygwin will not be used on Windows!
+
+The testbox will provide the unixy utilties that ships with kBuild and possibly
+some additional ones from tools/*.*/bin in the VirtualBox tree (wget, unzip,
+zip, and so on).  The test drivers will avoid invoking any of these utilites
+directly and instead rely on generic utility methods in the test driver
+framework.  That way we can more easily reimplement the functionality of the
+core utilites and drop the dependency on them.  It also allows us to quickly
+work around platform specific oddities and bugs.
+
+
+Test Drivers
+------------
+
+The test drivers are programs that will do the actual testing.  In addition to
+run under the testbox script, they can be executed in the VirtualBox development
+environment.  This is important for bug analysis and for simplifying local
+testing by the developers before commiting changes.  It also means the test
+drivers can be developed locally in the VirtualBox development environment.
+
+The main difference between executing a driver under the testbox script and
+running it manually is that there is no test manager in the latter case.  The
+test result reporter will not talk to the server, but report things to a local
+log file and/or standard out/err.  When invoked manually, all the necessary
+arguments will need to be specified by hand of course - it should be possible
+to extract them from a test set as well.
+
+For the early implementation stages, an implementation of the reporter interface
+that talks to the tinderbox base test manager will be needed.  This will be
+dropped later on when a new test manager is ready.
+
+As hinted at in other sections, there will be a common framework
+(libraries/packages/classes) for taking care of the tedious bits that every
+test driver needs to do.  Sharing code is essential to easing test driver
+development as well as reducing their complexity.  The framework will contain:
+
+    - A generic way of submitting output.  This will be a generic interface with
+      multiple implementation, the TESTBOX_REPORTER environment variable
+      will decide which of them to use.  The interface will have very specific
+      methods to allow the reporter to do a best possible job in reporting the
+      results to the test manager.
+
+    - Helpers for typical tasks, like:
+        - Copying files.
+        - Deleting files, directory trees and scratch space.
+        - Unzipping files.
+        - Creating ISOs
+        - And such things.
+
+    - Helpers for installing and uninstalling VirtualBox.
+
+    - Helpers for defining VMs. (The VBox API where available.)
+
+    - Helpers for controlling VMs. (The VBox API where available.)
+
+The VirtualBox bits will be separate from the more generic ones, simply because
+this is cleaner it will allow us to reuse the system for testing other products.
+
+The framework will be packaged in a zip file other than the test driver so we
+don't waste time and space downloading the same common code.
+
+The test driver will poll for the file
+${TESTBOX_PATH_SCRIPTS}/test-driver-abort and abort all testing when it sees it.
+
+The test driver can be invoked in three modes: execute, help and cleanup.  The
+default is execute mode, the help shows an configuration summary and the cleanup
+is for cleaning up after a reboot or aborted run.  The latter is done by the
+testbox script on startup and after abort - the driver is expected to clean up
+by itself after a normal run.
+
+
+
+The Server Side
+===============
+
+The server side will be implemented using a webserver (apache), a database
+(postgres) and cgi scripts (Python).  In addition a cron job (Python) running
+once a minute will generate static html for frequently used pages and maybe
+execute some other tasks for driving the testing forwards.  The order queries
+from the testbox script is the primary driving force in the system.  The total
+makes up the test manager.
+
+The test manager can be split up into three rough parts:
+
+  - Configuration (of tests, testgroups and testboxes).
+  - Execution     (of tests, collecting and organizing the output).
+  - Analysis      (of test output, mostly about presentation).
+
+
+Test Manager: Requirements
+==========================
+
+List of requirements:
+
+  - Two level testing - L1 quick smoke tests and L2 longer tests performed on
+    builds passing L1.  (Klaus (IIRC) ment this could be realized using
+    test dependency.)
+  - Black listing builds (by revision or similar) known to be bad.
+  - Distinguish between build types so we can do a portion of the testing with
+    strict builds.
+  - Easy to re-configure build source for testing different branch or for
+    testing a release candidate. (Directory based is fine.)
+  - Useful to be able to partition testboxes (run specific builds on some
+    boxes, let an engineer have a few boxes for a while).
+  - Interation with ILOM/...: reset systems.
+  - Be able to suspend testing on selected testboxes when doing maintance
+    (where automatically resuming testing on reboot is undesired) or similar
+    activity.
+  - Abort testing on seleced testboxes.
+  - Scheduling of tests requiring more than one testbox.
+  - Scheduling of tests that cannot be executing concurrently on several
+    machines because of some global resource like an iSCSI target.
+  - Jump the scheduling queue.  Scheduling of specified test the next time a
+    testbox is available (optionally specifying which testbox to schedule it
+    on).
+  - Configure tests with variable configuration to get better coverage. Two modes:
+      - TM generates the permutations based on one or more sets of test script arguments.
+      - Each configuration permuation is specified manually.
+  - Test specification needs to be flexible (select tests, disable test, test
+    scheduling (run certain tests nightly), ... ).
+  - Test scheduling by hour+weekday and by priority.
+  - Test dependencies (test A depends on test B being successful).
+  - Historize all configuration data, in particular test configs (permutations
+    included) and testboxes.
+  - Test sets has at a minimum a build reference, a testbox reference and a
+    primary log associated with it.
+  - Test sets stores further result as a recursive collection of:
+      - hierachical subtest name (slash sep)
+      - test parameters / config
+      - bool fail/succ
+      - attributes (typed?)
+      - test time
+      - e.g. throughput
+      - subresults
+      - log
+      - screenshots, video,...
+  - The test sets database structure needs to designed such that data mining
+    can be done in an efficient manner.
+  - Presentation/analysis: graphs!, categorize bugs, columns reorganizing
+    grouped by test (hierarchical), overviews, result for last day.
+
+
+
+Test Manager: Configuration
+===========================
+
+
+Testboxes
+---------
+
+Configuration of testboxes doesn't involve much work normally.  A testbox
+is added manually to the test manager by entering the DNS entry and/or IP
+address (the test manager resolves the missing one when necessary) as well as
+the system UUID (when obtainable - should be displayed by the testbox script
+installer).  Queries from unregistered testboxes will be declined as a kind of
+security measure, the incident should be logged in the webserver log if
+possible.   In later dealings with the client the System UUID will be the key
+identifier.  It's permittable for the IP address to change when the testbox
+isn't online, but not while testing (just imagine live migration tests and
+network tests).  Ideally, the testboxes should not change IP address.
+
+The testbox edit function must allow changing the name and system UUID.
+
+One further idea for the testbox configuration is indicating what they are
+capable of to filter out tests and test configurations that won't work on that
+testbox.  To examplify this take the ACP2 installation test.  If the test
+manager does not make sure the testbox have VT-x or AMD-v capabilities, the test
+is surely going to fail.  Other testbox capabilities would be total number of
+CPU cores, memory size, scratch space.  These testbox capabilities should be
+collected automatically on bootup by the testbox script together with OS name,
+OS version and OS bitness.
+
+A final thought, instead of outright declining all requests from new testboxes,
+we could record the unregistered testboxes with ip, UUID, name, os info and
+capabilities but mark them as inactive.  The test operator can then activate
+them on an activation page or edit the testbox or something.
+
+
+Testcases
+---------
+
+We use the term testcase for a test.
+
+
+Testgroups
+----------
+
+Testcases are organized into groups.  A testcase can be member of more than one
+group.  The testcase gets a priority assigned to it in connection with the
+group membership.
+
+Testgroups are picked up by a testbox partition (aka scheduling group) and a
+prioirty, scheduling time restriction and dependencies on other test groups are
+associated with the assignment.  A testgroup can be used by several testbox
+partitions.
+
+(This used to be called 'testsuites' but was renamed to avoid confusion with
+the VBox Test Suite.)
+
+
+Scheduling
+----------
+
+The initial scheduler will be modelled after what we're doing already on in the
+tinderbox driven testing.  It's best described as a best effort continuous
+integration scheduler.  Meaning, it will always use the latest build suitable
+for a testcase.  It will schedule on a testcase level, using the combined
+priority of the testcase in the test group and the test group with the testbox
+partition, trying to spread the test case argument varation out accordingly
+over the whole scheduilng queue.  Which argument variation to start with, is
+not undefined (random would be best).
+
+Later, we may add other schedulers as needed.
+
+
+
+The Test Manager Database
+=========================
+
+First a general warning:
+
+    The guys working on this design are not database experts, web
+    programming experts or similar, rather we are low level guys
+    who's main job is x86 & AMD64 virtualization.  So, please don't
+    be too hard on us. :-)
+
+
+A logical table layout can be found in TestManagerDatabaseMap.png (created by
+Oracle SQL Data Modeler, stored in TestManagerDatabase.dmd).  The physical
+database layout can be found in TestManagerDatabaseInit.pgsql postgreSQL
+script.  The script is commented.
+
+
+Data History
+------------
+
+We need to somehow track configuration changes over time.  We also need to
+be able to query the exact configuration a test set was run with so we can
+understand and make better use of the results.
+
+There are different techniques for archiving this, one is tuple-versioning
+( http://en.wikipedia.org/wiki/Tuple-versioning ), another is log trigger
+( http://en.wikipedia.org/wiki/Log_trigger ).  We use tuple-versioning in
+this database, with 'effective' as start date field name and 'expire' as
+the end (exclusive).
+
+Tuple-versioning has a shortcomming wrt to keys, both primary and foreign.
+The primary key of a table employing tuple-versioning is really
+'id' + 'valid_period', where the latter is expressed using two fields
+([effective...expire-1]).  Only, how do you tell the database engine that
+it should not allow overlapping valid_periods?  Useful suggestions are
+welcomed. :-)
+
+Foreign key references to a table using tuple-versioning is running into
+trouble because of the time axsis and that to our knowledge foreign keys
+must reference exactly one row in the other table.  When time is involved
+what we wish to tell the database is that at any given time, there actually
+is exactly one row we want to match in the other table, only we've no idea
+how to express this.  So, many foreign keys are not expressed in SQL of this
+database.
+
+In some cases, we extend the tuple-versioning with a generation ID so that
+normal foreign key referencing can be used.  We only use this for recording
+(references in testset) and scheduling (schedqueue), as using it more widely
+would force updates (gen_id changes) to propagate into all related tables.
+
+See also:
+    - http://en.wikipedia.org/wiki/Slowly_changing_dimension
+    - http://en.wikipedia.org/wiki/Change_data_capture
+    - http://en.wikipedia.org/wiki/Temporal_database
+
+
+
+Test Manager: Execution
+=======================
+
+
+
+Test Manager: Scenarios
+=======================
+
+
+
+#1 - Testbox Signs On (At Bootup)
+---------------------------------
+
+The testbox supplies a number of inputs when reporting for duty:
+    - IP address.
+    - System UUID.
+    - OS name.
+    - OS version.
+    - CPU architecture.
+    - CPU count (= threads).
+    - CPU VT-x/AMD-V capability.
+    - CPU nested paging capability.
+    - Chipset I/O MMU capability.
+    - Memory size.
+    - Scratch size space (for testing).
+    - Testbox Script revision.
+
+Results:
+    - ACK or NACK.
+    - Testbox ID and name on ACK.
+
+After receiving a ACK the testbox will ask for work to do, i.e. continue with
+scenario #2.  In the NACK case, it will sleep for 60 seconds and try again.
+
+
+Actions:
+
+1. Validate the testbox by looking the UUID up in the TestBoxes table.
+   If not found, NACK the request.  SQL::
+
+        SELECT  idTestBox, sName
+        FROM    TestBoxes
+        WHERE   uuidSystem = :sUuid
+          AND   tsExpire = 'infinity'::timestamp;
+
+2. Check if any of the information by testbox script has changed.  The two
+   sizes are normalized first, memory size rounded to nearest 4 MB and scratch
+   space is rounded down to nearest 64 MB.  If anything changed, insert a new
+   row in the testbox table and historize the current one, i.e. set
+   OLD.tsExpire to NEW.tsEffective and get a new value for NEW.idGenTestBox.
+
+3. Check with TestBoxStatuses:
+        a) If there is an row for the testbox in it already clean up change it
+           to 'idle' state and deal with any open testset like described in
+           scenario #9.
+        b) If there is no row, add one with 'idle' state.
+
+4. ACK the request and pass back the idTestBox.
+
+
+Note! Testbox.enabled is not checked here, that is only relevant when it asks
+      for a new task (scenario #2 and #5).
+
+Note! Should the testbox script detect changes in any of the inputs, it should
+      redo the sign in.
+
+Note! In scenario #8, the box will not sign on until it has done the reboot and
+      cleanup reporting!
+
+
+#2 - Testbox Asks For Work To Do
+---------------------------------
+
+
+Inputs:
+    - The testbox is supplying its IP indirectly.
+    - The testbox should supply its UUID and ID directly.
+
+Results:
+    - IDLE, WAIT, EXEC, REBOOT, UPGRADE, UPGRADE-AND-REBOOT, SPECIAL or DEAD.
+
+Actions:
+
+1. Validate the ID and IP by selecting the currently valid testbox row::
+
+     SELECT  idGenTestBox, fEnabled, idSchedGroup, enmPendingCmd
+     FROM    TestBoxes
+     WHERE   id = :id
+       AND   uuidSystem = :sUuid
+       AND   ip = :ip
+       AND   tsExpire = 'infinity'::timestamp;
+
+   If NOT found return DEAD to the testbox client (it will go back to sign on
+   mode and retry every 60 seconds or so - see scenario #1).
+
+   Note! The WUI will do all necessary clean-ups when deleting a testbox, so
+         contrary to the initial plans, we don't need to do anything more for
+         the DEAD status.
+
+2. Check with TestBoxStatuses (maybe joined with query from 1).
+
+   If enmState is 'gang-gathering': Goto scenario #6 on timeout or pending
+   'abort' or 'reboot' command.  Otherwise, tell the testbox to WAIT [done].
+
+   If enmState is 'gang-testing': The gang has been gathered and execution
+   has been triggered.  Goto 5.
+
+   If enmState is not 'idle', change it to 'idle'.
+
+   If idTestSet is not NULL, CALL scenario #9 to it up.
+
+   If there is a pending abort command, remove it.
+
+   If there is a pending command and the old state doesn't indicate that it was
+   being executed, GOTO scenario #3.
+
+   Note! There should be a TestBoxStatuses row after executing scenario #1,
+         however should none be found for some funky reason, returning DEAD
+         will fix the problem (see above)
+
+3. If the testbox was marked as disabled, respond with an IDLE command to the
+   testbox [done]. (Note! Must do this after TestBoxStatuses maintainance from
+   point 2, or abandoned tests won't be cleaned up after a testbox is disabled.)
+
+4. Consider testcases in the scheduling queue, pick the first one which the
+   testbox can execute.  There is a concurrency issue here, so we put and
+   exclusive lock on the SchedQueues table while considering its content.
+
+   The cursor we open looks something like this::
+
+     SELECT  idItem, idGenTestCaseArgs,
+             idTestSetGangLeader, cMissingGangMembers
+     FROM    SchedQueues
+     WHERE   idSchedGroup = :idSchedGroup
+        AND  (   bmHourlySchedule is NULL
+              OR get_bit(bmHourlySchedule, :iHourOfWeek) = 1 ) --< does this work?
+     ORDER BY ASC idItem;
+
+  If there no rows are returned (this can happen because no testgroups are
+  associated with this scheduling group, the scheduling group is disabled,
+  or because the queue is being regenerated), we will tell the testbox to
+  IDLE [done].
+
+  For each returned row we will:
+     a) Check testcase/group dependencies.
+     b) Select a build (and default testsuite) satisfying the dependencies.
+     c) Check the testcase requirements with that build in mind.
+     d) If idTestSetGangLeader is NULL, try allocate the necessary resources.
+     e) If it didn't check out, fetch the next row and redo from (a).
+     f) Tentatively create a new test set row.
+     g) If not gang scheduling:
+            - Next state: 'testing'
+        ElIf we're the last gang participant:
+            - Set idTestSetGangLeader to NULL.
+            - Set cMissingGangMembers to 0.
+            - Next state: 'gang-testing'
+        ElIf we're the first gang member:
+            - Set cMissingGangMembers to TestCaseArgs.cGangMembers - 1.
+            - Set idTestSetGangLeader to our idTestSet.
+            - Next state: 'gang-gathering'
+        Else:
+            - Decrement cMissingGangMembers.
+            - Next state: 'gang-gathering'
+
+        If we're not gang scheduling OR cMissingGangMembers is 0:
+            Move the scheduler queue entry to the end of the queue.
+
+        Update our TestBoxStatuses row with the new state and test set.
+        COMMIT;
+
+5. If state is 'testing' or 'gang-testing':
+        EXEC reponse.
+
+        The EXEC response for a gang scheduled testcase includes a number of
+        extra arguments so that the script knows the position of the testbox
+        it is running on and of the other members.  This means the that the
+        TestSet.iGangMemberNo is passed using --gang-member-no and the IP
+        addresses of the all gang members using --gang-ipv4-<memb-no> <ip>.
+   Else (state is 'gang-gathering'):
+        WAIT
+
+
+
+#3 - Pending Command When Testbox Asks For Work
+-----------------------------------------------
+
+This is a subfunction of scenario #2 and #5.
+
+As seen in scenario #2, the testbox will send 'abort' commands to /dev/null
+when it finds one when not executing a test.  This includes when it reports
+that the test has completed (no need to abort a completed test, wasting lot
+of effort when standing at the finish line).
+
+The other commands, though, are passed back to the testbox.  The testbox
+script will respond with an ACK or NACK as it sees fit.  If NACKed, the
+pending command will be removed (pending_cmd set to none) and that's it.
+If ACKed, the state of the testbox will change to that appropriate for the
+command and the pending_cmd set to none.  Should the testbox script fail to
+respond, the command will be repeated the next time it asks for work.
+
+
+
+#4 - Testbox Uploads Results During Test
+----------------------------------------
+
+
+TODO
+
+
+#5 - Testbox Completes Test and Asks For Work
+---------------------------------------------
+
+This is very similar to scenario #2
+
+TODO
+
+
+#6 - Gang Gathering Timeout
+---------------------------
+
+This is a subfunction of scenario #2.
+
+When gathering a gang of testboxes for a testcase, we do not want to wait
+forever and have testboxes doing nothing for hours while waiting for partners.
+So, the gathering has a reasonable timeout (imagine something like 20-30 mins).
+
+Also, we need some way of dealing with 'abort' and 'reboot' commands being
+issued while waiting.  The easy way out is pretent it's a time out.
+
+When changing the status to 'gang-timeout' we have to be careful.  First of all,
+we need to exclusively lock the SchedQueues and TestBoxStatuses (in that order)
+and re-query our status.  If it changed redo the checks in scenario #2 point 2.
+
+If we still want to timeout/abort, change the state from 'gang-gathering' to
+'gang-gathering-timedout' on all the gang members that has gathered so far.
+Then reset the scheduling queue record and move it to the end of the queue.
+
+
+When acting on 'gang-timeout' the TM will fail the testset in a manner similar
+to scenario #9.  No need to repeat that.
+
+
+
+#7 - Gang Cleanup
+-----------------
+
+When a testbox completes a gang scheduled test, we will have to serialize
+resource cleanup (both globally and on testboxes) as they stop.  More details
+can be found in the documentation of 'gang-cleanup'.
+
+So, the transition from 'gang-testing' is always to 'gang-cleanup'.  When we
+can safely leave 'gang-cleanup' is decided by the query::
+
+        SELECT  COUNT(*)
+        FROM    TestBoxStatuses,
+                TestSets
+        WHERE   TestSets.idTestSetGangLeader = :idTestSetGangLeader
+            AND TestSets.idTestBox = TestBoxStatuses.idTestBox
+            AND TestBoxStatuses.enmState = 'gang-running'::TestBoxState_T;
+
+As long as there are testboxes still running, we stay in the 'gang-cleanup'
+state.  Once there are none, we continue closing the testset and such.
+
+
+
+#8 - Testbox Reports A Crash During Test Execution
+--------------------------------------------------
+
+TODO
+
+
+#9 - Cleaning Up Abandoned Testcase
+-----------------------------------
+
+This is a subfunction of scenario #1 and #2.  The actions taken are the same in
+both situations.  The precondition for taking this path is that the row in the
+testboxstatus table is refering to a testset (i.e. testset_id is not NULL).
+
+
+Actions:
+
+1. If the testset is incomplete, we need to completed:
+        a) Add a message to the root TestResults row, creating one if necesary,
+           that explains that the test was abandoned.  This is done
+           by inserting/finding the string into/in TestResultStrTab and adding
+           a row to TestResultMsgs with idStrMsg set to that string id and
+           enmLevel set to 'failure'.
+        b) Mark the testset as failed.
+
+2. Free any global resources referenced by the test set.  This is done by
+   deleting all rows in GlobalResourceStatuses matching the testbox id.
+
+3. Set the idTestSet to NULL in the TestBoxStatuses row.
+
+
+
+#10 - Cleaning Up a Disabled/Dead TestBox
+-----------------------------------------
+
+The UI needs to be able to clean up the remains of a testbox which for some
+reason is out of action.  Normal cleaning up of abandoned testcases requires
+that the testbox signs on or asks for work, but if the testbox is dead or
+in some way indisposed, it won't be doing any of that.  So, the testbox
+sheriff needs to have a way of cleaning up after it.
+
+It's basically a manual scenario #9 but with some safe guards, like checking
+that the box hasn't been active for the last 1-2 mins (max idle/wait time * 2).
+
+
+Note! When disabling a box that still executing the testbox script, this
+      cleanup isn't necessary as it will happen automatically.   Also, it's
+      probably desirable that the testbox finishes what ever it is doing first
+      before going dormant.
+
+
+
+Test Manager: Analysis
+=======================
+
+One of the testbox sheriff's tasks is to try figure out the reason why something
+failed.  The test manager will provide facilities for doing so from very early
+in it's implementation.
+
+
+We need to work out some useful status reports for the early implementation.
+Later there will be more advanced analysis tools, where for instance we can
+create graphs from selected test result values or test execution times.
+
+
+
+Implementation Plan
+===================
+
+This has changed for various reasons.  The current plan is to implement the
+infrastructure (TM & testbox script) first and do a small deployment with the
+2-5 test drivers in the Testsuite as basis.  Once the bugs are worked out, we
+will convert the rest of the tests and start adding new ones.
+
+We just need to finally get this done, no point in doing it piecemeal by now!
+
+
+Test Manager Implementation Sub-Tasks
+-------------------------------------
+
+The implementation of the test manager and adjusting/completing of the testbox
+script and the test drivers are tasks which can be done by more than one
+person.  Splitting up the TM implementation into smaller tasks should allow
+parallel development of different tasks and get us working code sooner.
+
+
+Milestone #1
+------------
+
+The goal is to getting the fundamental testmanager engine implemented, debugged
+and working.  With the exception of testboxes, the configuration will be done
+via SQL inserts.
+
+Tasks in somewhat prioritized order:
+
+    - Kick off test manager.  It will live in testmanager/.  Salvage as much as
+      possible from att/testserv.  Create basic source and file layout.
+
+    - Adjust the testbox script, part one.  There currently is a testbox script
+      in att/testbox, this shall be moved up into testboxscript/.  The script
+      needs to be adjusted according to the specification layed down earlier
+      in this document.  Installers or installation scripts for all relevant
+      host OSes are required.  Left for part two is result reporting beyond the
+      primary log.  This task must be 100% feature complete, on all host OSes,
+      there is no room for FIXME, XXX or @todo here.
+
+    - Implement the schedule queue generator.
+
+    - Implement the testbox dispatcher in TM.  Support all the testbox script
+      responses implemented above, including upgrading the testbox script.
+
+    - Implement simple testbox management page.
+
+    - Implement some basic activity and result reports so that we can see
+      what's going on.
+
+    - Create a testmanager / testbox test setup.  This lives in selftest/.
+
+      1. Set up something that runs, no fiddly bits. Debug till it works.
+      2. Create a setup that tests testgroup dependencies, i.e. real tests
+         depending on smoke tests.
+      3. Create a setup that exercises testcase dependency.
+      4. Create a setup that exercises global resource allocation.
+      5. Create a setup that exercises gang scheduling.
+
+    - Check that all features work.
+
+
+Milestone #2
+------------
+
+The goal is getting to VBox testing.
+
+Tasks in somewhat prioritized order:
+
+    - Implement full result reporting in the testbox script and testbox driver.
+      A testbox script specific reporter needs to be implemented for the
+      testdriver framework.  The testbox script needs to forward the results to
+      the test manager, or alternatively the testdriver report can talk
+      directly to the TM.
+
+    - Implement the test manager side of the test result reporting.
+
+    - Extend the selftest with some setup that report all kinds of test
+      results.
+
+    - Implement script/whatever feeding builds to the test manager from the
+      tinderboxes.
+
+    - The toplevel test driver is a VBox thing that must be derived from the
+      base TestDriver class or maybe the VBox one.  It should move from
+      toptestdriver to testdriver and be renamed to vboxtltd or smth.
+
+    - Create a vbox testdriver that boots the t-xppro VM once and that's it.
+
+    - Create a selftest setup which tests booting t-xppro taking builds from
+      the tinderbox.
+
+
+Milestone #3
+------------
+
+The goal for this milestone is configuration and converting current testscases,
+the result will be the a minimal test deployment (4-5 new testboxes).
+
+Tasks in somewhat prioritized order:
+
+    - Implement testcase configuration.
+
+    - Implement testgroup configuration.
+
+    - Implement build source configuration.
+
+    - Implement scheduling group configuration.
+
+    - Implement global resource configuration.
+
+    - Re-visit the testbox configuration.
+
+    - Black listing of builds.
+
+    - Implement simple failure analysis and reporting.
+
+    - Implement the initial smoke tests modelled on the current smoke tests.
+
+    - Implement installation tests for Windows guests.
+
+    - Implement installation tests for Linux guests.
+
+    - Implement installation tests for Solaris guest.
+
+    - Implement installation tests for OS/2 guest.
+
+    - Set up a small test deployment.
+
+
+Further work
+------------
+
+After milestone #3 has been reached and issues found by the other team members
+have been addressed, we will probably go for full deployment.
+
+Beyond this point we will need to improve reporting and analysis.  There may be
+configuration aspects needing reporting as well.
+
+Once deployed, a golden rule will be that all new features shall have test
+coverage.  Preferrably, implemented by someone else and prior to the feature
+implementation.
+
+
+
+
+Discussion Logs
+===============
+
+2009-07-21,22,23 Various Discussions with Michal and/or Klaus
+-------------------------------------------------------------
+
+- Scheduling of tests requiring more than one testbox.
+- Scheduling of tests that cannot be executing concurrently on several machines
+  because of some global resource like an iSCSI target.
+- Manually create the test config permutations instead of having the test
+  manager create all possible ones and wasting time.
+- Distinguish between built types so we can run smoke tests on strick builds as
+  well as release ones.
+
+
+2009-07-20 Brief Discussion with Michal
+----------------------------------------
+
+- Installer for the testbox script to make bringing up a new testbox even
+  smoother.
+
+
+2009-07-16 Raw Input
+--------------------
+
+- test set. recursive collection of:
+    - hierachical subtest name (slash sep)
+    - test parameters / config
+    - bool fail/succ
+    - attributes (typed?)
+    - test time
+    - e.g. throughput
+    - subresults
+    - log
+    - screenshots,....
+
+- client package (zip) dl from server (maybe client caching)
+
+
+- thoughts on bits to do at once.
+    - We *really* need the basic bits ASAP.
+    - client -> support for test driver
+    - server -> controls configs
+    - cleanup on both sides
+
+
+2009-07-15 Raw Input
+--------------------
+
+- testing should start automatically
+- switching to branch too tedious
+- useful to be able to partition testboxes (run specific builds on some boxes, let an engineer have a few boxes for a while).
+- test specification needs to be more flexible (select tests, disable test, test scheduling (run certain tests nightly), ... )
+- testcase dependencies (blacklisting builds, run smoketests on box A before long tests on box B, ...)
+- more testing flexibility, more test than just install/moke. For instance unit tests, benchmarks, ...
+- presentation/analysis: graphs!, categorize bugs, columns reorganizing grouped by test (hierarchical), overviews, result for last day.
+- testcase specificion, variables (e.g. I/O-APIC, SMP, HWVIRT, SATA...) as sub-tests
+- interation with ILOM/...: reset systems
+- Changes needs LDAP authentication
+- historize all configuration  w/ name
+- ability to run testcase locally (provided the VDI/ISO/whatever extra requirements can be met).
+
+
+-----
+
+.. [1] no such footnote
+
+-----
+
+:Status: $Id: AutomaticTestingRevamp.txt $
+:Copyright: Copyright (C) 2010-2017 Oracle Corporation.
+