CRM shell regression tests

* WARNING * WARNING * WARNING * WARNING * WARNING * WARNING *
* 
* evaltest.sh uses eval to an extent you don't really want to
* know about. Beware. Beware twice. Any input from the testcases
* directory is considered to be trusted. So, think twice before
* devising your tests lest you kill your precious data. Got it?
* Good.
*
* Furthermore, we are deliberately small on testing the user
* input and no one should try to predict what is to happen on
* random input from the testcases.
* 
* WARNING * WARNING * WARNING * WARNING * WARNING * WARNING *

Manifest

	regression.sh: the top level program
	evaltest.sh: the engine test engine

	crm-interface: interface to crm
	descriptions: describe what we are about to do
	defaults: the default settings for test commands

	testcases/: here are the testcases and filters
	crmtestout/: here goes the output

All volatile data lives in the testcases/ directory.

NB: You should never ever need to edit regression.sh and
evaltest.sh. If you really have to, please talk to me and I will
try to fix it so that you do not have to.

Please write new test cases. The more the merrier :)

Usage

The usage is:

	./regression.sh ["prepare"] ["set:"<setname>|<testcase>]

Test cases are collected in test sets. The default test set is
basicset and running regression.sh without arguments will do all
tests from that set.

To show progress, for each test a '.' is printed. Once all tests
have been evaluated, the output is checked against the expect
file. If successful, "PASS" is printed, otherwise "FAIL".

Specifying "prepare" will make regression.sh create expect
output files for the given set of tests or testcase.

The script may start and stop lrmd and stonithd if they are not
running to support the crm ra set of commands.

The following files may be generated:

	output/<testcase>.out: the output of the testcase
	output/regression.out: the output of regression.sh
	output/crm.out: the output of crm tools/lrmd/stonithd etc

On success output from testcases is removed and regression.out is
empty.

Driving the test cases yourself

evaltest.sh accepts input from stdin, evaluates it immediately,
and prints results to stdout/stderr. One can perhaps get a better
feeling of what's actually going on by running it interactively.

Test cases

Tests are mainly written in the crm shell language with some simple
regression test directives (starting with '%' and
session/show/showxml).

Special operations

There are special operations with which it is possible to change
environment and do other useful things. All special ops start
with the '%' sign and may be followed by additional parameters.

%setenv
	change the environment variable; see defaults for the
	set of global variables and resetvars() in evaltest.sh

%stop
	skip the rest of the tests

%extcheck
	feed the output of the next test case to the specified
	external program/filter; the program should either reside in
	testcases/ or be in the PATH, i.e.
	
	%extcheck cat
	
	simulates a null op :)

	see testcases/metadata for some examples

%ext
	run an external command provided in the rest of the line; for
	example:

	%ext date

	would print the current time (not very useful for regression
	testing).

%repeat num
	repeat the next test num times
	there are several variables which are substituted in the test
	lines, so that we can simulate a for loop:

	s/%t/$test_cnt/g
	s/%l/$line/g
	s/%j/$job_cnt/g
	s/%i/$repeat_cnt/g

	for example, to add 10 resources:

	%repeat 10
	configure primitive p-%i ocf:pacemaker:Dummy

Filters and other auxiliary files

Some output is necessarily very volatile, such as time stamps.
It is possible to specify a filter for each testcase to get rid
of superfluous information. A filter is a filter in UNIX
sense, it takes input from stdin and prints results to stdout.

There is a common filter called very inventively
testcases/common.filter which is applied to all test cases.

Except files are a list of extended regular expressions fed to
egrep(1). That way one can filter out lines which are not
interesting. Again, the one applied to all is
testcases/common.excl.

A test may need an arbitrary script executed before or after the
test itself in order to ascertain some state. The two scripts
have extensions .pre and .post respectively. Their output is sent
to /dev/null and the exit status ignored. 

Finally, the daemon log files may be filtered using log_filter.

The full collection of auxiliary files follows:

	<TEST>.filter
	<TEST>.excl
	<TEST>.log_filter
	<TEST>.pre
	<TEST>.post