diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-17 06:48:59 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-17 06:48:59 +0000 |
commit | d835b2cae8abc71958b69362162e6a70c3d7ef63 (patch) | |
tree | 81052e3d2ce3e1bcda085f73d925e9d6257dec15 /test/README.regression | |
parent | Initial commit. (diff) | |
download | crmsh-upstream.tar.xz crmsh-upstream.zip |
Adding upstream version 4.6.0.upstream/4.6.0upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'test/README.regression')
-rw-r--r-- | test/README.regression | 154 |
1 files changed, 154 insertions, 0 deletions
diff --git a/test/README.regression b/test/README.regression new file mode 100644 index 0000000..839ce59 --- /dev/null +++ b/test/README.regression @@ -0,0 +1,154 @@ +CRM shell regression tests + +* WARNING * WARNING * WARNING * WARNING * WARNING * WARNING * +* +* evaltest.sh uses eval to an extent you don't really want to +* know about. Beware. Beware twice. Any input from the testcases +* directory is considered to be trusted. So, think twice before +* devising your tests lest you kill your precious data. Got it? +* Good. +* +* Furthermore, we are deliberately small on testing the user +* input and no one should try to predict what is to happen on +* random input from the testcases. +* +* WARNING * WARNING * WARNING * WARNING * WARNING * WARNING * + +Manifest + + regression.sh: the top level program + evaltest.sh: the engine test engine + + crm-interface: interface to crm + descriptions: describe what we are about to do + defaults: the default settings for test commands + + testcases/: here are the testcases and filters + crmtestout/: here goes the output + +All volatile data lives in the testcases/ directory. + +NB: You should never ever need to edit regression.sh and +evaltest.sh. If you really have to, please talk to me and I will +try to fix it so that you do not have to. + +Please write new test cases. The more the merrier :) + +Usage + +The usage is: + + ./regression.sh ["prepare"] ["set:"<setname>|<testcase>] + +Test cases are collected in test sets. The default test set is +basicset and running regression.sh without arguments will do all +tests from that set. + +To show progress, for each test a '.' is printed. Once all tests +have been evaluated, the output is checked against the expect +file. If successful, "PASS" is printed, otherwise "FAIL". + +Specifying "prepare" will make regression.sh create expect +output files for the given set of tests or testcase. + +The script may start and stop lrmd and stonithd if they are not +running to support the crm ra set of commands. + +The following files may be generated: + + output/<testcase>.out: the output of the testcase + output/regression.out: the output of regression.sh + output/crm.out: the output of crm tools/lrmd/stonithd etc + +On success output from testcases is removed and regression.out is +empty. + +Driving the test cases yourself + +evaltest.sh accepts input from stdin, evaluates it immediately, +and prints results to stdout/stderr. One can perhaps get a better +feeling of what's actually going on by running it interactively. + +Test cases + +Tests are mainly written in the crm shell language with some simple +regression test directives (starting with '%' and +session/show/showxml). + +Special operations + +There are special operations with which it is possible to change +environment and do other useful things. All special ops start +with the '%' sign and may be followed by additional parameters. + +%setenv + change the environment variable; see defaults for the + set of global variables and resetvars() in evaltest.sh + +%stop + skip the rest of the tests + +%extcheck + feed the output of the next test case to the specified + external program/filter; the program should either reside in + testcases/ or be in the PATH, i.e. + + %extcheck cat + + simulates a null op :) + + see testcases/metadata for some examples + +%ext + run an external command provided in the rest of the line; for + example: + + %ext date + + would print the current time (not very useful for regression + testing). + +%repeat num + repeat the next test num times + there are several variables which are substituted in the test + lines, so that we can simulate a for loop: + + s/%t/$test_cnt/g + s/%l/$line/g + s/%j/$job_cnt/g + s/%i/$repeat_cnt/g + + for example, to add 10 resources: + + %repeat 10 + configure primitive p-%i ocf:pacemaker:Dummy + +Filters and other auxiliary files + +Some output is necessarily very volatile, such as time stamps. +It is possible to specify a filter for each testcase to get rid +of superfluous information. A filter is a filter in UNIX +sense, it takes input from stdin and prints results to stdout. + +There is a common filter called very inventively +testcases/common.filter which is applied to all test cases. + +Except files are a list of extended regular expressions fed to +egrep(1). That way one can filter out lines which are not +interesting. Again, the one applied to all is +testcases/common.excl. + +A test may need an arbitrary script executed before or after the +test itself in order to ascertain some state. The two scripts +have extensions .pre and .post respectively. Their output is sent +to /dev/null and the exit status ignored. + +Finally, the daemon log files may be filtered using log_filter. + +The full collection of auxiliary files follows: + + <TEST>.filter + <TEST>.excl + <TEST>.log_filter + <TEST>.pre + <TEST>.post |