1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
|
INTRODUCTION & DESIGN
~~~~~~~~~~~~~~~~~~~~~
- Ocft is a testing tool for resource agents. Instead of the policy of HA,
it mainly concerns whether resource agents run correct locally. It can
design types of complicated environments to test the reliability of
resource agents. Precisely, it is to display whether resource agents can
return to correct or expected value. The advantage of the tool provides
us with competence to design conditions which can be recorded or reproduced.
Hence it is useful to debuggers.
* Components
** Test case generator (@sbindir@/ocft)
- Turning configuration files of test case to executable scripts.
** Configuration file (@datadir@/@PACKAGE_NAME@/ocft/configs/)
- Every configuration file directs only one resource agent and share the same
name with resource agent but contains more test cases.
** The testing script (/var/lib/@PACKAGE_NAME@/ocft/cases/)
- After the generator reads configuration files and generates many testing
scripts and the script is underway, the test begins.
* How to customize the environment of testing
- Ocft designs the running conditions through two ways, one is changing the
environment variables of resource agents (it is the interface left by OCF itself),
the other is modifying the OS environment of resource agents, such as altering
the permission of some key file or IP address of the machine.
* How to test
- Firstly, you need to sketch the all complex and uncommon environments against
a certain resource agent and keep in mind what consequences may be caused by
these uncommon environments.
Secondly, write the designed conditions and foreknown consequences into
configuration files, and then run the generator to translate the test case to
executable scripts.
Finally, you need running these scripts to observe the output and learn
the running status of each test case, which will compares the predicated result
with the actual one. If they differ, you will be able to find the bugs of the
resource agent.
- All of the output with test will be recorded into the log files, you can find them
in /var/lib/@PACKAGE_NAME@/ocft/cases/logs.
HOW TO WRITE CONFIGURATION FILE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- There are only 6 top level options that are all spelled by capital letters and "-".
Every top level option contains sub-options that they are initials.
* 'CONFIG' (top level option)
- Grammar: CONFIG
- The design in this option is global and influences every test case.
** 'Agent' (sub-option)
- Grammar: Agent AGENT_NAME
- The agent name you want to test.
** 'AgentRoot' (sub-option)
- Grammar: AgentRoot /usr/lib/ocf/resource.d/xxx
- A few agents will go to "linbit" or "pacemaker" directory, if you define this option,
ocft will use it to replace the default directory "heartbeat".
** 'InstallPackage' (sub-option)
- Grammar: InstallPackage package [package2 [...]]
- It will test whether the system have installed the service of the resource agent.
If not, it will download from Internet and have it installed automatically.
** 'HangTimeout' (sub-option)
- Grammar: HangTimeout secs
- If you alter some key options, some resource agents will get puzzled and stop,
which will influence the running of the following test case. Hence timeout setting is
needed, if the resource agent stops timeout, the scripts will kill this resource agent.
* 'VARIABLE' (top level option)
- Grammar:
VARIABLE
VAR1=value1
VAR2=value2
...
- Define the global variable here, the variables can be visited everywhere, they can be referenced
using $VAR_NAME. Note, the variables in VARIABLE are different from 'Env VAR1=value1', 'Env' can
affect the activity of agent, but the variables in VARIABLE just be shared with top level option.
* 'SETUP-AGENT' (top level option)
- Grammar:
SETUP-AGENT
bash scripts...
...
- Some of Agents may need to be initialized before testing, you can do it here with bash script.
* 'CLEANUP-AGENT' (top level option)
- Grammar:
CLEANUP-AGENT
bash scripts...
...
- If SETUP-AGENT set, usually you might be use this option do some cleaning work after test.
* 'CASE' & 'CASE-BLOCK' (top level option)
- Grammar: CASE "description" & CASE-BLOCK macro_name
- Usually, the conditions you designed are more than one and a few 'CASE "..."' will
appear in configuration file. It is worth noting that the following sub-options
have 2 spellings:
One is general, where shell affects the local environment; the other is special,
where each options added "@ipaddr". It can remotely execute shell codes. In other words,
it is to execute the shell codes from a remote host, which is meaningful when a resource
agent needs 2 hosts. This remote shell is not a remote execution only through "ssh", but
running a remote shell in the background while the test case is running. The remote shell
runs in the background till the end and saves the results during the process. That is to
say, you can alternatively carry out local and remote shell code segments.
The "CASE-BLOCK" option is a macro definer, the statements in "CASE-BLOCK" will be inserted
into "CASE" if you "Include" the "macro_name".
** 'Env' (sub-option)
- Grammar: Env VARIABLE=value
- It is to set up an environment variable of the resource agent. They usually appear to
be OCF_RESKEY_xxx. One point is to be noted is there is no blank by both sides of "=".
** 'Unenv' (sub-option)
- Grammar: Unenv VARIABLE [VARIABLE2 [...]]
- Remove the environment variable.
** 'Include' (sub-option)
- Grammar: Include macro_name
- It will be replaced by statements in 'macro_name', of course, you should define the
content of 'macro_name' with 'CASE-BLOCK' first.
** 'Bash' (sub-option)
- Grammar: Bash bash_codes
- This option is to set up the environment of OS, where you can insert BASH code to
customize the system randomly. Note, do not cause unrecoverable consequences to the
system.
** 'BashAtExit' (sub-option)
- Grammar: BashAtExit bash_codes
- This option is to recover the OS environment in order to run another test case
correctly. Of cause you can use 'Bash' option to recover it. However, if mistakes occur
in the process, the script will quit directly instead of running your recovery codes.
If it happens, you ought to use BashAtExit which can restore the system environment
before you quit.
** 'AgentRun' (sub-option)
- Grammar: AgentRun cmd [ret_value]
- This option is to run resource agent. "cmd" is the parameter of the resource agent,
such as "start, status, stop ...". The second parameter is optional. It will compare the
actual returned value with the expected value when the script has run recourse agent.
If differs, bugs will be found.
|
