1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
|
Testing Apache with the Perl Test Harness
Prerequisites
-------------
These two modules must first be installed;
- perl-ExtUtils-MakeMaker
- perl-Test
You'll need to install the CPAN modules listed in:
Apache-Test/lib/Bundle/ApacheTest.pm
All you have to do to install them all in one shot is:
perl -MCPAN -e 'install Bundle::ApacheTest'
Which are also available in one tarball here:
http://perl.apache.org/~dougm/httpd-test-bundle-0.02.tar.gz
Note: Crypt::SSLeay requires OpenSSL to be installed (only required
for t/TEST -ssl): http://www.openssl.org/
More accurate results may be obtained by using the same openssl command
line and libraries as consumed by APR-util and mod_ssl, due to X509
formatting behavior differences.
For an extensive documentation see
http://perl.apache.org/docs/general/testing/testing.html
or
http://svn.apache.org/viewvc/perl/modperl/docs/trunk/src/docs/general/testing/testing.pod
To run the tests for all Apache web server modules, some additional
CPAN modules will be required. If the tests don't work, make sure
that you have up to date versions of each of these perl modules:
- HTTP::DAV (DAV tests)
- DateTime (mod_include tests)
- Time::HiRes
- Protocol::HTTP2::Client and AnyEvent (mod_http2 tests)
- Test
- Test::Harness
- Crypt::SSLeay
- Net::SSLeay
- IO::Socket::SSL
- IO::Socket::IP
- IO::Select
- LWP::Protocol::https
Quick Start
-----------
If you don't care how it works and just want to run the tests, here's
how you go about doing that.
1. You need an installation of Apache. (1.3.x thru trunk)
2. Any DSOs you wish to use should be configured in that Apache's
httpd.conf (the test harness will pick this configuration up)
3. Setup:
perl Makefile.PL -apxs /path/to/apache/bin/apxs
4. Run the tests:
t/TEST
5. Evaluate test output.
Getting a little deeper
-----------------------
The test harness will run every .t file under the t/ directory. So
let's say you only want to run the tests for PHP. Do this:
t/TEST -httpd /path/to/apache/bin/httpd t/php
That will start the test server, run the .t tests under t/php and shut
down the test server. You can also control each of these steps.
This will start the test server:
t/TEST -httpd /path/to/apache/bin/httpd -start
This will run the PHP tests in the test environment:
t/TEST t/php
This will stop the test server:
t/TEST -stop
This will run the server under gdb (using -X):
t/TEST -d gdb
Note: At this point, you have a working test environment. You can
look in t/conf for the test server configuration files. These are
generated by the test harness. Once you have a working test
environment, you do not need to specify 'httpd' on the t/TEST command
line. For instance, to start the server up again, the command
t/TEST -start
would be sufficient.
Running Regression Tests
------------------------
For a full regression test, you should have all modules loaded. Build the server
with
configure --enable-modules=reallyall --enable-load-all-modules ...
among other things. Edit the generated httpd.conf and comment all mpm modules
that you do not want. Run "t/TEST -clean" again.
You will see some
skipped: cannot find module 'XXX'
as not all modules are in every apache release (but the tests run for all).
All in all, some >4k tests will run and the result needs to be: PASS
Trouble Shooting
----------------
If you have a "PASS" at the end of "t/TEST", congratulations! If not, this
sections gives some advise in order to find the cause. Feel free to expand
this to make life easier for others.
0. If your test startup hangs forever in "waiting for server to warm up", but
the test server is reachable under port 8529, you might be the victim of
ipv4/6 confusion. The default servername configured is "localhost" and
some operating systems define 127.0.0.1 *as well as* ::1 in /etc/hosts.
If the test server listens only on 0.0.0.0 it might not answer requests to
::1 and that causes the test startup to hang.
Solution: comment the ::1 line in /etc/hosts and see if that improves matters.
1. Run "t/TEST -clean" every time you change something in your Apache
configuration. The test suite picks up certain things from your installed
httpd.conf (such as LoadModule statements) and will not see your changes
unless you clean it.
2. Failures in proxy.t may originate from the fact that the test script cannot
open the specified port. This happens on some machines if you abort a test
run and the socket is not properly shut down. Check if the error goes
away after a reboot. (proxy.t tests are slow, so chances you interrupt tests
at that point are good.)
3. Failures in access.t may result from reverse lookups not working or giving
other answers than expected. In the cause 0 above, if the test client
connects via 127.0.0.1, a "Grant for localhost" might resolve to "::1"
and therefore will not match the access rules of the tests.
Solution: check that your servername is 'localhost' (which is
the default) and that it *always* resolves to 127.0.0.1.
4. If some ssl test cases fail, especially when t/ssl/proxy.t fails, the
reason can be mismatches between your installed SSL library and the one
httpd uses. The "openssl" binary found in your $PATH is used to create
binary setup files by t/TEST. If another version of openssl then tries
to read these from your Apache server process, it might fail.
Try the following:
> t/TEST -clean
> PATH=<bin dir of correct openssl>:$PATH t/TEST
If a lot of ssl tests fail, check in the error log for the presence of
a certificate validation error. If you find it, check the expiration date
of the TLS/SSL certificates used by the tests, they might be expired.
Running TEST -clean should delete the old ssl certificates, so they'll be
regenerated during the next run.
5. If you see failures in the modules/h2.t test cases, please notify the dev
mailing list with a description of your setup. These tests are quite young,
currently only valid in 2.4.x and later and interact with quite some other
modules as well as Openssl versions installed. Some tests require mod_ssl
so make sure to load it in the httpd conf.
6. Segmentation faults and core dumps occurring while executing the test suite
might indicate a real problem but always run again the tests after
a clean make install to avoid inconsistencies from old objects.
7. If you see error messages like "Parse errors: Bad plan.
You planned X tests but ran Y." it usually means that you are missing
a perl module or the tested httpd module depends on another one
not loaded in the httpd config.
8. If you see SSL certificate errors, remove t/conf/ssl/ca prior to
t/TEST -clean
9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking
/usr/bin/perl Makefile.PL -apxs ... will cause an older perl to be used.
Smoking Tests
-------------
Sometimes it's possible that the test is passing properly for the
first time, when it's run for the first time in the thread. But when
you run it again, the test might fail. It's important to run the
repetition smoke testing. For example to repeat the tests 5 times you
can run:
t/SMOKE -times=5
It's also possible that a test will pass when it's run after a
particular test, but if moved to run after a different state it may
fail. For this reason by default the tests run in random order.
Since it's important to be able to reproduce the problem with the
random testing, whenever -order=random is used, the used seed is
printed to STDERR. Which can be then fed into the future tests with:
via APACHE_TEST_SEED environment variable.
By adding the option -order=repeat, the tests will be run in
alphabetical order.
Combining these two important smoke testing techniques, one can run
tests with:
t/SMOKE -times=N -order=(repeat|random)
For example, to run the mod_rewrite tests 5 times, one would:
t/SMOKE -times=5 -verbose t/modules/rewrite.t
So the tests can be repeated N times, and run in the following three
modes:
- randomize all tests
- repeat the whole tests suite N times
For configuration options and default settings run:
t/SMOKE -help
For more information refer to the Apache::TestSmoke manpage.
Test Environment Configuration
------------------------------
The test server is configured with conf files like any normal Apache
server. The tricky part is those conf files are generated by the
harness just prior to starting the server. t/conf/httpd.conf is
generated by t/conf/httpd.conf.in. If that does not exist, the
harness will generate a working configuration and will include
LoadModule (and AddModule for Apache 1.3) directives from the
httpd.conf associated with the httpd binary you are using for testing.
If t/conf/extra.conf.in exists, t/conf/extra.conf will be generated
from that, and an Include directive for that file will be put in the
generated t/conf/httpd.conf. t/conf/apache_test_config.pm is
generated from the test configuration. It contains all the
information about the configuration of your test server. You can
access this information in test scripts by:
my $env = Apache::TestConfig->thaw;
Apache::TestConfig access apache_test_config.pm and returns a hash
reference with all the information. Look through
apache_test_config.pm, it's a lot of stuff. Once these conf files are
generated, you have a working test environment, and they must be
'cleaned' if you wish to make changes to them. To clean the
environment:
t/TEST -clean
(Now you will have to specify your httpd binary when starting back up
again.)
More Information
----------------
For more information on using the test harness and writing tests, see
the README in Apache-Test and the examples in Apache-Test/t.
The test harness was originally written by Doug MacEachern and is
discussed on the httpd dev mailing list (dev@httpd.apache.org).
It is also included in modperl-2.0 source along with tests for
modperl-2.0.
|
