243 lines
9.8 KiB
Text
243 lines
9.8 KiB
Text
|
|
Testing Apache with the Perl Test Harness
|
|
Prerequisites
|
|
-------------
|
|
These two modules must first be installed;
|
|
|
|
- perl-ExtUtils-MakeMaker
|
|
- perl-Test
|
|
|
|
You'll need to install the CPAN modules listed in:
|
|
Apache-Test/lib/Bundle/ApacheTest.pm
|
|
All you have to do to install them all in one shot is:
|
|
perl -MCPAN -e 'install Bundle::ApacheTest'
|
|
|
|
Which are also available in one tarball here:
|
|
http://perl.apache.org/~dougm/httpd-test-bundle-0.02.tar.gz
|
|
|
|
Note: Crypt::SSLeay requires OpenSSL to be installed (only required
|
|
for t/TEST -ssl): http://www.openssl.org/
|
|
More accurate results may be obtained by using the same openssl command
|
|
line and libraries as consumed by APR-util and mod_ssl, due to X509
|
|
formatting behavior differences.
|
|
|
|
For an extensive documentation see
|
|
http://perl.apache.org/docs/general/testing/testing.html
|
|
or
|
|
http://svn.apache.org/viewvc/perl/modperl/docs/trunk/src/docs/general/testing/testing.pod
|
|
|
|
To run the tests for all Apache web server modules, some additional
|
|
CPAN modules will be required. If the tests don't work, make sure
|
|
that you have up to date versions of each of these perl modules:
|
|
|
|
```
|
|
cpan App::cpanminus
|
|
cpanm Bundle::ApacheTest \
|
|
HTTP::DAV DateTime Time::HiRes \
|
|
Test::Harness Crypt::SSLeay Net::SSLeay IO::Socket::SSL \
|
|
IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \
|
|
AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI
|
|
```
|
|
|
|
|
|
Quick Start
|
|
-----------
|
|
|
|
If you don't care how it works and just want to run the tests, here's
|
|
how you go about doing that.
|
|
|
|
1. You need an installation of Apache. (1.3.x thru trunk)
|
|
2. Any DSOs you wish to use should be configured in that Apache's
|
|
httpd.conf (the test harness will pick this configuration up)
|
|
3. Setup:
|
|
perl Makefile.PL -apxs /path/to/apache/bin/apxs
|
|
4. Run the tests:
|
|
t/TEST
|
|
5. Evaluate test output.
|
|
|
|
Getting a little deeper
|
|
-----------------------
|
|
|
|
The test harness will run every .t file under the t/ directory. So
|
|
let's say you only want to run the tests for PHP. Do this:
|
|
t/TEST -httpd /path/to/apache/bin/httpd t/php
|
|
|
|
That will start the test server, run the .t tests under t/php and shut
|
|
down the test server. You can also control each of these steps.
|
|
|
|
This will start the test server:
|
|
t/TEST -httpd /path/to/apache/bin/httpd -start
|
|
|
|
This will run the PHP tests in the test environment:
|
|
t/TEST t/php
|
|
|
|
This will stop the test server:
|
|
t/TEST -stop
|
|
|
|
This will run the server under gdb (using -X):
|
|
t/TEST -d gdb
|
|
|
|
Note: At this point, you have a working test environment. You can
|
|
look in t/conf for the test server configuration files. These are
|
|
generated by the test harness. Once you have a working test
|
|
environment, you do not need to specify 'httpd' on the t/TEST command
|
|
line. For instance, to start the server up again, the command
|
|
t/TEST -start
|
|
would be sufficient.
|
|
|
|
Running Regression Tests
|
|
------------------------
|
|
For a full regression test, you should have all modules loaded. Build the server
|
|
with
|
|
configure --enable-modules=reallyall --enable-load-all-modules ...
|
|
among other things. Edit the generated httpd.conf and comment all mpm modules
|
|
that you do not want. Run "t/TEST -clean" again.
|
|
|
|
You will see some
|
|
skipped: cannot find module 'XXX'
|
|
as not all modules are in every apache release (but the tests run for all).
|
|
|
|
All in all, some >4k tests will run and the result needs to be: PASS
|
|
|
|
Trouble Shooting
|
|
----------------
|
|
If you have a "PASS" at the end of "t/TEST", congratulations! If not, this
|
|
sections gives some advise in order to find the cause. Feel free to expand
|
|
this to make life easier for others.
|
|
|
|
0. If your test startup hangs forever in "waiting for server to warm up", but
|
|
the test server is reachable under port 8529, you might be the victim of
|
|
ipv4/6 confusion. The default servername configured is "localhost" and
|
|
some operating systems define 127.0.0.1 *as well as* ::1 in /etc/hosts.
|
|
If the test server listens only on 0.0.0.0 it might not answer requests to
|
|
::1 and that causes the test startup to hang.
|
|
Solution: comment the ::1 line in /etc/hosts and see if that improves matters.
|
|
1. Run "t/TEST -clean" every time you change something in your Apache
|
|
configuration. The test suite picks up certain things from your installed
|
|
httpd.conf (such as LoadModule statements) and will not see your changes
|
|
unless you clean it.
|
|
2. Failures in proxy.t may originate from the fact that the test script cannot
|
|
open the specified port. This happens on some machines if you abort a test
|
|
run and the socket is not properly shut down. Check if the error goes
|
|
away after a reboot. (proxy.t tests are slow, so chances you interrupt tests
|
|
at that point are good.)
|
|
3. Failures in access.t may result from reverse lookups not working or giving
|
|
other answers than expected. In the cause 0 above, if the test client
|
|
connects via 127.0.0.1, a "Grant for localhost" might resolve to "::1"
|
|
and therefore will not match the access rules of the tests.
|
|
Solution: check that your servername is 'localhost' (which is
|
|
the default) and that it *always* resolves to 127.0.0.1.
|
|
4. If some ssl test cases fail, especially when t/ssl/proxy.t fails, the
|
|
reason can be mismatches between your installed SSL library and the one
|
|
httpd uses. The "openssl" binary found in your $PATH is used to create
|
|
binary setup files by t/TEST. If another version of openssl then tries
|
|
to read these from your Apache server process, it might fail.
|
|
Try the following:
|
|
> t/TEST -clean
|
|
> PATH=<bin dir of correct openssl>:$PATH t/TEST
|
|
If a lot of ssl tests fail, check in the error log for the presence of
|
|
a certificate validation error. If you find it, check the expiration date
|
|
of the TLS/SSL certificates used by the tests, they might be expired.
|
|
Running TEST -clean should delete the old ssl certificates, so they'll be
|
|
regenerated during the next run.
|
|
5. If you see failures in the modules/h2.t test cases, please notify the dev
|
|
mailing list with a description of your setup. These tests are quite young,
|
|
currently only valid in 2.4.x and later and interact with quite some other
|
|
modules as well as Openssl versions installed. Some tests require mod_ssl
|
|
so make sure to load it in the httpd conf.
|
|
6. Segmentation faults and core dumps occurring while executing the test suite
|
|
might indicate a real problem but always run again the tests after
|
|
a clean make install to avoid inconsistencies from old objects.
|
|
7. If you see error messages like "Parse errors: Bad plan.
|
|
You planned X tests but ran Y." it usually means that you are missing
|
|
a perl module or the tested httpd module depends on another one
|
|
not loaded in the httpd config.
|
|
8. If you see SSL certificate errors, remove t/conf/ssl/ca prior to
|
|
t/TEST -clean
|
|
9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking
|
|
/usr/bin/perl Makefile.PL -apxs ... will cause an older perl to be used.
|
|
|
|
Smoking Tests
|
|
-------------
|
|
|
|
Sometimes it's possible that the test is passing properly for the
|
|
first time, when it's run for the first time in the thread. But when
|
|
you run it again, the test might fail. It's important to run the
|
|
repetition smoke testing. For example to repeat the tests 5 times you
|
|
can run:
|
|
|
|
t/SMOKE -times=5
|
|
|
|
It's also possible that a test will pass when it's run after a
|
|
particular test, but if moved to run after a different state it may
|
|
fail. For this reason by default the tests run in random order.
|
|
|
|
Since it's important to be able to reproduce the problem with the
|
|
random testing, whenever -order=random is used, the used seed is
|
|
printed to STDERR. Which can be then fed into the future tests with:
|
|
via APACHE_TEST_SEED environment variable.
|
|
|
|
By adding the option -order=repeat, the tests will be run in
|
|
alphabetical order.
|
|
|
|
Combining these two important smoke testing techniques, one can run
|
|
tests with:
|
|
|
|
t/SMOKE -times=N -order=(repeat|random)
|
|
|
|
For example, to run the mod_rewrite tests 5 times, one would:
|
|
|
|
t/SMOKE -times=5 -verbose t/modules/rewrite.t
|
|
|
|
So the tests can be repeated N times, and run in the following three
|
|
modes:
|
|
|
|
- randomize all tests
|
|
- repeat the whole tests suite N times
|
|
|
|
For configuration options and default settings run:
|
|
|
|
t/SMOKE -help
|
|
|
|
For more information refer to the Apache::TestSmoke manpage.
|
|
|
|
|
|
Test Environment Configuration
|
|
------------------------------
|
|
|
|
The test server is configured with conf files like any normal Apache
|
|
server. The tricky part is those conf files are generated by the
|
|
harness just prior to starting the server. t/conf/httpd.conf is
|
|
generated by t/conf/httpd.conf.in. If that does not exist, the
|
|
harness will generate a working configuration and will include
|
|
LoadModule (and AddModule for Apache 1.3) directives from the
|
|
httpd.conf associated with the httpd binary you are using for testing.
|
|
If t/conf/extra.conf.in exists, t/conf/extra.conf will be generated
|
|
from that, and an Include directive for that file will be put in the
|
|
generated t/conf/httpd.conf. t/conf/apache_test_config.pm is
|
|
generated from the test configuration. It contains all the
|
|
information about the configuration of your test server. You can
|
|
access this information in test scripts by:
|
|
my $env = Apache::TestConfig->thaw;
|
|
Apache::TestConfig access apache_test_config.pm and returns a hash
|
|
reference with all the information. Look through
|
|
apache_test_config.pm, it's a lot of stuff. Once these conf files are
|
|
generated, you have a working test environment, and they must be
|
|
'cleaned' if you wish to make changes to them. To clean the
|
|
environment:
|
|
t/TEST -clean
|
|
(Now you will have to specify your httpd binary when starting back up
|
|
again.)
|
|
|
|
|
|
More Information
|
|
----------------
|
|
|
|
For more information on using the test harness and writing tests, see
|
|
the README in Apache-Test and the examples in Apache-Test/t.
|
|
|
|
The test harness was originally written by Doug MacEachern and is
|
|
discussed on the httpd dev mailing list (dev@httpd.apache.org).
|
|
|
|
It is also included in modperl-2.0 source along with tests for
|
|
modperl-2.0.
|