diff options
Diffstat (limited to '')
| -rw-r--r-- | test/README | 3 | ||||
| -rw-r--r-- | test/README.ci | 163 | ||||
| -rw-r--r-- | test/README.pytest | 139 |
3 files changed, 305 insertions, 0 deletions
diff --git a/test/README b/test/README new file mode 100644 index 0000000..9f8be50 --- /dev/null +++ b/test/README @@ -0,0 +1,3 @@ +This directory contains useful test code for testing various bits +of Apache functionality. This stuff is for the developers only, +so we might remove it on public releases. diff --git a/test/README.ci b/test/README.ci new file mode 100644 index 0000000..43600b0 --- /dev/null +++ b/test/README.ci @@ -0,0 +1,163 @@ + +Variables +--------- + +The CI scripts use the following environment variables: + +* APR_VERSION - if set, APR of this version is built and installed in + $HOME/root/apr-$APR_VERSION - a value of "trunk" means trunk is + used, "*.x" means a branch, otherwise a tagged version is implied. + +* APR_CONFIG - arguments to pass when running APR's configure script + if APR_VERSION is set + +* APU_VERSION - if set, APR-util of this version is built and + installed in $HOME/root/apr-util-$APU_VERSION - a value of "*.x" + means a branch, otherwise a tagged version is implied. (Since there + is no "trunk" for apr-util, that value cannot be used here.) + +* APU_CONFIG - arguments to pass when running APR-util's configure + script if APU_VERSION is set + +* CONFIG - arguments to pass to httpd's configure script. + +* BUILDCONFIG - arguments to pass when running httpd's ./buildconf script + +* MFLAGS - arguments to pass when running "make" for httpd. + +* SKIP_TESTING - if set, the Perl test framework is not run for the + build. + +* TEST_UBSAN - set for job using UBSan ("Undefined Behaviour Sanitizer") + +* TEST_MALLOC - set for job using enhanced malloc debugging. + +* TEST_INSTALL - set for job testing "make install" + +* TEST_VPATH - set for job testing srcdir!=builddir + +* TEST_LDAP - set for job with slapd, running LDAP tests + +* TEST_SSL - set for job with SSL/TLS testing variants + +* TESTS - a list of Perl framework tests to run + +* TEST_ARGS - arguments to pass to ./t/TEST in the Perl test framework + +* CLEAR_CACHE - if set, the cached $HOME/root is removed before each build + +Caching -- NOTE, BROKEN IN GITHUB ACTIONS -- +------- + +Perl modules installed in $HOME/perl5 are cached. + +Anything installed into the $HOME/root directory is cached - notably, +versions of APR/APR-util are installed here and cached across httpd +build jobs without needing to be rebuilt every time. + +The cached installs of APR/APR-util are refreshed if the +last-changed-revision of the build is stale. + +If APR_VERSION and APU_VERSION are both set to 1.x versions, then +CLEAR_CACHE should also be set to disable APR* caching. APR-util can +only be rebuilt if an APR checkout is present, so a APR-util cannot be +built from source alone. (i.e. the scripts do not handle the case of +cached, fresh APR plus a cached but stale APR-util) + +Travis to Github Actions Migration TODO +--------------------------------------- + +* better path filtering so e.g. CHANGES changes don't trigger CI +* support branch conditionals again (some tests are 2.4.x only, some trunk only) +* make caching work properly for APR + CPAN modules + - this is using the wrong model at the moment + - the cache key needs to be based off (source code, job configuration) + - rather than done on the fly in test/travis_before_linux.sh + - pebble + Rustls builds should also be cached +* turn on failure notifications? +* test across different Ubuntu versions again + - and test against OpenSSL 1.x since we're now ONLY building against 3.x +* update the docs below for testing from PRs/feature branches +* introduce some job ordering rather than having a flat/concurrent + set, if the default "./configure && make && test" works *then* start + jobs doing 200 different variations on ./configure --enable-XXX + i.e. stop burning CPU time for a typo which breaks every job + +TODO list +--------- + +* non-x86 builds +* MacOS build +* Windows build +* clang-on-Linux build +* Use containers for non-Ubuntu-based Linux testing + +Known Failures +-------------- + +Some known failures: + +* prefork, and more rarely non-prefork testing sometimes catches child + segfaults under pool-debug from assert()s in (e.g.) + __pthread_tpp_change_priority where one child is destroying threads + which another is waiting for, or iterating through via + apr_pool_walk_tree(). + + See dev@httpd threads: + msg <5f4abde1b5789_13fde2ecacb40795a1@travis-tasks-5b566d48fc-drkb9.mail> + msg <73060f7b-df7f-ad3c-a9fa-dd666a59b31e@kippdata.de> and + https://bz.apache.org/bugzilla/show_bug.cgi?id=63098 + https://bz.apache.org/bugzilla/show_bug.cgi?id=46185 + + Not clear if there is a real bug here which can be reproduced + outside of pool-debug. + +Testing from a Feature Branch [*** N/A FOR GITHUB ACTIONS ***] +----------------------------- + +An SVN branch off trunk should be mirrored to github, and will be +tested in the same way that trunk is in CI, so this workflow is +available for those familiar with using Subversion and the standard +ASF/httpd repository layout. + +Tested branches are listed at: https://travis-ci.com/github/apache/httpd/branches + +Travis will also run the tests for a PR filed against the httpd Github +repository at https://github.com/apache/httpd or from a fork of this +repository if enabled for the Travis user. + +A workflow to enable testing would be as follows, substituting +$USERNAME for your github username: + + $ git clone https://github.com/apache/httpd + $ cd httpd + $ git remote add $USERNAME git@github.com:$USERNAME/httpd.git + $ git checkout -b my-feature origin/trunk + ... do some work ... + $ git commit ... + $ git push -u $USERNAME my-feature:my-feature + +To enable testing for a fork, visit the settings page at +https://travis-ci.com/$USERNAME/httpd/settings - you may need to sync +your account via https://travis-ci.com/account/repositories for a +freshly created fork. + +To create a Pull Request, go to the URL produced in the "git push" +command output when pushing to your fork, which is something like: +https://github.com/apache/httpd/compare/trunk...$USERNAME:trunk + +Once a PR has been created, travis will run the tests and link the +results from a PR comment. All tested PRs are listed here: +https://travis-ci.com/github/apache/httpd/pull_requests + +To merge from github back to SVN trunk, create a patch from e.g.: + + $ git diff origin/trunk..my-feature + +and then apply it in SVN. To rebase a feature once trunk has +diverged, from a feature branch run: + + $ git pull + $ git rebase -i origin/trunk + +and follow the standard rebase steps. diff --git a/test/README.pytest b/test/README.pytest new file mode 100644 index 0000000..474030b --- /dev/null +++ b/test/README.pytest @@ -0,0 +1,139 @@ +Apache httpd pytest suite +========================= +Using pytest (<https://docs.pytest.org/en/6.2.x/>) and a Python >= 3.8 +for a more flexible testing of Apache httpd. + +Install +------- +If not already installed, you will need to install 'pytest' and 'OpenSSL' for +python: +> apt install python3-pip +> pip install -U pytest +> pip install -U pyopenssl + +And for 'h2load': +> apt install nghttp2-client + + +Usage +----- +In your httpd source checkout, do: + +> make install +> pytest + +and all tests defined run on the installed executable and modules. +> pytest test/modules/core +runs all the core tests. You can run tests also by name selection: +> pytest -k test_core +runs all test cases starting with 'test_core'. Similar: +> pytest -k test_core_001_04 +runs the test cases starting with that. + +Test output gets more verbose, by adding one or several '-v'. This +also raises the error log level of the tested modules. +> pytest -vvv -k test_h2_004_01 +run the specific test with mod_http2 at log level TRACE2. + +By default, test cases will configure httpd with mpm_event. You +can change that with the invocation: +> MPM=worker pytest test/modules/http2 + +Some tests rely on additional tools installed in your environment +and will 'skip' if those are not present. In a non-verbose run, +these will appear as 's' in the output. If you run pytest more +verbose, the skipped test cases will mention a reason for why +they are disabled. + +For example, most tests in test/modules/md require an installation +of 'pebble', an ACME test server, and look for it in $PATH. + + +Workings +-------- +All tests start httpd on their own and try hard to shut it down +afterwards. You can abort tests with ^C and they clean up. + +httpd is started/stopped repeatedly in testing as configurations +for individual test cases are changed. This is a main difference to +the Perl test framework which starts the server with all possible +combinations configured that are needed by tests. + +In test/gen/apache a server root is created with config, logs and htdocs +directories. test/gen/apache/logs/error_log will be the log. +Configs start in test/gen/apache/conf/httpd.conf. modules.conf is +dynamically created for the list of modules that a test suite needs. + +Test cases write their specific setup in test.conf and reload/restart +the httpd process. This makes for a small configuration in a test case. + + +Development +----------- + +Adding a test in an existing file is done by adding a method. Its name +must start with 'test_' and the common practice is to have the name +of the test suite there as well. All http2 tests start with 'test_h2_'. + +Following this can be any characters. If you make test cases of a +certain feature with a common prefix, it is easier to invoke just +them using the '-k' selector on the command line. + +You can also add just a new file to a test suite, if you do a new +range of test cases that do not fit anywhere else. A very simple +one is found in test/modules/http2/test_001_httpd_alive.py. + +There is a python class defined with 2 methods. One is the test +method itself and the other one ' is + + @pytest.fixture(autouse=True, scope='class') + def _class_scope(self, env): + code + +is marked as a pytest 'fixture'. This is some pytest magic. +'autouse=True' means that this fixture is run, even though +no test case uses it directly. scope='class' means that it +is run once for all test cases in this class. + +As you see, this fixture gets a parameter named 'env' and +that is the name of another pytest fixture, defined in the +file 'conftest.py' in the same directory. + + @pytest.fixture(scope="package") + def env(pytestconfig) -> H2TestEnv: + code + +This one runs one time per 'package', meaning for all test +cases in this directory. And it gets the 'pytestconfig' +as parameter which is a standard pytest fixture. + +So, when you run 'pytest -k test_h2_004', pytest will look +at _all_ test cases defined and collect those with that +prefix. For each directory with test cases found, it will +process the 'conftest.py', boot-strapping the 'env' fixture, +and the process the files with active test cases. + +As a result, if you invoke just a single test case, only +the fixtures needed for that test case are created. This +gives good turn around times when debugging a test case. + +If you want to add a new test suite, create a new directory. +Add the files '__init__.py', 'conftest.py' and a first +'test_suitename_something.py'. test/modules/core is the +simplest example. 'test/modules/http2' shows how you load +other modules. 'test/modules/md' checks and starts external +processes (an ACME test server). + + +Infrastructure +-------------- +The test cases rely on the classes provided in 'test/pyhttpd' +for common code in managing a httpd test instance and do +provide some base setups: +- a small set of virtual hosts with some files +- access to paths and definitions (env fixture) +- start/stop httpd and inspect the error log +- run clients like curl and nghttp +- create certificates for your hosts and make curl use + the root certs (so no --insecure calls by curl). + |