From 293913568e6a7a86fd1479e1cff8e2ecb58d6568 Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sat, 13 Apr 2024 15:44:03 +0200 Subject: Adding upstream version 16.2. Signed-off-by: Daniel Baumann --- doc/src/sgml/regress.sgml | 917 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 917 insertions(+) create mode 100644 doc/src/sgml/regress.sgml (limited to 'doc/src/sgml/regress.sgml') diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml new file mode 100644 index 0000000..69f627d --- /dev/null +++ b/doc/src/sgml/regress.sgml @@ -0,0 +1,917 @@ + + + + Regression Tests + + + regression tests + + + + test + + + + The regression tests are a comprehensive set of tests for the SQL + implementation in PostgreSQL. They test + standard SQL operations as well as the extended capabilities of + PostgreSQL. + + + + Running the Tests + + + The regression tests can be run against an already installed and + running server, or using a temporary installation within the build + tree. Furthermore, there is a parallel and a + sequential mode for running the tests. The + sequential method runs each test script alone, while the + parallel method starts up multiple server processes to run groups + of tests in parallel. Parallel testing adds confidence that + interprocess communication and locking are working correctly. + Some tests may run sequentially even in the parallel + mode in case this is required by the test. + + + + Running the Tests Against a Temporary Installation + + + To run the parallel regression tests after building but before installation, + type: + +make check + + in the top-level directory. (Or you can change to + src/test/regress and run the command there.) + Tests which are run in parallel are prefixed with +, and + tests which run sequentially are prefixed with -. + At the end you should see something like: + + +# All 213 tests passed. + + + or otherwise a note about which tests failed. See below before assuming that a + failure represents a serious problem. + + + + Because this test method runs a temporary server, it will not work + if you did the build as the root user, since the server will not start as + root. Recommended procedure is not to do the build as root, or else to + perform testing after completing the installation. + + + + If you have configured PostgreSQL to install + into a location where an older PostgreSQL + installation already exists, and you perform make check + before installing the new version, you might find that the tests fail + because the new programs try to use the already-installed shared + libraries. (Typical symptoms are complaints about undefined symbols.) + If you wish to run the tests before overwriting the old installation, + you'll need to build with configure --disable-rpath. + It is not recommended that you use this option for the final installation, + however. + + + + The parallel regression test starts quite a few processes under your + user ID. Presently, the maximum concurrency is twenty parallel test + scripts, which means forty processes: there's a server process and a + psql process for each test script. + So if your system enforces a per-user limit on the number of processes, + make sure this limit is at least fifty or so, else you might get + random-seeming failures in the parallel test. If you are not in + a position to raise the limit, you can cut down the degree of parallelism + by setting the MAX_CONNECTIONS parameter. For example: + +make MAX_CONNECTIONS=10 check + + runs no more than ten tests concurrently. + + + + + Running the Tests Against an Existing Installation + + + To run the tests after installation (see ), + initialize a data directory and start the + server as explained in , then type: + +make installcheck + +or for a parallel test: + +make installcheck-parallel + + The tests will expect to contact the server at the local host and the + default port number, unless directed otherwise by PGHOST and + PGPORT environment variables. The tests will be run in a + database named regression; any existing database by this name + will be dropped. + + + + The tests will also transiently create some cluster-wide objects, such as + roles, tablespaces, and subscriptions. These objects will have names + beginning with regress_. Beware of + using installcheck mode with an installation that has + any actual global objects named that way. + + + + + Additional Test Suites + + + The make check and make installcheck commands + run only the core regression tests, which test built-in + functionality of the PostgreSQL server. The source + distribution contains many additional test suites, most of them having + to do with add-on functionality such as optional procedural languages. + + + + To run all test suites applicable to the modules that have been selected + to be built, including the core tests, type one of these commands at the + top of the build tree: + +make check-world +make installcheck-world + + These commands run the tests using temporary servers or an + already-installed server, respectively, just as previously explained + for make check and make installcheck. Other + considerations are the same as previously explained for each method. + Note that make check-world builds a separate instance + (temporary data directory) for each tested module, so it requires more + time and disk space than make installcheck-world. + + + + On a modern machine with multiple CPU cores and no tight operating-system + limits, you can make things go substantially faster with parallelism. + The recipe that most PostgreSQL developers actually use for running all + tests is something like + +make check-world -j8 >/dev/null + + with a limit near to or a bit more than the number + of available cores. Discarding stdout + eliminates chatter that's not interesting when you just want to verify + success. (In case of failure, the stderr + messages are usually enough to determine where to look closer.) + + + + Alternatively, you can run individual test suites by typing + make check or make installcheck in the appropriate + subdirectory of the build tree. Keep in mind that make + installcheck assumes you've installed the relevant module(s), not + only the core server. + + + + The additional tests that can be invoked this way include: + + + + + + Regression tests for optional procedural languages. + These are located under src/pl. + + + + + Regression tests for contrib modules, + located under contrib. + Not all contrib modules have tests. + + + + + Regression tests for the interface libraries, + located in src/interfaces/libpq/test and + src/interfaces/ecpg/test. + + + + + Tests for core-supported authentication methods, + located in src/test/authentication. + (See below for additional authentication-related tests.) + + + + + Tests stressing behavior of concurrent sessions, + located in src/test/isolation. + + + + + Tests for crash recovery and physical replication, + located in src/test/recovery. + + + + + Tests for logical replication, + located in src/test/subscription. + + + + + Tests of client programs, located under src/bin. + + + + + + When using installcheck mode, these tests will create + and destroy test databases whose names + include regression, for + example pl_regression + or contrib_regression. Beware of + using installcheck mode with an installation that has + any non-test databases named that way. + + + + Some of these auxiliary test suites use the TAP infrastructure explained + in . + The TAP-based tests are run only when PostgreSQL was configured with the + option . This is recommended for + development, but can be omitted if there is no suitable Perl installation. + + + + Some test suites are not run by default, either because they are not secure + to run on a multiuser system, because they require special software or + because they are resource intensive. You can decide which test suites to + run additionally by setting the make or environment + variable PG_TEST_EXTRA to a whitespace-separated list, + for example: + +make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance' + + The following values are currently supported: + + + kerberos + + + Runs the test suite under src/test/kerberos. This + requires an MIT Kerberos installation and opens TCP/IP listen sockets. + + + + + + ldap + + + Runs the test suite under src/test/ldap. This + requires an OpenLDAP installation and opens + TCP/IP listen sockets. + + + + + + ssl + + + Runs the test suite under src/test/ssl. This opens TCP/IP listen sockets. + + + + + + load_balance + + + Runs the test src/interfaces/libpq/t/004_load_balance_dns.pl. + This requires editing the system hosts file and + opens TCP/IP listen sockets. + + + + + + wal_consistency_checking + + + Uses wal_consistency_checking=all while running + certain tests under src/test/recovery. Not + enabled by default because it is resource intensive. + + + + + + Tests for features that are not supported by the current build + configuration are not run even if they are mentioned in + PG_TEST_EXTRA. + + + + In addition, there are tests in src/test/modules + which will be run by make check-world but not + by make installcheck-world. This is because they + install non-production extensions or have other side-effects that are + considered undesirable for a production installation. You can + use make install and make + installcheck in one of those subdirectories if you wish, + but it's not recommended to do so with a non-test server. + + + + + Locale and Encoding + + + By default, tests using a temporary installation use the + locale defined in the current environment and the corresponding + database encoding as determined by initdb. It + can be useful to test different locales by setting the appropriate + environment variables, for example: + +make check LANG=C +make check LC_COLLATE=en_US.utf8 LC_CTYPE=fr_CA.utf8 + + For implementation reasons, setting LC_ALL does not + work for this purpose; all the other locale-related environment + variables do work. + + + + When testing against an existing installation, the locale is + determined by the existing database cluster and cannot be set + separately for the test run. + + + + You can also choose the database encoding explicitly by setting + the variable ENCODING, for example: + +make check LANG=C ENCODING=EUC_JP + + Setting the database encoding this way typically only makes sense + if the locale is C; otherwise the encoding is chosen automatically + from the locale, and specifying an encoding that does not match + the locale will result in an error. + + + + The database encoding can be set for tests against either a temporary or + an existing installation, though in the latter case it must be + compatible with the installation's locale. + + + + + Custom Server Settings + + + Custom server settings to use when running a regression test suite can be + set in the PGOPTIONS environment variable (for settings + that allow this): + +make check PGOPTIONS="-c debug_parallel_query=regress -c work_mem=50MB" + + When running against a temporary installation, custom settings can also be + set by supplying a pre-written postgresql.conf: + +echo 'log_checkpoints = on' > test_postgresql.conf +echo 'work_mem = 50MB' >> test_postgresql.conf +make check EXTRA_REGRESS_OPTS="--temp-config=test_postgresql.conf" + + + + + This can be useful to enable additional logging, adjust resource limits, + or enable extra run-time checks such as . + + + + + Extra Tests + + + The core regression test suite contains a few test files that are not + run by default, because they might be platform-dependent or take a + very long time to run. You can run these or other extra test + files by setting the variable EXTRA_TESTS. For + example, to run the numeric_big test: + +make check EXTRA_TESTS=numeric_big + + + + + + + Test Evaluation + + + Some properly installed and fully functional + PostgreSQL installations can + fail some of these regression tests due to + platform-specific artifacts such as varying floating-point representation + and message wording. The tests are currently evaluated using a simple + diff comparison against the outputs + generated on a reference system, so the results are sensitive to + small system differences. When a test is reported as + failed, always examine the differences between + expected and actual results; you might find that the + differences are not significant. Nonetheless, we still strive to + maintain accurate reference files across all supported platforms, + so it can be expected that all tests pass. + + + + The actual outputs of the regression tests are in files in the + src/test/regress/results directory. The test + script uses diff to compare each output + file against the reference outputs stored in the + src/test/regress/expected directory. Any + differences are saved for your inspection in + src/test/regress/regression.diffs. + (When running a test suite other than the core tests, these files + of course appear in the relevant subdirectory, + not src/test/regress.) + + + + If you don't + like the diff options that are used by default, set the + environment variable PG_REGRESS_DIFF_OPTS, for + instance PG_REGRESS_DIFF_OPTS='-c'. (Or you + can run diff yourself, if you prefer.) + + + + If for some reason a particular platform generates a failure + for a given test, but inspection of the output convinces you that + the result is valid, you can add a new comparison file to silence + the failure report in future test runs. See + for details. + + + + Error Message Differences + + + Some of the regression tests involve intentional invalid input + values. Error messages can come from either the + PostgreSQL code or from the host + platform system routines. In the latter case, the messages can + vary between platforms, but should reflect similar + information. These differences in messages will result in a + failed regression test that can be validated by + inspection. + + + + + Locale Differences + + + If you run the tests against a server that was + initialized with a collation-order locale other than C, then + there might be differences due to sort order and subsequent + failures. The regression test suite is set up to handle this + problem by providing alternate result files that together are + known to handle a large number of locales. + + + + To run the tests in a different locale when using the + temporary-installation method, pass the appropriate + locale-related environment variables on + the make command line, for example: + +make check LANG=de_DE.utf8 + + (The regression test driver unsets LC_ALL, so it + does not work to choose the locale using that variable.) To use + no locale, either unset all locale-related environment variables + (or set them to C) or use the following + special invocation: + +make check NO_LOCALE=1 + + When running the tests against an existing installation, the + locale setup is determined by the existing installation. To + change it, initialize the database cluster with a different + locale by passing the appropriate options + to initdb. + + + + In general, it is advisable to try to run the + regression tests in the locale setup that is wanted for + production use, as this will exercise the locale- and + encoding-related code portions that will actually be used in + production. Depending on the operating system environment, you + might get failures, but then you will at least know what + locale-specific behaviors to expect when running real + applications. + + + + + Date and Time Differences + + + Most of the date and time results are dependent on the time zone + environment. The reference files are generated for time zone + PST8PDT (Berkeley, California), and there will be + apparent failures if the tests are not run with that time zone setting. + The regression test driver sets environment variable + PGTZ to PST8PDT, which normally + ensures proper results. + + + + + Floating-Point Differences + + + Some of the tests involve computing 64-bit floating-point numbers (double + precision) from table columns. Differences in + results involving mathematical functions of double + precision columns have been observed. The float8 and + geometry tests are particularly prone to small differences + across platforms, or even with different compiler optimization settings. + Human eyeball comparison is needed to determine the real + significance of these differences which are usually 10 places to + the right of the decimal point. + + + + Some systems display minus zero as -0, while others + just show 0. + + + + Some systems signal errors from pow() and + exp() differently from the mechanism + expected by the current PostgreSQL + code. + + + + + Row Ordering Differences + + +You might see differences in which the same rows are output in a +different order than what appears in the expected file. In most cases +this is not, strictly speaking, a bug. Most of the regression test +scripts are not so pedantic as to use an ORDER BY for every single +SELECT, and so their result row orderings are not well-defined +according to the SQL specification. In practice, since we are +looking at the same queries being executed on the same data by the same +software, we usually get the same result ordering on all platforms, +so the lack of ORDER BY is not a problem. Some queries do exhibit +cross-platform ordering differences, however. When testing against an +already-installed server, ordering differences can also be caused by +non-C locale settings or non-default parameter settings, such as custom values +of work_mem or the planner cost parameters. + + + +Therefore, if you see an ordering difference, it's not something to +worry about, unless the query does have an ORDER BY that your +result is violating. However, please report it anyway, so that we can add an +ORDER BY to that particular query to eliminate the bogus +failure in future releases. + + + +You might wonder why we don't order all the regression test queries explicitly +to get rid of this issue once and for all. The reason is that that would +make the regression tests less useful, not more, since they'd tend +to exercise query plan types that produce ordered results to the +exclusion of those that don't. + + + + + Insufficient Stack Depth + + + If the errors test results in a server crash + at the select infinite_recurse() command, it means that + the platform's limit on process stack size is smaller than the + parameter indicates. This + can be fixed by running the server under a higher stack + size limit (4MB is recommended with the default value of + max_stack_depth). If you are unable to do that, an + alternative is to reduce the value of max_stack_depth. + + + + On platforms supporting getrlimit(), the server should + automatically choose a safe value of max_stack_depth; + so unless you've manually overridden this setting, a failure of this + kind is a reportable bug. + + + + + The <quote>random</quote> Test + + + The random test script is intended to produce + random results. In very rare cases, this causes that regression + test to fail. Typing: + +diff results/random.out expected/random.out + + should produce only one or a few lines of differences. You need + not worry unless the random test fails repeatedly. + + + + + Configuration Parameters + + + When running the tests against an existing installation, some non-default + parameter settings could cause the tests to fail. For example, changing + parameters such as enable_seqscan or + enable_indexscan could cause plan changes that would + affect the results of tests that use EXPLAIN. + + + + + + + Variant Comparison Files + + + Since some of the tests inherently produce environment-dependent + results, we have provided ways to specify alternate expected + result files. Each regression test can have several comparison files + showing possible results on different platforms. There are two + independent mechanisms for determining which comparison file is used + for each test. + + + + The first mechanism allows comparison files to be selected for + specific platforms. There is a mapping file, + src/test/regress/resultmap, that defines + which comparison file to use for each platform. + To eliminate bogus test failures for a particular platform, + you first choose or make a variant result file, and then add a line to the + resultmap file. + + + + Each line in the mapping file is of the form + +testname:output:platformpattern=comparisonfilename + + The test name is just the name of the particular regression test + module. The output value indicates which output file to check. For the + standard regression tests, this is always out. The + value corresponds to the file extension of the output file. + The platform pattern is a pattern in the style of the Unix + tool expr (that is, a regular expression with an implicit + ^ anchor at the start). It is matched against the + platform name as printed by config.guess. + The comparison file name is the base name of the substitute result + comparison file. + + + + For example: some systems lack a working strtof function, + for which our workaround causes rounding errors in the + float4 regression test. + Therefore, we provide a variant comparison file, + float4-misrounded-input.out, which includes + the results to be expected on these systems. To silence the bogus + failure message on Cygwin + platforms, resultmap includes: + +float4:out:.*-.*-cygwin.*=float4-misrounded-input.out + + which will trigger on any machine where the output of + config.guess matches .*-.*-cygwin.*. + Other lines in resultmap select the variant comparison + file for other platforms where it's appropriate. + + + + The second selection mechanism for variant comparison files is + much more automatic: it simply uses the best match among + several supplied comparison files. The regression test driver + script considers both the standard comparison file for a test, + testname.out, and variant files named + testname_digit.out + (where the digit is any single digit + 0-9). If any such file is an exact match, + the test is considered to pass; otherwise, the one that generates + the shortest diff is used to create the failure report. (If + resultmap includes an entry for the particular + test, then the base testname is the substitute + name given in resultmap.) + + + + For example, for the char test, the comparison file + char.out contains results that are expected + in the C and POSIX locales, while + the file char_1.out contains results sorted as + they appear in many other locales. + + + + The best-match mechanism was devised to cope with locale-dependent + results, but it can be used in any situation where the test results + cannot be predicted easily from the platform name alone. A limitation of + this mechanism is that the test driver cannot tell which variant is + actually correct for the current environment; it will just pick + the variant that seems to work best. Therefore it is safest to use this + mechanism only for variant results that you are willing to consider + equally valid in all contexts. + + + + + + TAP Tests + + + Various tests, particularly the client program tests + under src/bin, use the Perl TAP tools and are run + using the Perl testing program prove. You can pass + command-line options to prove by setting + the make variable PROVE_FLAGS, for example: + +make -C src/bin check PROVE_FLAGS='--timer' + + See the manual page of prove for more information. + + + + The make variable PROVE_TESTS + can be used to define a whitespace-separated list of paths relative + to the Makefile invoking prove + to run the specified subset of tests instead of the default + t/*.pl. For example: + +make check PROVE_TESTS='t/001_test1.pl t/003_test3.pl' + + + + + The TAP tests require the Perl module IPC::Run. + This module is available from + CPAN + or an operating system package. + They also require PostgreSQL to be + configured with the option . + + + + Generically speaking, the TAP tests will test the executables in a + previously-installed installation tree if you say make + installcheck, or will build a new local installation tree from + current sources if you say make check. In either + case they will initialize a local instance (data directory) and + transiently run a server in it. Some of these tests run more than one + server. Thus, these tests can be fairly resource-intensive. + + + + It's important to realize that the TAP tests will start test server(s) + even when you say make installcheck; this is unlike + the traditional non-TAP testing infrastructure, which expects to use an + already-running test server in that case. Some PostgreSQL + subdirectories contain both traditional-style and TAP-style tests, + meaning that make installcheck will produce a mix of + results from temporary servers and the already-running test server. + + + + Environment Variables + + + Data directories are named according to the test filename, and will be + retained if a test fails. If the environment variable + PG_TEST_NOCLEAN is set, data directories will be + retained regardless of test status. For example, retaining the data + directory regardless of test results when running the + pg_dump tests: + +PG_TEST_NOCLEAN=1 make -C src/bin/pg_dump check + + This environment variable also prevents the test's temporary directories + from being removed. + + + + Many operations in the test suites use a 180-second timeout, which on slow + hosts may lead to load-induced timeouts. Setting the environment variable + PG_TEST_TIMEOUT_DEFAULT to a higher number will change + the default to avoid this. + + + + + + + Test Coverage Examination + + + The PostgreSQL source code can be compiled with coverage testing + instrumentation, so that it becomes possible to examine which + parts of the code are covered by the regression tests or any other + test suite that is run with the code. This is currently supported + when compiling with GCC, and it requires the gcov + and lcov packages. + + + + Coverage with Autoconf and Make + + A typical workflow looks like this: + +./configure --enable-coverage ... OTHER OPTIONS ... +make +make check # or other test suite +make coverage-html + + Then point your HTML browser + to coverage/index.html. + + + + If you don't have lcov or prefer text output over an + HTML report, you can run + +make coverage + + instead of make coverage-html, which will + produce .gcov output files for each source file + relevant to the test. (make coverage and make + coverage-html will overwrite each other's files, so mixing them + might be confusing.) + + + + You can run several different tests before making the coverage report; + the execution counts will accumulate. If you want + to reset the execution counts between test runs, run: + +make coverage-clean + + + + + You can run the make coverage-html or make + coverage command in a subdirectory if you want a coverage + report for only a portion of the code tree. + + + + Use make distclean to clean up when done. + + + + + Coverage with Meson + + A typical workflow looks like this: + +meson setup -Db_coverage=true ... OTHER OPTIONS ... builddir/ +meson compile -C builddir/ +meson test -C builddir/ +cd builddir/ +ninja coverage-html + + Then point your HTML browser + to ./meson-logs/coveragereport/index.html. + + + + You can run several different tests before making the coverage report; + the execution counts will accumulate. + + + + + -- cgit v1.2.3