diff options
Diffstat (limited to 'doc/src/sgml/html/regress-evaluation.html')
| -rw-r--r-- | doc/src/sgml/html/regress-evaluation.html | 166 |
1 files changed, 166 insertions, 0 deletions
diff --git a/doc/src/sgml/html/regress-evaluation.html b/doc/src/sgml/html/regress-evaluation.html new file mode 100644 index 0000000..ab1fa6c --- /dev/null +++ b/doc/src/sgml/html/regress-evaluation.html @@ -0,0 +1,166 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>32.2. Test Evaluation</title><link rel="stylesheet" type="text/css" href="stylesheet.css" /><link rev="made" href="pgsql-docs@lists.postgresql.org" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /><link rel="prev" href="regress-run.html" title="32.1. Running the Tests" /><link rel="next" href="regress-variant.html" title="32.3. Variant Comparison Files" /></head><body id="docContent" class="container-fluid col-10"><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">32.2. Test Evaluation</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="regress-run.html" title="32.1. Running the Tests">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="regress.html" title="Chapter 32. Regression Tests">Up</a></td><th width="60%" align="center">Chapter 32. Regression Tests</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 13.4 Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="regress-variant.html" title="32.3. Variant Comparison Files">Next</a></td></tr></table><hr></hr></div><div class="sect1" id="REGRESS-EVALUATION"><div class="titlepage"><div><div><h2 class="title" style="clear: both">32.2. Test Evaluation</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.6">32.2.1. Error Message Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.7">32.2.2. Locale Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.8">32.2.3. Date and Time Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.9">32.2.4. Floating-Point Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.10">32.2.5. Row Ordering Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.11">32.2.6. Insufficient Stack Depth</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.12">32.2.7. The <span class="quote">“<span class="quote">random</span>”</span> Test</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#id-1.6.19.6.13">32.2.8. Configuration Parameters</a></span></dt></dl></div><p> + Some properly installed and fully functional + <span class="productname">PostgreSQL</span> installations can + <span class="quote">“<span class="quote">fail</span>”</span> some of these regression tests due to + platform-specific artifacts such as varying floating-point representation + and message wording. The tests are currently evaluated using a simple + <code class="command">diff</code> comparison against the outputs + generated on a reference system, so the results are sensitive to + small system differences. When a test is reported as + <span class="quote">“<span class="quote">failed</span>”</span>, always examine the differences between + expected and actual results; you might find that the + differences are not significant. Nonetheless, we still strive to + maintain accurate reference files across all supported platforms, + so it can be expected that all tests pass. + </p><p> + The actual outputs of the regression tests are in files in the + <code class="filename">src/test/regress/results</code> directory. The test + script uses <code class="command">diff</code> to compare each output + file against the reference outputs stored in the + <code class="filename">src/test/regress/expected</code> directory. Any + differences are saved for your inspection in + <code class="filename">src/test/regress/regression.diffs</code>. + (When running a test suite other than the core tests, these files + of course appear in the relevant subdirectory, + not <code class="filename">src/test/regress</code>.) + </p><p> + If you don't + like the <code class="command">diff</code> options that are used by default, set the + environment variable <code class="envar">PG_REGRESS_DIFF_OPTS</code>, for + instance <code class="literal">PG_REGRESS_DIFF_OPTS='-c'</code>. (Or you + can run <code class="command">diff</code> yourself, if you prefer.) + </p><p> + If for some reason a particular platform generates a <span class="quote">“<span class="quote">failure</span>”</span> + for a given test, but inspection of the output convinces you that + the result is valid, you can add a new comparison file to silence + the failure report in future test runs. See + <a class="xref" href="regress-variant.html" title="32.3. Variant Comparison Files">Section 32.3</a> for details. + </p><div class="sect2" id="id-1.6.19.6.6"><div class="titlepage"><div><div><h3 class="title">32.2.1. Error Message Differences</h3></div></div></div><p> + Some of the regression tests involve intentional invalid input + values. Error messages can come from either the + <span class="productname">PostgreSQL</span> code or from the host + platform system routines. In the latter case, the messages can + vary between platforms, but should reflect similar + information. These differences in messages will result in a + <span class="quote">“<span class="quote">failed</span>”</span> regression test that can be validated by + inspection. + </p></div><div class="sect2" id="id-1.6.19.6.7"><div class="titlepage"><div><div><h3 class="title">32.2.2. Locale Differences</h3></div></div></div><p> + If you run the tests against a server that was + initialized with a collation-order locale other than C, then + there might be differences due to sort order and subsequent + failures. The regression test suite is set up to handle this + problem by providing alternate result files that together are + known to handle a large number of locales. + </p><p> + To run the tests in a different locale when using the + temporary-installation method, pass the appropriate + locale-related environment variables on + the <code class="command">make</code> command line, for example: +</p><pre class="programlisting"> +make check LANG=de_DE.utf8 +</pre><p> + (The regression test driver unsets <code class="envar">LC_ALL</code>, so it + does not work to choose the locale using that variable.) To use + no locale, either unset all locale-related environment variables + (or set them to <code class="literal">C</code>) or use the following + special invocation: +</p><pre class="programlisting"> +make check NO_LOCALE=1 +</pre><p> + When running the tests against an existing installation, the + locale setup is determined by the existing installation. To + change it, initialize the database cluster with a different + locale by passing the appropriate options + to <code class="command">initdb</code>. + </p><p> + In general, it is advisable to try to run the + regression tests in the locale setup that is wanted for + production use, as this will exercise the locale- and + encoding-related code portions that will actually be used in + production. Depending on the operating system environment, you + might get failures, but then you will at least know what + locale-specific behaviors to expect when running real + applications. + </p></div><div class="sect2" id="id-1.6.19.6.8"><div class="titlepage"><div><div><h3 class="title">32.2.3. Date and Time Differences</h3></div></div></div><p> + Most of the date and time results are dependent on the time zone + environment. The reference files are generated for time zone + <code class="literal">PST8PDT</code> (Berkeley, California), and there will be + apparent failures if the tests are not run with that time zone setting. + The regression test driver sets environment variable + <code class="envar">PGTZ</code> to <code class="literal">PST8PDT</code>, which normally + ensures proper results. + </p></div><div class="sect2" id="id-1.6.19.6.9"><div class="titlepage"><div><div><h3 class="title">32.2.4. Floating-Point Differences</h3></div></div></div><p> + Some of the tests involve computing 64-bit floating-point numbers (<code class="type">double + precision</code>) from table columns. Differences in + results involving mathematical functions of <code class="type">double + precision</code> columns have been observed. The <code class="literal">float8</code> and + <code class="literal">geometry</code> tests are particularly prone to small differences + across platforms, or even with different compiler optimization settings. + Human eyeball comparison is needed to determine the real + significance of these differences which are usually 10 places to + the right of the decimal point. + </p><p> + Some systems display minus zero as <code class="literal">-0</code>, while others + just show <code class="literal">0</code>. + </p><p> + Some systems signal errors from <code class="function">pow()</code> and + <code class="function">exp()</code> differently from the mechanism + expected by the current <span class="productname">PostgreSQL</span> + code. + </p></div><div class="sect2" id="id-1.6.19.6.10"><div class="titlepage"><div><div><h3 class="title">32.2.5. Row Ordering Differences</h3></div></div></div><p> +You might see differences in which the same rows are output in a +different order than what appears in the expected file. In most cases +this is not, strictly speaking, a bug. Most of the regression test +scripts are not so pedantic as to use an <code class="literal">ORDER BY</code> for every single +<code class="literal">SELECT</code>, and so their result row orderings are not well-defined +according to the SQL specification. In practice, since we are +looking at the same queries being executed on the same data by the same +software, we usually get the same result ordering on all platforms, +so the lack of <code class="literal">ORDER BY</code> is not a problem. Some queries do exhibit +cross-platform ordering differences, however. When testing against an +already-installed server, ordering differences can also be caused by +non-C locale settings or non-default parameter settings, such as custom values +of <code class="varname">work_mem</code> or the planner cost parameters. + </p><p> +Therefore, if you see an ordering difference, it's not something to +worry about, unless the query does have an <code class="literal">ORDER BY</code> that your +result is violating. However, please report it anyway, so that we can add an +<code class="literal">ORDER BY</code> to that particular query to eliminate the bogus +<span class="quote">“<span class="quote">failure</span>”</span> in future releases. + </p><p> +You might wonder why we don't order all the regression test queries explicitly +to get rid of this issue once and for all. The reason is that that would +make the regression tests less useful, not more, since they'd tend +to exercise query plan types that produce ordered results to the +exclusion of those that don't. + </p></div><div class="sect2" id="id-1.6.19.6.11"><div class="titlepage"><div><div><h3 class="title">32.2.6. Insufficient Stack Depth</h3></div></div></div><p> + If the <code class="literal">errors</code> test results in a server crash + at the <code class="literal">select infinite_recurse()</code> command, it means that + the platform's limit on process stack size is smaller than the + <a class="xref" href="runtime-config-resource.html#GUC-MAX-STACK-DEPTH">max_stack_depth</a> parameter indicates. This + can be fixed by running the server under a higher stack + size limit (4MB is recommended with the default value of + <code class="varname">max_stack_depth</code>). If you are unable to do that, an + alternative is to reduce the value of <code class="varname">max_stack_depth</code>. + </p><p> + On platforms supporting <code class="function">getrlimit()</code>, the server should + automatically choose a safe value of <code class="varname">max_stack_depth</code>; + so unless you've manually overridden this setting, a failure of this + kind is a reportable bug. + </p></div><div class="sect2" id="id-1.6.19.6.12"><div class="titlepage"><div><div><h3 class="title">32.2.7. The <span class="quote">“<span class="quote">random</span>”</span> Test</h3></div></div></div><p> + The <code class="literal">random</code> test script is intended to produce + random results. In very rare cases, this causes that regression + test to fail. Typing: +</p><pre class="programlisting"> +diff results/random.out expected/random.out +</pre><p> + should produce only one or a few lines of differences. You need + not worry unless the random test fails repeatedly. + </p></div><div class="sect2" id="id-1.6.19.6.13"><div class="titlepage"><div><div><h3 class="title">32.2.8. Configuration Parameters</h3></div></div></div><p> + When running the tests against an existing installation, some non-default + parameter settings could cause the tests to fail. For example, changing + parameters such as <code class="varname">enable_seqscan</code> or + <code class="varname">enable_indexscan</code> could cause plan changes that would + affect the results of tests that use <code class="command">EXPLAIN</code>. + </p></div></div><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navfooter"><hr></hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="regress-run.html" title="32.1. Running the Tests">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="regress.html" title="Chapter 32. Regression Tests">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="regress-variant.html" title="32.3. Variant Comparison Files">Next</a></td></tr><tr><td width="40%" align="left" valign="top">32.1. Running the Tests </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 13.4 Documentation">Home</a></td><td width="40%" align="right" valign="top"> 32.3. Variant Comparison Files</td></tr></table></div></body></html>
\ No newline at end of file |