summaryrefslogtreecommitdiffstats
path: root/test/README
diff options
context:
space:
mode:
Diffstat (limited to 'test/README')
-rw-r--r--test/README332
1 files changed, 332 insertions, 0 deletions
diff --git a/test/README b/test/README
new file mode 100644
index 0000000..408a6a2
--- /dev/null
+++ b/test/README
@@ -0,0 +1,332 @@
+Writing APR tests
+
+All APR tests should be executable in 2 ways, as an individual program, or
+as a part of the full test suite. The full test suite is controlled with
+the testall program. At the beginning of the testall.c file, there is an
+array of functions called tests. The testall program loops through this
+array calling each function. Each function returns a CuSuite variable, which
+is then added to the SuiteList. Once all Suites have been added, the SuiteList
+is executed, and the output is printed to the screen. All functions in the
+array should follow the same basic format:
+
+The Full Suite
+--------------
+
+/* The driver function. This must return a CuSuite variable, which will
+ * then be used to actually run the tests. Essentially, all Suites are a
+ * collection of tests. The driver will take each Suite, and put it in a
+ * SuiteList, which is a collection of Suites.
+ */
+CuSuite *testtime(void)
+{
+ /* The actual suite, this must be created for each test program. Please
+ * give it a useful name, that will inform the user of the feature being
+ * tested.
+ */
+ CuSuite *suite = CuSuiteNew("Test Time");
+
+ /* Each function must be added to the suite. Each function represents
+ * a single test. It is possible to test multiple features in a single
+ * function, although no tests currently do that.
+ */
+ SUITE_ADD_TEST(suite, test_now);
+ SUITE_ADD_TEST(suite, test_gmtstr);
+ SUITE_ADD_TEST(suite, test_localstr);
+ SUITE_ADD_TEST(suite, test_exp_get_gmt);
+ SUITE_ADD_TEST(suite, test_exp_get_lt);
+ SUITE_ADD_TEST(suite, test_imp_gmt);
+ SUITE_ADD_TEST(suite, test_rfcstr);
+ SUITE_ADD_TEST(suite, test_ctime);
+ SUITE_ADD_TEST(suite, test_strftime);
+ SUITE_ADD_TEST(suite, test_strftimesmall);
+ SUITE_ADD_TEST(suite, test_exp_tz);
+ SUITE_ADD_TEST(suite, test_strftimeoffset);
+
+ /* You must return the suite so that the driver knows which suites to
+ * run.
+ */
+ return suite;
+}
+
+Building the full driver
+------------------------
+
+All you need to do to build the full driver is run:
+
+ make
+
+To run it, run:
+
+ ./testall
+
+Running individual tests
+------------------------
+
+It is not possible to build individual tests, however it is possible to
+run individual tests. When running the test suite, specify the name of the
+tests that you want to run on the command line. For example:
+
+ ./testall teststr testrand
+
+Will run the Strings and Random generator tests.
+
+Reading the test suite output
+-----------------------------
+
+Once you run the test suite, you will get output like:
+
+All APR Tests:
+ Test Strings: ....
+ Test Time: ............
+
+16 tests run: 16 passed, 0 failed, 0 not implemented.
+
+Known test failures are documented in ../STATUS.
+
+There are a couple of things to look at with this. First, if you look at the
+first function in this document, you should notice that the string passed to
+the CuSuiteNew function is in the output. That is why the string should
+explain the feature you are testing.
+
+Second, this test passed completely. This is obvious in two ways. First, and
+most obvious, the summary line tells you that 16 tests were run and 16 tests
+passed. However, the results can also be found in the lines above. Every
+'.' in the output represents a passed test.
+
+If a test fails, the output will look like:
+
+All APR Tests:
+ Test Strings: ....
+ Test Time: ..F.........
+
+16 tests run: 15 passed, 1 failed, 0 not implemented.
+
+This is not very useful, because you don't know which test failed. However,
+once you know that a test failed, you can run the suite again, with the
+-v option. If you do this, you will get something like:
+
+All APR Tests:
+ Test Strings: ....
+ Test Time: ..F.........
+
+16 tests run: 15 passed, 1 failed, 0 not implemented.
+Failed tests:
+1) test_localstr: assert failed
+
+In this case, we know the test_localstr function failed, and there is an
+Assert in this that failed (I modified the test to fail for this document).
+Now, you can look at what that test does, and why it would have failed.
+
+There is one other possible output for the test suite (run with -v):
+
+All APR Tests:
+ Test Strings: ....
+ Test Time: ..N.........
+
+16 tests run: 15 passed, 0 failed, 1 not implemented.
+
+Not Implemented tests:
+
+Not Implemented tests:
+1) test_localstr: apr_time_exp_lt not implemented on this platform
+
+The 'N' means that a function has returned APR_ENOTIMPL. This should be
+treated as an error, and the function should be implemented as soon as
+possible.
+
+Adding New test Suites to the full driver
+-------------------------------------------
+
+To add a new Suite to the full driver, you must make a couple of modifications.
+
+1) Edit test_apr.h, and add the prototype for the function.
+2) Edit testall.c, and add the function and name to the tests array.
+3) Edit Makefile.in, and add the .lo file to the testall target.
+
+Once those four things are done, your tests will automatically be added
+to the suite.
+
+Writting an ABTS unit test
+--------------------------
+
+The aim of this quick and dirty Howto is to give a short introduction
+to APR (Apache Portable Runtime) unit tests, and how to write
+one. During my Google's Summer of Code 2005 project, I discovered a
+small bug in the APR-Util's date parsing routines, and I needed to
+write a unit test for the fixed code. I decided to write this
+documentation because I did not find any. Thanks to Garrett Rooney for
+his help on writing the unit test !
+
+The APR and APR-Util libraries provide a platform independent API for
+software developers. They contain a lot of modules, including network
+programming, threads, string and memory management, etc. All these
+functions need to be heavily tested so that developers can be sure the
+library is reliable.
+
+The ABTS give APR developers the ability to build a complete test
+suite for the bunch of tests they wrote, which can then be ran under
+various platforms. In this Howto, I will try teach you how to write an
+ABTS unit test.
+
+As you may probably know, a unit test is a simple routine which tests
+a very specific feature of the tested software or library. To build a
+unit test, you need three different things :
+
+ * the to-be-tested function,
+ * the input data that will be given to the function,
+ * the expected output data.
+
+The principle of a unit test is very simple : for each entry in your
+set of input data, we pass it to our function, fetch what the function
+returned and compare it to the corresponding expected output data. Of
+course, the more edge cases you can test, the better your input data
+set is.
+
+The ABTS aims to quicken the write of unit test, and make them
+available to the whole test suite by providing a set of preprocessor
+macros. Adding a unit test to a test suite can be easily done by the
+following piece of code :
+
+abts_suite *testdaterfc(abts_suite *suite)
+{
+ suite = ADD_SUITE(suite);
+ abts_run_test(suite, test_date_rfc, NULL);
+
+ return suite;
+}
+
+Where test_date_rfc is the name of the function performing the
+test. Writing such a function is, in the light of the explanation I
+just gave, pretty much easy too. As I said, we need to check every
+entry of our input data set. That gives us a loop. For each loop
+iteration, we call our to-be-tested function, grab its result and
+compare the returned value with the expected one.
+
+Test functions must have the following prototype :
+
+static void my_test_function(abts_case *tc, void *data);
+
+The comparison step is performed by the ABTS, thus giving the
+whole test suite the correct behavior if your unit test fails. Here
+comes a list of the available test methods :
+
+ABTS_INT_EQUAL(tc, a, b)
+ABTS_INT_NEQUAL(tc, a, b)
+ABTS_STR_EQUAL(tc, a, b)
+ABTS_STR_NEQUAL(tc, a, b, c)
+ABTS_PTR_NOTNULL(tc, b)
+ABTS_PTR_EQUAL(tc, a, b)
+ABTS_TRUE(tc, b)
+ABTS_FAIL(tc, b)
+ABTS_NOT_IMPL(tc, b)
+ABTS_ASSERT(tc, a, b)
+
+The first argument, tc is a reference to the unit test currently
+processed by the test suite (passed to your test function). The other
+parameters are the data to be tested. For example, the following line
+will never make your unit test fail :
+
+ABTS_INT_EQUAL(tc, 1, 1);
+
+See, it's easy ! Let's take a look at the complete example :
+testdaterfc. We want to test our date string parser. For this, we will
+use some chosen date strings (from mail headers for example) written
+in various formats but that should all be handled by our function, and
+their equivalents in correct RFC822 format.
+
+The function we want to test returns an apr_time_t}, which will be
+directly given as input to the apr_rfc822_date() function, thus
+producing the corresponding RFC822 date string. All we need to do
+after this is to call the correct test method from the ABTS macros !
+
+You can take a look at the apr-util/test/testdaterfc.c file for the
+complete source code of this unit test.
+
+Although this Howto is very small and mostly dedicated to the
+testdaterfc unit test, I hope you'll find it useful. Good luck !
+
+Writing tests for CuTest (no longer used)
+-----------------------------------------
+
+There are a couple of rules for writing good tests for the test suite.
+
+1) All tests can determine for themselves if it passed or not. This means
+that there is no reason for the person running the test suite to interpret
+the results of the tests.
+2) Never use printf to add to the output of the test suite. The suite
+library should be able to print all of the information required to debug
+a problem.
+3) Functions should be tested with both positive and negative tests. This
+means that you should test things that should both succeed and fail.
+4) Just checking the return code does _NOT_ make a useful test. You must
+check to determine that the test actually did what you expected it to do.
+
+An example test
+---------------
+
+Finally, we will look at a quick test:
+
+/* All tests are passed a CuTest variable. This is how the suite determines
+ * if the test succeeded or failed.
+ */
+static void test_localstr(CuTest *tc)
+{
+ apr_status_t rv;
+ apr_time_exp_t xt;
+ time_t os_now;
+
+ rv = apr_time_exp_lt(&xt, now);
+ os_now = now / APR_USEC_PER_SEC;
+
+ /* If the function can return APR_ENOTIMPL, then you should check for it.
+ * This allows platform implementors to know if they have to implement
+ * the function.
+ */
+ if (rv == APR_ENOTIMPL) {
+ CuNotImpl(tc, "apr_time_exp_lt");
+ }
+
+ /* It often helps to ensure that the return code was APR_SUCESS. If it
+ * wasn't, then we know the test failed.
+ */
+ CuAssertTrue(tc, rv == APR_SUCCESS);
+
+ /* Now that we know APR thinks it worked properly, we need to check the
+ * output to ensure that we got what we expected.
+ */
+ CuAssertStrEquals(tc, "2002-08-14 12:05:36.186711 -25200 [257 Sat] DST",
+ print_time(p, &xt));
+}
+
+Notice, the same test can fail for any of a number of reasons. The first
+test to fail ends the test.
+
+CuTest
+------
+
+CuTest is an open source test suite written by Asim Jalis. It has been
+released under the zlib/libpng license. That license can be found in the
+CuTest.c and CuTest.h files.
+
+The version of CuTest that is included in the APR test suite has been modified
+from the original distribution in the following ways:
+
+1) The original distribution does not have a -v flag, the details are always
+printed.
+2) The NotImplemented result does not exist.
+3) SuiteLists do not exist. In the original distribution, you can add suites
+to suites, but it just adds the tests in the first suite to the list of tests
+in the original suite. The output wasn't as detailed as I wanted, so I created
+SuiteLists.
+
+The first two modifications have been sent to the original author of CuTest,
+but they have not been integrated into the base distribution. The SuiteList
+changes will be sent to the original author soon.
+
+The modified version of CuTest is not currently in any CVS or Subversion
+server. In time, it will be hosted at rkbloom.net.
+
+There are currently no docs for how to write tests, but the teststr and
+testtime programs should give an idea of how it is done. In time, a document
+should be written to define how tests are written.
+