diff options
Diffstat (limited to 'test/README')
| -rw-r--r-- | test/README | 332 |
1 files changed, 332 insertions, 0 deletions
diff --git a/test/README b/test/README new file mode 100644 index 0000000..408a6a2 --- /dev/null +++ b/test/README @@ -0,0 +1,332 @@ +Writing APR tests + +All APR tests should be executable in 2 ways, as an individual program, or +as a part of the full test suite. The full test suite is controlled with +the testall program. At the beginning of the testall.c file, there is an +array of functions called tests. The testall program loops through this +array calling each function. Each function returns a CuSuite variable, which +is then added to the SuiteList. Once all Suites have been added, the SuiteList +is executed, and the output is printed to the screen. All functions in the +array should follow the same basic format: + +The Full Suite +-------------- + +/* The driver function. This must return a CuSuite variable, which will + * then be used to actually run the tests. Essentially, all Suites are a + * collection of tests. The driver will take each Suite, and put it in a + * SuiteList, which is a collection of Suites. + */ +CuSuite *testtime(void) +{ + /* The actual suite, this must be created for each test program. Please + * give it a useful name, that will inform the user of the feature being + * tested. + */ + CuSuite *suite = CuSuiteNew("Test Time"); + + /* Each function must be added to the suite. Each function represents + * a single test. It is possible to test multiple features in a single + * function, although no tests currently do that. + */ + SUITE_ADD_TEST(suite, test_now); + SUITE_ADD_TEST(suite, test_gmtstr); + SUITE_ADD_TEST(suite, test_localstr); + SUITE_ADD_TEST(suite, test_exp_get_gmt); + SUITE_ADD_TEST(suite, test_exp_get_lt); + SUITE_ADD_TEST(suite, test_imp_gmt); + SUITE_ADD_TEST(suite, test_rfcstr); + SUITE_ADD_TEST(suite, test_ctime); + SUITE_ADD_TEST(suite, test_strftime); + SUITE_ADD_TEST(suite, test_strftimesmall); + SUITE_ADD_TEST(suite, test_exp_tz); + SUITE_ADD_TEST(suite, test_strftimeoffset); + + /* You must return the suite so that the driver knows which suites to + * run. + */ + return suite; +} + +Building the full driver +------------------------ + +All you need to do to build the full driver is run: + + make + +To run it, run: + + ./testall + +Running individual tests +------------------------ + +It is not possible to build individual tests, however it is possible to +run individual tests. When running the test suite, specify the name of the +tests that you want to run on the command line. For example: + + ./testall teststr testrand + +Will run the Strings and Random generator tests. + +Reading the test suite output +----------------------------- + +Once you run the test suite, you will get output like: + +All APR Tests: + Test Strings: .... + Test Time: ............ + +16 tests run: 16 passed, 0 failed, 0 not implemented. + +Known test failures are documented in ../STATUS. + +There are a couple of things to look at with this. First, if you look at the +first function in this document, you should notice that the string passed to +the CuSuiteNew function is in the output. That is why the string should +explain the feature you are testing. + +Second, this test passed completely. This is obvious in two ways. First, and +most obvious, the summary line tells you that 16 tests were run and 16 tests +passed. However, the results can also be found in the lines above. Every +'.' in the output represents a passed test. + +If a test fails, the output will look like: + +All APR Tests: + Test Strings: .... + Test Time: ..F......... + +16 tests run: 15 passed, 1 failed, 0 not implemented. + +This is not very useful, because you don't know which test failed. However, +once you know that a test failed, you can run the suite again, with the +-v option. If you do this, you will get something like: + +All APR Tests: + Test Strings: .... + Test Time: ..F......... + +16 tests run: 15 passed, 1 failed, 0 not implemented. +Failed tests: +1) test_localstr: assert failed + +In this case, we know the test_localstr function failed, and there is an +Assert in this that failed (I modified the test to fail for this document). +Now, you can look at what that test does, and why it would have failed. + +There is one other possible output for the test suite (run with -v): + +All APR Tests: + Test Strings: .... + Test Time: ..N......... + +16 tests run: 15 passed, 0 failed, 1 not implemented. + +Not Implemented tests: + +Not Implemented tests: +1) test_localstr: apr_time_exp_lt not implemented on this platform + +The 'N' means that a function has returned APR_ENOTIMPL. This should be +treated as an error, and the function should be implemented as soon as +possible. + +Adding New test Suites to the full driver +------------------------------------------- + +To add a new Suite to the full driver, you must make a couple of modifications. + +1) Edit test_apr.h, and add the prototype for the function. +2) Edit testall.c, and add the function and name to the tests array. +3) Edit Makefile.in, and add the .lo file to the testall target. + +Once those four things are done, your tests will automatically be added +to the suite. + +Writting an ABTS unit test +-------------------------- + +The aim of this quick and dirty Howto is to give a short introduction +to APR (Apache Portable Runtime) unit tests, and how to write +one. During my Google's Summer of Code 2005 project, I discovered a +small bug in the APR-Util's date parsing routines, and I needed to +write a unit test for the fixed code. I decided to write this +documentation because I did not find any. Thanks to Garrett Rooney for +his help on writing the unit test ! + +The APR and APR-Util libraries provide a platform independent API for +software developers. They contain a lot of modules, including network +programming, threads, string and memory management, etc. All these +functions need to be heavily tested so that developers can be sure the +library is reliable. + +The ABTS give APR developers the ability to build a complete test +suite for the bunch of tests they wrote, which can then be ran under +various platforms. In this Howto, I will try teach you how to write an +ABTS unit test. + +As you may probably know, a unit test is a simple routine which tests +a very specific feature of the tested software or library. To build a +unit test, you need three different things : + + * the to-be-tested function, + * the input data that will be given to the function, + * the expected output data. + +The principle of a unit test is very simple : for each entry in your +set of input data, we pass it to our function, fetch what the function +returned and compare it to the corresponding expected output data. Of +course, the more edge cases you can test, the better your input data +set is. + +The ABTS aims to quicken the write of unit test, and make them +available to the whole test suite by providing a set of preprocessor +macros. Adding a unit test to a test suite can be easily done by the +following piece of code : + +abts_suite *testdaterfc(abts_suite *suite) +{ + suite = ADD_SUITE(suite); + abts_run_test(suite, test_date_rfc, NULL); + + return suite; +} + +Where test_date_rfc is the name of the function performing the +test. Writing such a function is, in the light of the explanation I +just gave, pretty much easy too. As I said, we need to check every +entry of our input data set. That gives us a loop. For each loop +iteration, we call our to-be-tested function, grab its result and +compare the returned value with the expected one. + +Test functions must have the following prototype : + +static void my_test_function(abts_case *tc, void *data); + +The comparison step is performed by the ABTS, thus giving the +whole test suite the correct behavior if your unit test fails. Here +comes a list of the available test methods : + +ABTS_INT_EQUAL(tc, a, b) +ABTS_INT_NEQUAL(tc, a, b) +ABTS_STR_EQUAL(tc, a, b) +ABTS_STR_NEQUAL(tc, a, b, c) +ABTS_PTR_NOTNULL(tc, b) +ABTS_PTR_EQUAL(tc, a, b) +ABTS_TRUE(tc, b) +ABTS_FAIL(tc, b) +ABTS_NOT_IMPL(tc, b) +ABTS_ASSERT(tc, a, b) + +The first argument, tc is a reference to the unit test currently +processed by the test suite (passed to your test function). The other +parameters are the data to be tested. For example, the following line +will never make your unit test fail : + +ABTS_INT_EQUAL(tc, 1, 1); + +See, it's easy ! Let's take a look at the complete example : +testdaterfc. We want to test our date string parser. For this, we will +use some chosen date strings (from mail headers for example) written +in various formats but that should all be handled by our function, and +their equivalents in correct RFC822 format. + +The function we want to test returns an apr_time_t}, which will be +directly given as input to the apr_rfc822_date() function, thus +producing the corresponding RFC822 date string. All we need to do +after this is to call the correct test method from the ABTS macros ! + +You can take a look at the apr-util/test/testdaterfc.c file for the +complete source code of this unit test. + +Although this Howto is very small and mostly dedicated to the +testdaterfc unit test, I hope you'll find it useful. Good luck ! + +Writing tests for CuTest (no longer used) +----------------------------------------- + +There are a couple of rules for writing good tests for the test suite. + +1) All tests can determine for themselves if it passed or not. This means +that there is no reason for the person running the test suite to interpret +the results of the tests. +2) Never use printf to add to the output of the test suite. The suite +library should be able to print all of the information required to debug +a problem. +3) Functions should be tested with both positive and negative tests. This +means that you should test things that should both succeed and fail. +4) Just checking the return code does _NOT_ make a useful test. You must +check to determine that the test actually did what you expected it to do. + +An example test +--------------- + +Finally, we will look at a quick test: + +/* All tests are passed a CuTest variable. This is how the suite determines + * if the test succeeded or failed. + */ +static void test_localstr(CuTest *tc) +{ + apr_status_t rv; + apr_time_exp_t xt; + time_t os_now; + + rv = apr_time_exp_lt(&xt, now); + os_now = now / APR_USEC_PER_SEC; + + /* If the function can return APR_ENOTIMPL, then you should check for it. + * This allows platform implementors to know if they have to implement + * the function. + */ + if (rv == APR_ENOTIMPL) { + CuNotImpl(tc, "apr_time_exp_lt"); + } + + /* It often helps to ensure that the return code was APR_SUCESS. If it + * wasn't, then we know the test failed. + */ + CuAssertTrue(tc, rv == APR_SUCCESS); + + /* Now that we know APR thinks it worked properly, we need to check the + * output to ensure that we got what we expected. + */ + CuAssertStrEquals(tc, "2002-08-14 12:05:36.186711 -25200 [257 Sat] DST", + print_time(p, &xt)); +} + +Notice, the same test can fail for any of a number of reasons. The first +test to fail ends the test. + +CuTest +------ + +CuTest is an open source test suite written by Asim Jalis. It has been +released under the zlib/libpng license. That license can be found in the +CuTest.c and CuTest.h files. + +The version of CuTest that is included in the APR test suite has been modified +from the original distribution in the following ways: + +1) The original distribution does not have a -v flag, the details are always +printed. +2) The NotImplemented result does not exist. +3) SuiteLists do not exist. In the original distribution, you can add suites +to suites, but it just adds the tests in the first suite to the list of tests +in the original suite. The output wasn't as detailed as I wanted, so I created +SuiteLists. + +The first two modifications have been sent to the original author of CuTest, +but they have not been integrated into the base distribution. The SuiteList +changes will be sent to the original author soon. + +The modified version of CuTest is not currently in any CVS or Subversion +server. In time, it will be hosted at rkbloom.net. + +There are currently no docs for how to write tests, but the teststr and +testtime programs should give an idea of how it is done. In time, a document +should be written to define how tests are written. + |