A testing system for B2


Introduction for users
Command line options
Introduction for developers
Changing the working directory
Examining the working directory and changing it
Test result
Reference documentation
Method __init__
Method set_tree
Method write
Method copy
Method touch
Method run_build_system
Method read
Method read_and_strip
Methods for declaring expectations
Methods for ignoring changes
Methods for explicitly specifying results
Helper class List

Introduction for users

The testing system for B2 is a small set of Python modules and scripts for automatically testing user-obversable behaviour. It uses components from testing systems of Scons and Subversion, together with some additional functionality.

To run the tests you need to:

  1. Get the source tree of B2 (located at tools/build in Boost)
  2. Have Python installed. Version 2.1 is known to work.
  3. Build Boost.Jam. See $boost_build_root/engine/index.html for instructions.
  4. Configure at least one toolset. You can edit site-config.jam or user-config.jam to add new toolsets. Or you can create file test-config.jam in $boost_build_root/test directory. In this case, site-config.jam and user-config.jam will be ignored for testing.
  5. Make sure that in the configuration jam file(s) that you use you generate no console output, ie. with the Boost Build 'ECHO' rule. Such console output in the configuration jam file(s) will cause a number of tests to automatically fail which would otherwise succeed.

When all is set, you can run all the tests using the test_all.py script or you can run a specific test by starting its Python script directly.

Examples:

python test_all.py
python generators_test.py

If everything is OK, you will see a list of passed tests. Otherwise, a failure will be reported.

Command line options

Test scripts will use the toolset you configured to be the default or you can specify a specific one on the command line:

python test_all.py borland
python generators_test.py msvc-7.1

Other test script flags you can specify on the command line are:

Introduction for developers

It is suggested that every new functionality come together with tests, and that bugfixes are accompanied by tests. There's no need to say that tests are good, but two points are extremely important:

Adding a new test is simple:

  1. Go to $boost_build_root/test/test_all.py and add new test name to the list at the end of the file. Suppose the test name is "hello".
  2. Add a new python module, in this example "hello.py", to do the actual testing.

The module, in general will perform these basic actions:

  1. Set up the initial working directory state
  2. Run the build system and check the results:
    1. generated output,
    2. changes made to the working directory,
    3. new content of the working directory.
  3. Add, remove or touch files or change their content and then repeat the previous step until satisfied.
  4. Clean up

The "hello.py" module might contain:

from BoostBuild import List

# Create a temporary working directory
t = BoostBuild.Tester()

# Create the needed files
t.write("jamroot.jam", "")
t.write("jamfile.jam", """
exe hello : hello.cpp ;
""")
t.write("hello.cpp", """
int main()
{
    return 0;
}

""")

t.run_build_system()

# First, create a list of three pathnames.
file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
# Second, assert that those files were added as result of the last build system invocation.
t.expect_addition(file_list)

# Invoke the build system once again.
t.run_build_system("clean")
# Check if the files added previously were removed.
t.expect_removal(file_list)

# Remove temporary directories
t.cleanup()

The test directory contains a file "template.py" which can be used as a start for your own tests.

Overview of the most important methods of class Tester follows.

Changing the working directory

The class Tester creates a temporary directory in its constructor and changes to that directory. It can be modified by calling these methods:

Examining the working directory and changing it

The method read, inherited from the TestCmd class, can be used to read any file in the working directory and check its content. Tester adds another method for tracking changes. Whenever the build system is run (using run_build_system ), the working dir state before and after running is recorded. In addition, difference between the two states -- i.e. lists of files that were added, removed, modified or touched -- are stored in two member variables - tree_difference and unexpected_difference.

After than, the test author may specify that some change is expected, for example, by calling expect_addition("foo"). This call will check if the file was indeed added, and if so, will remove its name from the list of added files in unexpected_difference. Likewise, it is possible to specify that some changes are not interesting, for example a call to ignore("*.obj") will just remove every file with the ".obj" extension from unexpected_difference.

When test has finished with expectations and ignoring, the member unexpected_difference will contain the list of all changes not yet accounted for. It is possible to assure that this list is empty by calling the expect_nothing_more member function.

Test result

Any of the expect* methods below will fail the test if the expectation is not met. It is also possible to perform manually arbitrary test and explicitly cause the test to either pass or fail. Ordinary filesystem functions can be used to work with the directory tree. Methods pass_test and fail_test are used to explicitly give the test outcome.

Typically, after test termination, the working directory is erased. See the "--preserve" command line option for information on how to preserve the working directory content for failed tests for debugging purposes.

Reference documentation

The test system is composed of class Tester, derived form TestCmd.TestCmd, and helper class List. Tester and List methods are described below.

The documentation frequently refers to filename. In all cases, files are specified in unix style: a sequence of components, separated by "/". This is true on all platforms. In some contexts a list of files is allowed. In those cases any object with a sequence interface is allowed.

Method __init__(self, arguments="", executable="bjam", match=TestCmd.match_exact, boost_build_path=None, translate_suffixes=True, pass_toolset=True, use_test_config=True, ignore_toolset_requirements=True, workdir="", **keywords)

Optional arguments:

Optional arguments inherited from the base class:

Effects:

  1. Remembers the current working directory in member original_workdir.
  2. Determines the location of the executable (bjam by default) and build system files, assuming that the current directory is tools/build/test. Formulates jam invocation command, which will include explicit setting for the BOOST_BUILD_PATH variable and arguments passed to this methods, if any. This command will be used by subsequent invocation of run_build_system. Finally, initializes the base class.
  3. Changes the current working directory to the temporary working directory created by the base constructor.
  4. If you want to run a test in an existing directory, pass it as workdir.
  5. Most parameters passed to this constructor function may be overruled for each specific test system run using run_build_system parameters.

Method set_tree(self, tree_location)

Effects:

Replaces the content of the current working directory with the content of directory at tree_location. If tree_location is not absolute pathname, it will be treated as relative to self.original_workdir. This methods also explicitly makes the copied files writeable.

Method write(self, name, content)

Effects:

Writes the specified content to the file given by name under the temporary working directory. If the file already exists, it is overwritten. Any required directories are automatically created.

Method copy(self, src, dst)

Effects:

Equvivalent to self.write(self.read(src), dst).

Method touch(self, names)

Effects:

Sets the access and modification times for all files in names to the current time. All the elements in names should be relative paths.

Method run_build_system(self, extra_args="", subdir="", stdout=None, stderr="", status=0, match=None, pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None, expected_duration=None, **kw)

Effects:

  1. Stores the state of the working directory in self.previous_tree.
  2. Changes to subdir, if it is specified. It is relative to the original_workdir or the workdir specified in __init.
  3. Invokes the bjam executable, passing extra_args to it. The binary should be located under <test_invocation_dir>/../jam/src/bin.<platform>. This is to make sure tests use the version of jam build from CVS.
  4. Compares the stdout, stderr and exit status of build system invocation with values to appropriate parameters, if they are not None. If any difference is found, the test fails.
  5. If the expected_duration parameter is specified then it represents the maximal allowed time in seconds for the test to run. The test will be marked as failed if its duration is greater than the given expected_duration parameter value.
  6. Stores the new state of the working directory in self.tree. Computes the difference between previous and current trees and stores them in variables self.tree_difference and self.unexpected_difference. Both variables are instances of class tree.Trees_different, which have four attributes: added_files, removed_files, modified_files and touched_files. Each is a list of strings.

Method read(self, name)

Effects:

Read the specified file and returns it content. Raises an exception is the file is absent.

Method read_and_strip(self, name)

Effects:

Read the specified file and returns it content, after removing trailing whitespace from every line. Raises an exception is the file is absent.

Rationale:

Although this method is questionable, there are a lot of cases when jam or shells it uses insert spaces. It seems that introducing this method is much simpler than dealing with all those cases.

Methods for declaring expectations

Accordingly to the number of changes kinds that are detected, there are four methods that specify that test author expects a specific change to occur. They check self.unexpected_difference, and if the change is present there, it is removed. Otherwise, test fails.

Each method accepts a list of names. Those names use / path separator on all systems. Additionally, the test system translates suffixes appropriately. For the test to be portable, suffixes should use Windows convention: exe for executables, dll for dynamic libraries and lib for static libraries. Lastly, the string "$toolset" in file names is replaced by the name of tested toolset.

Note: The List helper class might be useful to create lists of names.

Note: The file content can be examined using the TestCmd.read function.

The members are:

Note that expect_modification is used to check that a either file content or timestamp has changed. The rationale is that some compilers change content even if sources does not change, and it's easier to have a method which checks for both content and time changes.

There's also a member expect_nothing_more, which checks that all the changes are either expected or ignored, in other words that unexpected_difference is empty by now.

Lastly, there's a method to compare file content with expected content:

expect_content(self, name, content, exact=0)

The method fails the test if the content of file identified by 'name' is different from 'content'. If 'exact' is true, the file content is used as-is, otherwise, two transformations are applied:

Methods for ignoring changes

There are five methods which ignore changes made to the working tree. They silently remove elements from self.unexpected_difference, and don't generate error if element is not found. They accept shell style wildcard.

The following methods correspond to four kinds of changes:

The method ignore(self, wildcard) ignores all the changes made to files that match a wildcard.

Methods for explicitly specifying results

Method pass_test(self, condition=1)

At this moment, the method should not be used.

Method fail_test(self, condition=1)

Effects: Cause the test to fail if condition is true.

Helper class List

The class has sequence interface and two additional methods.

Method __init__(self, string)

Effects: Splits the string on unescaped spaces and tabs. The split components can further be retrieved using standard sequence access.

Method __mul__(self, other)

Effects: Returns an List instance, which elements are all possible concatenations of two string, first of which is from self, and second of which is from other.

The class also defines __str__ and __repr__ methods. Finally, there's __coerce__ method which allows to convert strings to instances of List.

Example:

    l = "a b" * List("c d")
    for e in l:
        print e

will output:

    ac
    ad
    bc
    bd


Last modified: May 02, 2008

© Copyright Vladimir Prus 2002, 2003, 2004, 2005.
© Copyright Jurko Gospodnetic 2008.
Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)