diff options
Diffstat (limited to '')
| -rw-r--r-- | docbook/wsdg_src/wsdg_tests.adoc | 298 |
1 files changed, 298 insertions, 0 deletions
diff --git a/docbook/wsdg_src/wsdg_tests.adoc b/docbook/wsdg_src/wsdg_tests.adoc new file mode 100644 index 00000000..a9435ca7 --- /dev/null +++ b/docbook/wsdg_src/wsdg_tests.adoc @@ -0,0 +1,298 @@ +// WSDG Chapter Tests + +[#ChapterTests] +== Wireshark Tests + +The Wireshark sources include a collection of Python scripts that test +the features of Wireshark, TShark, Dumpcap, and other programs that +accompany Wireshark. These are located in the `test` directory of the +Wireshark source tree. + +The command line options of Wireshark and its companion command line +tools are numerous. These tests help to ensure that we don't introduce +bugs as Wireshark grows and evolves. + +[#TestsQuickStart] +=== Quick Start + +The recommended steps to prepare for and run tests with a UN*X toolchain: + +* Install two Python packages, pytest: `pip install pytest pytest-xdist` +* Build programs (“wireshark”, “tshark”, etc.): `ninja` +* Build additional programs for the “unittests” suite: `ninja test-programs` +* Run tests in the build directory: `pytest` + +Replace `ninja` by `make` as needed. + +If building with <<#ChWindowsBuild,Microsoft Visual Studio>> the analogous steps are: + +* Install pytest Python packages: `python -m pip install pytest pytest-xdist` +* Build programs: `msbuild /m /p:Configuration=RelWithDebInfo Wireshark.sln` +* Build test-programs: `msbuild /m /p:Configuration=RelWithDebInfo test-programs.vcxproj` +* Run tests: `python -m pytest` + +TIP: Depending on your PATH, you may need to run the pytest module as a +script from your Python interpreter, e.g, `python -m pytest` or +`python3 -m pytest` instead of `pytest`. + +The test suite will attempt to test as much as possible and skip tests +when its dependencies are not satisfied. For example, packet capture +tests require a Loopback interface and capture privileges. To avoid +capture tests, pass the `--disable-capture` option. + +List available tests with `pytest --collectonly`. Enable verbose output +with `pytest --verbose`. For more details, see <<ChTestsRun>>. + +You can also run the "ninja test" target instead of invoking pytest +directly. This will automatically build the test programs dependency, +so it may be preferred for that reason. + +[#ChTestsStructure] +=== Test suite structure + +The following sections describes how the test suite is organized. + +[#TestCoverage] +==== Test Coverage And Availability + +The testing framework can run programs and check their stdout, stderr, +and exit codes. It cannot interact with the Wireshark UI. Tests cover +capture, command line options, decryption, file format support and +conversion, Lua scripting, and other functionality. + +Available tests depend on the libraries with which Wireshark was built. +For example, some decryption tests depend on a minimum version of +Libgcrypt and Lua tests depend on Lua. + +Capture tests depend on the permissions of the user running the test +script. We assume that the test user has capture permissions on Windows +and macOS and capture tests are enabled by default on those platforms. + +TIP: Build the "test-capture" target on Linux (using sudo) to set dumpcap +permissions and enable capture tests. + +If a feature is unavailable, the test will be skipped. For example, if +an old version of Libgcrypt is in use, then some decryption tests will +be skipped while other tests can still run to completion. + +[#TestsLayout] +==== Suites, Cases, and Tests + +The test suite uses pytest as a test runner. Tests are organized according to +suites, cases, and individual tests. Suites correspond to Python modules +that match the pattern “suite_*.py”. Cases correspond to one or more +classes in each module, and case class methods matching the pattern +”test_*” correspond to individual tests. For example, the invalid +capture filter test in the TShark capture command line options test case +in the command line options suite has the ID +“suite_clopts.py::TestTsharkCaptureClopts::test_tshark_invalid_capfilter”. + +[#TestsPytest] +==== pytest fixtures + +A test has typically additional dependencies, like the path to an +executable, the path to a capture file, a configuration directory, the +availability of an optional library, and so on. + +https://pytest.org/[pytest] is a test framework which has full +parallelization support (test-level instead of just suite-level), +provides nice test reports, and allows +https://docs.pytest.org/en/latest/fixture.html[modular fixtures]. + +A fixture is a function decorated with `@pytest.fixture` and can +either call `pytest.skip("reason")` to skip tests that depend on the +fixture, or return/yield a value. +Test functions (and other fixture functions) can receive the fixture +value by using the name of the fixture function as function parameters. +Common fixtures are available in `fixtures_ws.py` and includes +`cmd_tshark` for the path to the `tshark` executable and `capture_file` +for a factory function that produces the path to a capture file. + +[#ChTestsRun] +=== Listing And Running Tests + +Tests are run with https://pytest.org/[pytest]. Pytest features versus the +"unittest" standard library module include finer +test selection, full parallelism, nicer test execution summaries, better output +in case of failures (containing the contents of variables) and the ability to +open the PDB debugger on failing tests. + +To get started, install pytest 3.0 or newer and +https://pypi.org/project/pytest-xdist/[pytest-xdist]: + +[source,sh] +---- +# Install required packages on Ubuntu 18.04 or Debian jessie-backports +$ sudo apt install python3-pytest python3-pytest-xdist + +# Install required packages on other systems +$ pip install pytest pytest-xdist +---- + +Run `pytest` in the Wireshark build directory, Wireshark binaries are assumed to +be present in the `run` subdirectory (or `run\RelWithDebInfo` on Windows). + +[source,sh] +---- +# Run all tests +$ cd /path/to/wireshark/build +$ pytest + +# Run all except capture tests +$ pytest --disable-capture + +# Run all tests with "decryption" in its name +$ pytest -k decryption + +# Run all tests with an explicit path to the Wireshark executables +$ pytest --program-path /path/to/wireshark/build/run +---- + +To list tests without actually executing them, use the `--collect-only` option: + +[source,sh] +---- +# List all tests +$ pytest --collect-only + +# List only tests containing both "dfilter" and "tvb" +$ pytest --collect-only -k "dfilter and tvb" +---- + +The test suite will fail tests when programs are missing. When only a +subset of programs are built or when some programs are disabled, then +the test suite can be instructed to skip instead of fail tests: + +[source,sh] +---- +# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF) +$ pytest --skip-missing-programs dumpcap,rawshark + +# Run tests and ignore all tests with missing program dependencies +$ pytest --skip-missing-programs all +---- + +To open a Python debugger (PDB) on failing tests, use the `--pdb` option and +disable parallelism with the `-n0` option: + +[source,sh] +---- +# Run decryption tests sequentially and open a debugger on failing tests +$ pytest -n0 --pdb -k decryption +---- + +[#ChTestsDevelop] +=== Adding Or Modifying Tests + +Tests must be in a Python module whose name matches “suite_*.py”. The +module must contain one or more subclasses with a name starting with +"Test" something, for example "class TestDissectionHttp2:". Each test case +method whose name starts with “test_” constitutes an individual test. + +Success or failure conditions are signalled using regular assertions +with the "assert" Python keyword. + +Test dependencies (such as programs, directories, or the environment +variables) are injected through method parameters. Commonly used +fixtures include `cmd_tshark` and `capture_file`. + +Processes (tshark, capinfos, etc.) are run using the "subprocess" Python module, +or the Wireshark `subprocesstest` module with some convenience functions. +Possible functions include `subprocesstest.run()`, `subprocesstest.check_run()` +or creating `subprocess.Popen` object if the utility functions are not sufficient for some reason. +Usually this is only required if two-way communication is performed with +the child process. `subprocesstest.check_run()` is exactly the same as +calling `subprocesstest.run()` with `check=True` as an argument, only +a bit more expressive. + +Check the documentation for the Python subprocess module for a full description +of the arguments available to the `subprocesstest.run()` convenience wrapper +and the `subprocess.Popen` object. + +All of the current tests run one or more of Wireshark's suite of +executables and either check their return code or their output. A +simple example is “suite_clopts.py::TestBasicClopts::test_existing_file”, +which reads a capture file using TShark and checks its exit code. + +[source,python] +---- +import subprocesstest +import pytest + +class TestBasicClopts: + def test_existing_file(self, cmd_tshark, capture_file, test_env): + subprocess.check_run((cmd_tshark, '-r', capture_file('dhcp.pcap')), env=test_env) +---- + +Output can be checked using `assert subprocesstest.grep_output()`, +`assert subprocesstest.count_output()` or any other `assert` statement. +`subprocesstest.check_run()` also asserts that the child process returns +the value 0 as exit code. + +[source,python] +---- +import subprocesstest +import pytest + +class TestDecrypt80211: + def test_80211_wpa_psk(self, cmd_tshark, capture_file, test_env): + tshark_proc = subprocesstest.run((cmd_tshark, + '-o', 'wlan.enable_decryption: TRUE', + '-Tfields', + '-e', 'http.request.uri', + '-r', capture_file('wpa-Induction.pcap.gz'), + '-Y', 'http', + ), capture_output=True, env=test_env) + assert 'favicon.ico' in tshark_proc.stdout +---- + +Tests can be run in parallel. This means that any files you create must +be unique for each test. Filenames based on the current test name are +generated using fixtures such as "capture_file" and "result_file". By default +pytest generates paths in the system's temporary directory and the last three +pytest runs are kept. Temporary files from older runs are automatically deleted. + +[#ChTestsExternal] +=== External Tests + +You can create your own Python test files outside of the Wireshark source tree. +To include your tests when running the Wireshark test suite, simply add the +directory containing your test files to the `pytest` command line. Note that +filenames must match the same conventions as discussed above. + +In order for your tests to have access to the Wireshark test fixtures, you will +need this line in each test file: + +[source,python] +---- +from fixtures_ws import * +---- + +[#ChTestsExtFixtures] +==== Custom Fixtures + +You may wish to define your own test fixtures -- for example, a fixture similar +to `capture_file` but which gives the path to a file in your external test +directory. Here is an example Python file containing such a fixture. It presumes +a subdirectory named `extra_captures` which exists in the same directory, and +which contains your extra capture files. + +[source,python] +---- +# my_fixtures.py +# To use in your own tests, import like so: +# from my_fixtures import * + +from pathlib import Path +import pytest + +@pytest.fixture(scope='session') +def extra_file(): + def resolver(filename): + return Path(__file__).parent.joinpath("extra_captures", filename) + return resolver +---- + +NOTE: If you give your fixture the same name as an existing Wireshark fixture, +any tests using your fixture library will lose access to the Wireshark fixture +of the same name. This can lead to confusing behavior and is not recommended. |