// WSDG Chapter Tests [#ChapterTests] == Wireshark Tests The Wireshark sources include a collection of Python scripts that test the features of Wireshark, TShark, Dumpcap, and other programs that accompany Wireshark. These are located in the `test` directory of the Wireshark source tree. The command line options of Wireshark and its companion command line tools are numerous. These tests help to ensure that we don't introduce bugs as Wireshark grows and evolves. [#TestsQuickStart] === Quick Start The recommended steps to prepare for and run tests with a UN*X toolchain: * Install two Python packages, pytest: `pip install pytest pytest-xdist` * Build programs (“wireshark”, “tshark”, etc.): `ninja` * Build additional programs for the “unittests” suite: `ninja test-programs` * Run tests in the build directory: `pytest` Replace `ninja` by `make` as needed. If building with <<#ChWindowsBuild,Microsoft Visual Studio>> the analogous steps are: * Install pytest Python packages: `python -m pip install pytest pytest-xdist` * Build programs: `msbuild /m /p:Configuration=RelWithDebInfo Wireshark.sln` * Build test-programs: `msbuild /m /p:Configuration=RelWithDebInfo test-programs.vcxproj` * Run tests: `python -m pytest` TIP: Depending on your PATH, you may need to run the pytest module as a script from your Python interpreter, e.g, `python -m pytest` or `python3 -m pytest` instead of `pytest`. The test suite will attempt to test as much as possible and skip tests when its dependencies are not satisfied. For example, packet capture tests require a Loopback interface and capture privileges. To avoid capture tests, pass the `--disable-capture` option. List available tests with `pytest --collectonly`. Enable verbose output with `pytest --verbose`. For more details, see <>. You can also run the "ninja test" target instead of invoking pytest directly. This will automatically build the test programs dependency, so it may be preferred for that reason. [#ChTestsStructure] === Test suite structure The following sections describes how the test suite is organized. [#TestCoverage] ==== Test Coverage And Availability The testing framework can run programs and check their stdout, stderr, and exit codes. It cannot interact with the Wireshark UI. Tests cover capture, command line options, decryption, file format support and conversion, Lua scripting, and other functionality. Available tests depend on the libraries with which Wireshark was built. For example, some decryption tests depend on a minimum version of Libgcrypt and Lua tests depend on Lua. Capture tests depend on the permissions of the user running the test script. We assume that the test user has capture permissions on Windows and macOS and capture tests are enabled by default on those platforms. TIP: Build the "test-capture" target on Linux (using sudo) to set dumpcap permissions and enable capture tests. If a feature is unavailable, the test will be skipped. For example, if an old version of Libgcrypt is in use, then some decryption tests will be skipped while other tests can still run to completion. [#TestsLayout] ==== Suites, Cases, and Tests The test suite uses pytest as a test runner. Tests are organized according to suites, cases, and individual tests. Suites correspond to Python modules that match the pattern “suite_*.py”. Cases correspond to one or more classes in each module, and case class methods matching the pattern ”test_*” correspond to individual tests. For example, the invalid capture filter test in the TShark capture command line options test case in the command line options suite has the ID “suite_clopts.py::TestTsharkCaptureClopts::test_tshark_invalid_capfilter”. [#TestsPytest] ==== pytest fixtures A test has typically additional dependencies, like the path to an executable, the path to a capture file, a configuration directory, the availability of an optional library, and so on. https://pytest.org/[pytest] is a test framework which has full parallelization support (test-level instead of just suite-level), provides nice test reports, and allows https://docs.pytest.org/en/latest/fixture.html[modular fixtures]. A fixture is a function decorated with `@pytest.fixture` and can either call `pytest.skip("reason")` to skip tests that depend on the fixture, or return/yield a value. Test functions (and other fixture functions) can receive the fixture value by using the name of the fixture function as function parameters. Common fixtures are available in `fixtures_ws.py` and includes `cmd_tshark` for the path to the `tshark` executable and `capture_file` for a factory function that produces the path to a capture file. [#ChTestsRun] === Listing And Running Tests Tests are run with https://pytest.org/[pytest]. Pytest features versus the "unittest" standard library module include finer test selection, full parallelism, nicer test execution summaries, better output in case of failures (containing the contents of variables) and the ability to open the PDB debugger on failing tests. To get started, install pytest 3.0 or newer and https://pypi.org/project/pytest-xdist/[pytest-xdist]: [source,sh] ---- # Install required packages on Ubuntu 18.04 or Debian jessie-backports $ sudo apt install python3-pytest python3-pytest-xdist # Install required packages on other systems $ pip install pytest pytest-xdist ---- Run `pytest` in the Wireshark build directory, Wireshark binaries are assumed to be present in the `run` subdirectory (or `run\RelWithDebInfo` on Windows). [source,sh] ---- # Run all tests $ cd /path/to/wireshark/build $ pytest # Run all except capture tests $ pytest --disable-capture # Run all tests with "decryption" in its name $ pytest -k decryption # Run all tests with an explicit path to the Wireshark executables $ pytest --program-path /path/to/wireshark/build/run ---- To list tests without actually executing them, use the `--collect-only` option: [source,sh] ---- # List all tests $ pytest --collect-only # List only tests containing both "dfilter" and "tvb" $ pytest --collect-only -k "dfilter and tvb" ---- The test suite will fail tests when programs are missing. When only a subset of programs are built or when some programs are disabled, then the test suite can be instructed to skip instead of fail tests: [source,sh] ---- # Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF) $ pytest --skip-missing-programs dumpcap,rawshark # Run tests and ignore all tests with missing program dependencies $ pytest --skip-missing-programs all ---- To open a Python debugger (PDB) on failing tests, use the `--pdb` option and disable parallelism with the `-n0` option: [source,sh] ---- # Run decryption tests sequentially and open a debugger on failing tests $ pytest -n0 --pdb -k decryption ---- [#ChTestsDevelop] === Adding Or Modifying Tests Tests must be in a Python module whose name matches “suite_*.py”. The module must contain one or more subclasses with a name starting with "Test" something, for example "class TestDissectionHttp2:". Each test case method whose name starts with “test_” constitutes an individual test. Success or failure conditions are signalled using regular assertions with the "assert" Python keyword. Test dependencies (such as programs, directories, or the environment variables) are injected through method parameters. Commonly used fixtures include `cmd_tshark` and `capture_file`. Processes (tshark, capinfos, etc.) are run using the "subprocess" Python module, or the Wireshark `subprocesstest` module with some convenience functions. Possible functions include `subprocesstest.run()`, `subprocesstest.check_run()` or creating `subprocess.Popen` object if the utility functions are not sufficient for some reason. Usually this is only required if two-way communication is performed with the child process. `subprocesstest.check_run()` is exactly the same as calling `subprocesstest.run()` with `check=True` as an argument, only a bit more expressive. Check the documentation for the Python subprocess module for a full description of the arguments available to the `subprocesstest.run()` convenience wrapper and the `subprocess.Popen` object. All of the current tests run one or more of Wireshark's suite of executables and either check their return code or their output. A simple example is “suite_clopts.py::TestBasicClopts::test_existing_file”, which reads a capture file using TShark and checks its exit code. [source,python] ---- import subprocesstest import pytest class TestBasicClopts: def test_existing_file(self, cmd_tshark, capture_file, test_env): subprocess.check_run((cmd_tshark, '-r', capture_file('dhcp.pcap')), env=test_env) ---- Output can be checked using `assert subprocesstest.grep_output()`, `assert subprocesstest.count_output()` or any other `assert` statement. `subprocesstest.check_run()` also asserts that the child process returns the value 0 as exit code. [source,python] ---- import subprocesstest import pytest class TestDecrypt80211: def test_80211_wpa_psk(self, cmd_tshark, capture_file, test_env): tshark_proc = subprocesstest.run((cmd_tshark, '-o', 'wlan.enable_decryption: TRUE', '-Tfields', '-e', 'http.request.uri', '-r', capture_file('wpa-Induction.pcap.gz'), '-Y', 'http', ), capture_output=True, env=test_env) assert 'favicon.ico' in tshark_proc.stdout ---- Tests can be run in parallel. This means that any files you create must be unique for each test. Filenames based on the current test name are generated using fixtures such as "capture_file" and "result_file". By default pytest generates paths in the system's temporary directory and the last three pytest runs are kept. Temporary files from older runs are automatically deleted. [#ChTestsExternal] === External Tests You can create your own Python test files outside of the Wireshark source tree. To include your tests when running the Wireshark test suite, simply add the directory containing your test files to the `pytest` command line. Note that filenames must match the same conventions as discussed above. In order for your tests to have access to the Wireshark test fixtures, you will need this line in each test file: [source,python] ---- from fixtures_ws import * ---- [#ChTestsExtFixtures] ==== Custom Fixtures You may wish to define your own test fixtures -- for example, a fixture similar to `capture_file` but which gives the path to a file in your external test directory. Here is an example Python file containing such a fixture. It presumes a subdirectory named `extra_captures` which exists in the same directory, and which contains your extra capture files. [source,python] ---- # my_fixtures.py # To use in your own tests, import like so: # from my_fixtures import * from pathlib import Path import pytest @pytest.fixture(scope='session') def extra_file(): def resolver(filename): return Path(__file__).parent.joinpath("extra_captures", filename) return resolver ---- NOTE: If you give your fixture the same name as an existing Wireshark fixture, any tests using your fixture library will lose access to the Wireshark fixture of the same name. This can lead to confusing behavior and is not recommended.