diff options
Diffstat (limited to 'docbook/wsdg_src/wsdg_tests.adoc')
| -rw-r--r-- | docbook/wsdg_src/wsdg_tests.adoc | 298 |
1 files changed, 0 insertions, 298 deletions
diff --git a/docbook/wsdg_src/wsdg_tests.adoc b/docbook/wsdg_src/wsdg_tests.adoc deleted file mode 100644 index a9435ca7..00000000 --- a/docbook/wsdg_src/wsdg_tests.adoc +++ /dev/null @@ -1,298 +0,0 @@ -// WSDG Chapter Tests - -[#ChapterTests] -== Wireshark Tests - -The Wireshark sources include a collection of Python scripts that test -the features of Wireshark, TShark, Dumpcap, and other programs that -accompany Wireshark. These are located in the `test` directory of the -Wireshark source tree. - -The command line options of Wireshark and its companion command line -tools are numerous. These tests help to ensure that we don't introduce -bugs as Wireshark grows and evolves. - -[#TestsQuickStart] -=== Quick Start - -The recommended steps to prepare for and run tests with a UN*X toolchain: - -* Install two Python packages, pytest: `pip install pytest pytest-xdist` -* Build programs (“wireshark”, “tshark”, etc.): `ninja` -* Build additional programs for the “unittests” suite: `ninja test-programs` -* Run tests in the build directory: `pytest` - -Replace `ninja` by `make` as needed. - -If building with <<#ChWindowsBuild,Microsoft Visual Studio>> the analogous steps are: - -* Install pytest Python packages: `python -m pip install pytest pytest-xdist` -* Build programs: `msbuild /m /p:Configuration=RelWithDebInfo Wireshark.sln` -* Build test-programs: `msbuild /m /p:Configuration=RelWithDebInfo test-programs.vcxproj` -* Run tests: `python -m pytest` - -TIP: Depending on your PATH, you may need to run the pytest module as a -script from your Python interpreter, e.g, `python -m pytest` or -`python3 -m pytest` instead of `pytest`. - -The test suite will attempt to test as much as possible and skip tests -when its dependencies are not satisfied. For example, packet capture -tests require a Loopback interface and capture privileges. To avoid -capture tests, pass the `--disable-capture` option. - -List available tests with `pytest --collectonly`. Enable verbose output -with `pytest --verbose`. For more details, see <<ChTestsRun>>. - -You can also run the "ninja test" target instead of invoking pytest -directly. This will automatically build the test programs dependency, -so it may be preferred for that reason. - -[#ChTestsStructure] -=== Test suite structure - -The following sections describes how the test suite is organized. - -[#TestCoverage] -==== Test Coverage And Availability - -The testing framework can run programs and check their stdout, stderr, -and exit codes. It cannot interact with the Wireshark UI. Tests cover -capture, command line options, decryption, file format support and -conversion, Lua scripting, and other functionality. - -Available tests depend on the libraries with which Wireshark was built. -For example, some decryption tests depend on a minimum version of -Libgcrypt and Lua tests depend on Lua. - -Capture tests depend on the permissions of the user running the test -script. We assume that the test user has capture permissions on Windows -and macOS and capture tests are enabled by default on those platforms. - -TIP: Build the "test-capture" target on Linux (using sudo) to set dumpcap -permissions and enable capture tests. - -If a feature is unavailable, the test will be skipped. For example, if -an old version of Libgcrypt is in use, then some decryption tests will -be skipped while other tests can still run to completion. - -[#TestsLayout] -==== Suites, Cases, and Tests - -The test suite uses pytest as a test runner. Tests are organized according to -suites, cases, and individual tests. Suites correspond to Python modules -that match the pattern “suite_*.py”. Cases correspond to one or more -classes in each module, and case class methods matching the pattern -”test_*” correspond to individual tests. For example, the invalid -capture filter test in the TShark capture command line options test case -in the command line options suite has the ID -“suite_clopts.py::TestTsharkCaptureClopts::test_tshark_invalid_capfilter”. - -[#TestsPytest] -==== pytest fixtures - -A test has typically additional dependencies, like the path to an -executable, the path to a capture file, a configuration directory, the -availability of an optional library, and so on. - -https://pytest.org/[pytest] is a test framework which has full -parallelization support (test-level instead of just suite-level), -provides nice test reports, and allows -https://docs.pytest.org/en/latest/fixture.html[modular fixtures]. - -A fixture is a function decorated with `@pytest.fixture` and can -either call `pytest.skip("reason")` to skip tests that depend on the -fixture, or return/yield a value. -Test functions (and other fixture functions) can receive the fixture -value by using the name of the fixture function as function parameters. -Common fixtures are available in `fixtures_ws.py` and includes -`cmd_tshark` for the path to the `tshark` executable and `capture_file` -for a factory function that produces the path to a capture file. - -[#ChTestsRun] -=== Listing And Running Tests - -Tests are run with https://pytest.org/[pytest]. Pytest features versus the -"unittest" standard library module include finer -test selection, full parallelism, nicer test execution summaries, better output -in case of failures (containing the contents of variables) and the ability to -open the PDB debugger on failing tests. - -To get started, install pytest 3.0 or newer and -https://pypi.org/project/pytest-xdist/[pytest-xdist]: - -[source,sh] ----- -# Install required packages on Ubuntu 18.04 or Debian jessie-backports -$ sudo apt install python3-pytest python3-pytest-xdist - -# Install required packages on other systems -$ pip install pytest pytest-xdist ----- - -Run `pytest` in the Wireshark build directory, Wireshark binaries are assumed to -be present in the `run` subdirectory (or `run\RelWithDebInfo` on Windows). - -[source,sh] ----- -# Run all tests -$ cd /path/to/wireshark/build -$ pytest - -# Run all except capture tests -$ pytest --disable-capture - -# Run all tests with "decryption" in its name -$ pytest -k decryption - -# Run all tests with an explicit path to the Wireshark executables -$ pytest --program-path /path/to/wireshark/build/run ----- - -To list tests without actually executing them, use the `--collect-only` option: - -[source,sh] ----- -# List all tests -$ pytest --collect-only - -# List only tests containing both "dfilter" and "tvb" -$ pytest --collect-only -k "dfilter and tvb" ----- - -The test suite will fail tests when programs are missing. When only a -subset of programs are built or when some programs are disabled, then -the test suite can be instructed to skip instead of fail tests: - -[source,sh] ----- -# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF) -$ pytest --skip-missing-programs dumpcap,rawshark - -# Run tests and ignore all tests with missing program dependencies -$ pytest --skip-missing-programs all ----- - -To open a Python debugger (PDB) on failing tests, use the `--pdb` option and -disable parallelism with the `-n0` option: - -[source,sh] ----- -# Run decryption tests sequentially and open a debugger on failing tests -$ pytest -n0 --pdb -k decryption ----- - -[#ChTestsDevelop] -=== Adding Or Modifying Tests - -Tests must be in a Python module whose name matches “suite_*.py”. The -module must contain one or more subclasses with a name starting with -"Test" something, for example "class TestDissectionHttp2:". Each test case -method whose name starts with “test_” constitutes an individual test. - -Success or failure conditions are signalled using regular assertions -with the "assert" Python keyword. - -Test dependencies (such as programs, directories, or the environment -variables) are injected through method parameters. Commonly used -fixtures include `cmd_tshark` and `capture_file`. - -Processes (tshark, capinfos, etc.) are run using the "subprocess" Python module, -or the Wireshark `subprocesstest` module with some convenience functions. -Possible functions include `subprocesstest.run()`, `subprocesstest.check_run()` -or creating `subprocess.Popen` object if the utility functions are not sufficient for some reason. -Usually this is only required if two-way communication is performed with -the child process. `subprocesstest.check_run()` is exactly the same as -calling `subprocesstest.run()` with `check=True` as an argument, only -a bit more expressive. - -Check the documentation for the Python subprocess module for a full description -of the arguments available to the `subprocesstest.run()` convenience wrapper -and the `subprocess.Popen` object. - -All of the current tests run one or more of Wireshark's suite of -executables and either check their return code or their output. A -simple example is “suite_clopts.py::TestBasicClopts::test_existing_file”, -which reads a capture file using TShark and checks its exit code. - -[source,python] ----- -import subprocesstest -import pytest - -class TestBasicClopts: - def test_existing_file(self, cmd_tshark, capture_file, test_env): - subprocess.check_run((cmd_tshark, '-r', capture_file('dhcp.pcap')), env=test_env) ----- - -Output can be checked using `assert subprocesstest.grep_output()`, -`assert subprocesstest.count_output()` or any other `assert` statement. -`subprocesstest.check_run()` also asserts that the child process returns -the value 0 as exit code. - -[source,python] ----- -import subprocesstest -import pytest - -class TestDecrypt80211: - def test_80211_wpa_psk(self, cmd_tshark, capture_file, test_env): - tshark_proc = subprocesstest.run((cmd_tshark, - '-o', 'wlan.enable_decryption: TRUE', - '-Tfields', - '-e', 'http.request.uri', - '-r', capture_file('wpa-Induction.pcap.gz'), - '-Y', 'http', - ), capture_output=True, env=test_env) - assert 'favicon.ico' in tshark_proc.stdout ----- - -Tests can be run in parallel. This means that any files you create must -be unique for each test. Filenames based on the current test name are -generated using fixtures such as "capture_file" and "result_file". By default -pytest generates paths in the system's temporary directory and the last three -pytest runs are kept. Temporary files from older runs are automatically deleted. - -[#ChTestsExternal] -=== External Tests - -You can create your own Python test files outside of the Wireshark source tree. -To include your tests when running the Wireshark test suite, simply add the -directory containing your test files to the `pytest` command line. Note that -filenames must match the same conventions as discussed above. - -In order for your tests to have access to the Wireshark test fixtures, you will -need this line in each test file: - -[source,python] ----- -from fixtures_ws import * ----- - -[#ChTestsExtFixtures] -==== Custom Fixtures - -You may wish to define your own test fixtures -- for example, a fixture similar -to `capture_file` but which gives the path to a file in your external test -directory. Here is an example Python file containing such a fixture. It presumes -a subdirectory named `extra_captures` which exists in the same directory, and -which contains your extra capture files. - -[source,python] ----- -# my_fixtures.py -# To use in your own tests, import like so: -# from my_fixtures import * - -from pathlib import Path -import pytest - -@pytest.fixture(scope='session') -def extra_file(): - def resolver(filename): - return Path(__file__).parent.joinpath("extra_captures", filename) - return resolver ----- - -NOTE: If you give your fixture the same name as an existing Wireshark fixture, -any tests using your fixture library will lose access to the Wireshark fixture -of the same name. This can lead to confusing behavior and is not recommended. |