summaryrefslogtreecommitdiffstats
path: root/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-27 10:05:51 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-27 10:05:51 +0000
commit5d1646d90e1f2cceb9f0828f4b28318cd0ec7744 (patch)
treea94efe259b9009378be6d90eb30d2b019d95c194 /tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
parentInitial commit. (diff)
downloadlinux-upstream.tar.xz
linux-upstream.zip
Adding upstream version 5.10.209.upstream/5.10.209upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
-rw-r--r--tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt105
1 files changed, 105 insertions, 0 deletions
diff --git a/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt b/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
new file mode 100644
index 000000000..a28571aff
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
@@ -0,0 +1,105 @@
+tdc - Adding test cases for tdc
+
+Author: Lucas Bates - lucasb@mojatatu.com
+
+ADDING TEST CASES
+-----------------
+
+User-defined tests should be added by defining a separate JSON file. This
+will help prevent conflicts when updating the repository. Refer to
+template.json for the required JSON format for test cases.
+
+Include the 'id' field, but do not assign a value. Running tdc with the -i
+option will generate a unique ID for that test case.
+
+tdc will recursively search the 'tc-tests' subdirectory (or the
+directories named with the -D option) for .json files. Any test case
+files you create in these directories will automatically be included.
+If you wish to store your custom test cases elsewhere, be sure to run
+tdc with the -f argument and the path to your file, or the -D argument
+and the path to your directory(ies).
+
+Be aware of required escape characters in the JSON data - particularly
+when defining the match pattern. Refer to the supplied json test files
+for examples when in doubt. The match pattern is written in json, and
+will be used by python. So the match pattern will be a python regular
+expression, but should be written using json syntax.
+
+
+TEST CASE STRUCTURE
+-------------------
+
+Each test case has required data:
+
+id: A unique alphanumeric value to identify a particular test case
+name: Descriptive name that explains the command under test
+skip: A completely optional key, if the corresponding value is "yes"
+ then tdc will not execute the test case in question. However,
+ this test case will still appear in the results output but
+ marked as skipped. This key can be placed anywhere inside the
+ test case at the top level.
+category: A list of single-word descriptions covering what the command
+ under test is testing. Example: filter, actions, u32, gact, etc.
+setup: The list of commands required to ensure the command under test
+ succeeds. For example: if testing a filter, the command to create
+ the qdisc would appear here.
+ This list can be empty.
+ Each command can be a string to be executed, or a list consisting
+ of a string which is a command to be executed, followed by 1 or
+ more acceptable exit codes for this command.
+ If only a string is given for the command, then an exit code of 0
+ will be expected.
+cmdUnderTest: The tc command being tested itself.
+expExitCode: The code returned by the command under test upon its termination.
+ tdc will compare this value against the actual returned value.
+verifyCmd: The tc command to be run to verify successful execution.
+ For example: if the command under test creates a gact action,
+ verifyCmd should be "$TC actions show action gact"
+matchPattern: A regular expression to be applied against the output of the
+ verifyCmd to prove the command under test succeeded. This pattern
+ should be as specific as possible so that a false positive is not
+ matched.
+matchCount: How many times the regex in matchPattern should match. A value
+ of 0 is acceptable.
+teardown: The list of commands to clean up after the test is completed.
+ The environment should be returned to the same state as when
+ this test was started: qdiscs deleted, actions flushed, etc.
+ This list can be empty.
+ Each command can be a string to be executed, or a list consisting
+ of a string which is a command to be executed, followed by 1 or
+ more acceptable exit codes for this command.
+ If only a string is given for the command, then an exit code of 0
+ will be expected.
+
+
+SETUP/TEARDOWN ERRORS
+---------------------
+
+If an error is detected during the setup/teardown process, execution of the
+tests will immediately stop with an error message and the namespace in which
+the tests are run will be destroyed. This is to prevent inaccurate results
+in the test cases. tdc will output a series of TAP results for the skipped
+tests.
+
+Repeated failures of the setup/teardown may indicate a problem with the test
+case, or possibly even a bug in one of the commands that are not being tested.
+
+It's possible to include acceptable exit codes with the setup/teardown command
+so that it doesn't halt the script for an error that doesn't matter. Turn the
+individual command into a list, with the command being first, followed by all
+acceptable exit codes for the command.
+
+Example:
+
+A pair of setup commands. The first can have exit code 0, 1 or 255, the
+second must have exit code 0.
+
+ "setup": [
+ [
+ "$TC actions flush action gact",
+ 0,
+ 1,
+ 255
+ ],
+ "$TC actions add action reclassify index 65536"
+ ],