diff options
Diffstat (limited to 'tests/integration/deckard/doc')
| -rw-r--r-- | tests/integration/deckard/doc/devel_guide.rst | 6 | ||||
| -rw-r--r-- | tests/integration/deckard/doc/scenario_example.rst | 337 | ||||
| -rw-r--r-- | tests/integration/deckard/doc/scenario_guide.rst | 431 | ||||
| -rw-r--r-- | tests/integration/deckard/doc/user_guide.rst | 286 |
4 files changed, 1060 insertions, 0 deletions
diff --git a/tests/integration/deckard/doc/devel_guide.rst b/tests/integration/deckard/doc/devel_guide.rst new file mode 100644 index 0000000..bf2e607 --- /dev/null +++ b/tests/integration/deckard/doc/devel_guide.rst @@ -0,0 +1,6 @@ +Notes for Deckard developers +============================ + +libfaketime +^^^^^^^^^^^ +Run-time changes to ``FAKETIME_`` environment variables might not be picked up by running process if ``FAKETIME_NO_CACHE=1`` variable is not set before the process starts. diff --git a/tests/integration/deckard/doc/scenario_example.rst b/tests/integration/deckard/doc/scenario_example.rst new file mode 100644 index 0000000..7fb6f3e --- /dev/null +++ b/tests/integration/deckard/doc/scenario_example.rst @@ -0,0 +1,337 @@ +Scenario example +================= +iter_ns_badaa.rpl +:: + + ; config options + stub-addr: 193.0.14.129 # K.ROOT-SERVERS.NET. + CONFIG_END + + SCENARIO_BEGIN Test iterator with NS falsely declaring referral answer as authoritative. + + ; K.ROOT-SERVERS.NET. + RANGE_BEGIN 0 100 + ADDRESS 193.0.14.129 + ENTRY_BEGIN + MATCH opcode qtype qname + ADJUST copy_id + REPLY QR NOERROR + SECTION QUESTION + . IN NS + SECTION ANSWER + . IN NS K.ROOT-SERVERS.NET. + SECTION ADDITIONAL + K.ROOT-SERVERS.NET. IN A 193.0.14.129 + ENTRY_END + + ENTRY_BEGIN + MATCH opcode subdomain + ADJUST copy_id copy_query + ; False declaration here + REPLY QR AA NOERROR + SECTION QUESTION + MORECOWBELL. IN A + SECTION AUTHORITY + MORECOWBELL. IN NS a.gtld-servers.net. + SECTION ADDITIONAL + a.gtld-servers.net. IN A 192.5.6.30 + ENTRY_END + + ENTRY_BEGIN + MATCH opcode qtype qname + ADJUST copy_id copy_query + REPLY QR NOERROR + SECTION QUESTION + a.gtld-servers.net. IN A + SECTION ANSWER + a.gtld-servers.net. IN A 192.5.6.30 + ENTRY_END + + ENTRY_BEGIN + MATCH opcode qtype qname + ADJUST copy_id copy_query + REPLY QR NOERROR + SECTION QUESTION + a.gtld-servers.net. IN AAAA + SECTION AUTHORITY + . SOA bla bla 1 2 3 4 5 + ENTRY_END + + RANGE_END + + ; a.gtld-servers.net. + RANGE_BEGIN 0 100 + ADDRESS 192.5.6.30 + + ENTRY_BEGIN + MATCH opcode qtype qname + ADJUST copy_id copy_query + REPLY QR AA NOERROR + SECTION QUESTION + CATALYST.MORECOWBELL. IN A + SECTION ANSWER + CATALYST.MORECOWBELL. IN A 10.20.30.40 + SECTION AUTHORITY + CATALYST.MORECOWBELL. IN NS a.gtld-servers.net. + ENTRY_END + + RANGE_END + + STEP 1 QUERY + ENTRY_BEGIN + REPLY RD + SECTION QUESTION + catalyst.morecowbell. IN A + ENTRY_END + + ; recursion happens here. + STEP 10 CHECK_ANSWER + ENTRY_BEGIN + MATCH all + REPLY QR RD RA NOERROR + SECTION QUESTION + catalyst.morecowbell. IN A + SECTION ANSWER + catalyst.morecowbell. IN A 10.20.30.40 + ENTRY_END + + SCENARIO_END + +Execution flow : + +First, STEP 1 QUERY will be performed. + +Python sends query to Resolver +:: + + id 31296 + opcode QUERY + rcode NOERROR + flags RD + edns 1 + eflags + payload 1280 + ;QUESTION + catalyst.morecowbell. IN A + ;ANSWER + ;AUTHORITY + ;ADDITIONAL + +At this scenario stub-addr is set to 193.0.14.129, thus Resolver have been configured to use address +193.0.14.129 as a root server. So it sends query to Python fake server which listen at address 193.0.14.129 + +:: + + > [plan] plan 'catalyst.morecowbell.' type 'A' + [resl] => using root hints + [resl] => querying: '193.0.14.129' score: 10 zone cut: '.' m12n: 'CaTALYSt.MoReCoWBEll.' type: 'A' + + +:: + + id 7367 + opcode QUERY + rcode NOERROR + flags + edns 0 + payload 1452 + ;QUESTION + CaTALYSt.MoReCoWBEll. IN A + ;ANSWER + ;AUTHORITY + ;ADDITIONAL + +Python fake server starts range analyzing to make answer. +Let's look at first range +:: + + RANGE_BEGIN 0 100 + ADDRESS 193.0.14.129 + +STEP ID is equal 1, so it matches the condition n1 <= step id <= n2 +Next, ADDRESS field is equal to 193.0.14.129. Since query was directed +specifically to 193.0.14.129, this range will be used. + +Next, Python walks through list of entries to choose eligible entry. +First entry at this range requires comparison of "opcode qtype qname" field list. +Since opcode is QUERY, first comparison is true. +Next, qtype field at question section is equal NS. +But qtype field at question section of incoming query is A. +So this comparison failed and this entry will be rejected. + +Next entry requires comparison of opcode and subdomain fields. +As we seen, opcode matches. +Let's look at domain names. +ENTRY datablock: +:: + + SECTION QUESTION + MORECOWBELL. IN A + +Incoming query : +:: + + ;QUESTION + CaTALYSt.MoReCoWBEll. IN A + +So, subdomain matches and second entry of first range used as answer pattern. +Python fake server sends answer to Resolver : +:: + + id 7367 + opcode QUERY + rcode NOERROR + flags QR AA + edns 0 + payload 1280 + ;QUESTION + CaTALYSt.MoReCoWBEll. IN A + ;ANSWER + ;AUTHORITY + MORECOWBELL. 3600 IN NS a.gtld-servers.net. + ;ADDITIONAL + a.gtld-servers.net. 3600 IN A 192.5.6.30 + +Note that additional section contains IP address. Because new address is found, +Python fake server immediately starts listening on this address. +Resolver sends next query to 192.5.6.30: + +:: + + [iter] <= referral response, follow + [ pc ] => answer cached for TTL=900 + [resl] => querying: '192.5.6.30' score: 10 zone cut: 'morecowbell.' m12n: 'catalyst.mOREcoWBEll.' type: 'A' + + +:: + + id 58167 + opcode QUERY + rcode NOERROR + flags + edns 0 + payload 1452 + ;QUESTION + catalyst.mOREcoWBEll. IN A + ;ANSWER + ;AUTHORITY + ;ADDITIONAL + +Since query is directed to 192.5.6.30, +this range will be analyzed : +:: + + ; a.gtld-servers.net. + RANGE_BEGIN 0 100 + ADDRESS 192.5.6.30 + +It has a single entry, which requires "opcode qtype qname" field list to be compared. +Opcode and qtype fields are the same as fields in incoming query. +Let's compare qname. +ENTRY datablock : +:: + + SECTION QUESTION + CATALYST.MORECOWBELL. IN A + +Incoming query : +:: + + ;QUESTION + catalyst.mOREcoWBEll. IN A + +So, qname also the same. All fields matches and Python server sends answer +derived from this entry : +:: + + id 58167 + opcode QUERY + rcode NOERROR + flags QR AA + edns 0 + payload 1280 + ;QUESTION + cAtaLyst.MoRECowBEll. IN A + ;ANSWER + CATALYST.MORECOWBELL. 3600 IN A 10.20.30.40 + ;AUTHORITY + CATALYST.MORECOWBELL. 3600 IN NS a.gtld-servers.net. + ;ADDITIONAL + +Here Python found new address 10.20.30.40 and starts listening. +Next queries and answers : + +:: + + [iter] <= referral response, follow + [plan] plan 'a.gtld-servers.net.' type 'AAAA' + [resl] => using root hints + [resl] => querying: '193.0.14.129' score: 54 zone cut: '.' m12n: 'A.Gtld-sERverS.nEt.' type: 'AAAA' + + +query; Resolver ---> Python (193.0.14.129) +:: + + id 13810 + opcode QUERY + rcode NOERROR + flags + edns 0 + payload 1452 + ;QUESTION + A.Gtld-sERverS.nEt. IN AAAA + ;ANSWER + ;AUTHORITY + ;ADDITIONAL + +answer; Python ---> Resolver +:: + + id 13810 + opcode QUERY + rcode NOERROR + flags QR + edns 0 + payload 1280 + ;QUESTION + A.gTld-serveRS.NET. IN AAAA + ;ANSWER + ;AUTHORITY + . 3600 IN SOA bla. bla. 1 2 3 4 5 + ;ADDITIONAL + + +at this point Resolver returns answer to query from STEP 1 QUERY. + +:: + + [iter] <= rcode: NOERROR + [ pc ] => answer cached for TTL=900 + [ rc ] => satisfied from cache + [iter] <= rcode: NOERROR + [resl] finished: 4, queries: 2, mempool: 16400 B + + +:: + + opcode QUERY + rcode NOERROR + flags QR RD RA + edns 0 + payload 4096 + ;QUESTION + catalyst.morecowbell. IN A + ;ANSWER + catalyst.morecowbell. 3600 IN A 10.20.30.40 + ;AUTHORITY + ;ADDITIONAL + +Now STEP 10 will be performed. Is has a single entry which contains +**MATCH all** clause. **MATCH all** means set of dns flags must be equal and +all sections presented in ENTRY must be equal to ones in answer. +Incoming answer has next flags were set: **QR RD AA**. ENTRY datablock contains +**REPLY QR RD RA NOERROR** clause. As we see, flags set is equal. Also, we can +see equality of question and answer sections of both dns messages. + +So, Python got expected answer and test is passed. + diff --git a/tests/integration/deckard/doc/scenario_guide.rst b/tests/integration/deckard/doc/scenario_guide.rst new file mode 100644 index 0000000..e588968 --- /dev/null +++ b/tests/integration/deckard/doc/scenario_guide.rst @@ -0,0 +1,431 @@ +Deckard scenario guide +====================== +.. contents:: + +On the highest level, Deckard scenario consists of three parts (in this order): + +- scenario-specific configuration in the header, +- declarative description of the simulated network environment, +- sequence of test steps. + +The scenario is stored as ASCII encoded text file with following structure: + +.. code-block:: + + ; configuration part starts on beginning of the file + ; comments start with semicolon + CONFIG_END + + SCENARIO_BEGIN ; SCENARIO block combines declarative description and sequence of steps + + + ; declarative description of network environment starts here + RANGE_BEGIN a b + ENTRY_BEGIN + ; entries inside RANGE block describe DNS messages + ; used as answers from simulated network + ENTRY_END + RANGE_END + + + ; sequence of test steps begins here + STEP x QUERY ; this is step number x + ENTRY_BEGIN + ; ENTRY inside STEP may describe DNS message sent as query + ENTRY_END + + STEP y CHECK_ANSWER ; arbitrary number of steps is allowed + ENTRY_BEGIN + ; also, ENTRY inside STEP may describe DNS message with expected answer + ENTRY_END + SCENARIO_END + +The scenario is processed as follows: + +- Deckard parses configuration block and generates configuration files for binaries under test +- binaries are executed in an isolated network environment +- Deckard walks through all ``STEP`` blocks and sends queries to the binary under test, and checks answers it receives +- when a binary attempts to contact another server, Deckard intercepts the communication and replies with scripted answer as defined in ``RANGE`` blocks + +To better understand this structure, we will walk-through from sequential steps through declarative description up to scenario-specific configuration. + +Scenario +-------- +Scenario part starts with ``SCENARIO_BEGIN`` and ends with ``SCENARIO_END`` statements, which are present after ``CONFIG_END`` keyword. ``SCENARIO_BEGIN`` keyword must be followed by scenario description: + +.. code-block:: + + SCENARIO_BEGIN Test basic query minimization www.example.com. + ... + SCENARIO_END + + + +Test steps (``STEP``) +--------------------- +One ``STEP`` describes one action during scenario execution. It might be action like send next query to binary under test, send reply to binary under test, change faked system time, or check the last answer. Sequence of two steps might look like this: + +.. code-block:: + + STEP 1 QUERY ; send query specified in the following ENTRY to the binary + ENTRY_BEGIN ; ENTRY defines content of DNS message + REPLY RD + SECTION QUESTION + www.example.com. IN A + ENTRY_END + + STEP 10 CHECK_ANSWER ; check that answer to the previous query matches following ENTRY + ENTRY_BEGIN + MATCH all ; MATCH specifies what fields in answer have to match the ENTRY + REPLY QR RD RA NOERROR + SECTION QUESTION + www.example.com. IN A + SECTION ANSWER + www.example.com. IN CNAME www.next.com. + www.next.com. IN A 10.20.30.40 + SECTION AUTHORITY + SECTION ADDITIONAL + ENTRY_END + + +Most important parts of a step are: + +- id - number specifying order in which steps are executed, e.g. ``1`` or ``10`` +- type - action to execute, e.g. ``QUERY`` or ``CHECK_ANSWER`` +- entry - DNS message content, while meaning of the message depends on the step *type* + +One ``STEP`` block starts with ``STEP`` keyword and continues until one of {``STEP``, +``RANGE``, ``END_SCENARIO``} keywords is found. + +Format +^^^^^^ + +.. code-block:: + + STEP id type [additional data] + +- id - step identifier, a positive integer value; all steps must have + different id's. This value used within RANGE block, see above. +- type - step type; can be ``QUERY`` | ``REPLY`` | ``CHECK_ANSWER`` | ``TIME_PASSES ELAPSE`` *seconds* + + - QUERY - send query defined by associated ``ENTRY`` to binary under test + - CHECK_ANSWER - check if last received answer matches associated ``ENTRY`` + - TIME_PASSES ELAPSE - move faked system time for binary under test by number of *seconds* to future + - REPLY - *use of this type is discouraged*; it defines one-shot reply to query from binary under test + +.. warning:: + - ``REPLY`` type is useful only if you know exact order of queries sent *by the binary under test* + - steps of this type are used only when no matching ``RANGE`` datablock exists + - priority of ``REPLY`` type is going to change in future + + +.. _entry: + +DNS messages (normal ``ENTRY``) +------------------------------- +One ``ENTRY`` describes one DNS message plus additional metadata, depending on intended use of the entry. There are three possible uses of entry which require little bit different entry format. An entry might define: + +#. *query message* to be sent in ``STEP QUERY`` +#. *expected message* to be compared with a message received from binary in ``STEP CHECK_ANSWER`` +#. *answer template message* to be used for simulating answers from network in ``RANGE`` block + +Particular use of data in an ``ENTRY`` depends on context and is different +for ``STEP`` types and ``RANGE`` blocks, see details below. + +In any case, entry starts with ``ENTRY_BEGIN`` and ends with ``ENTRY_END`` keywords and share ``REPLY`` and ``SECTION`` definitions. +Some fields in DNS messages have default values which can be overriden by explicit specification. + +Format of query messages (for ``STEP QUERY``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``STEP QUERY`` requires a DNS message which will be sent by Deckard to the binary under test. Structure of the entry is: + +.. code-block:: + + STEP <n> QUERY + ENTRY_BEGIN + REPLY <OPCODE flags> ; REPLY is a bad keyword name, OPCODE and flags will be sent out! + SECTION QUESTION ; it is possible to replace QUESTION section or omit it + <name> <class> <type> ; to simulate weird queries + ENTRY_END + +The message will be assigned a random message ID, converted into DNS wire format, and sent to the binary under test. + +.. warning:: The keyword ``REPLY`` in fact defines value of flags in the outgoing message. The confusing name is here for compatibility with the original ``testbound``. + + +Format of expected messages (for ``STEP CHECK_ANSWER``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``STEP CHECK_ANSWER`` requires a DNS message which will be compared with a reply received from the binary under test. Structure of the entry describing the expected message is: + +.. code-block:: + + ENTRY_BEGIN + MATCH <match element list> ; MATCH elements define what message fields will be compared + REPLY <OPCODE RCODE flags> ; REPLY field here defines expected OPCODE, RCODE as well as flags! + SECTION QUESTION + <name> <class> <type> ; to simulate weird queries + SECTION <type2> + <RR sets> + ENTRY_END + +Deckard will compare messages according to *<match element list>*. Any mismatch between *received* message and the *expected* message (specified by the entry) will result in test failure. (See chapter `entry matching`_.) + + +Format of answer templates (for ``RANGE``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Entries in ``RANGE`` blocks are used to answer queries *from binaries under test*. E.g. if a DNS resolver under test sends query ``. IN NS`` to a simulated server, Deckard will use matching entry associated with the simulated server for reply. Entry used for answer is selected using the same `entry matching`_ logic as with ``STEP CHECK_ANSWER``. The difference is that entry is automatically modified before sending out the answer. These modifications are specified by ``ADJUST`` and ``REPLY`` keywords. It's also possible to not send any reply using the ``ADJUST do_not_answer`` option. See chapters `entry adjusting`_ and `entry flags`_. + +.. code-block:: + + ENTRY_BEGIN + MATCH <match element list> ; all MATCH elements must match before using this answer template + ADJUST <adjust element list> ; ADJUST fields will be modified before answering + REPLY <OPCODE RCODE flags> ; OPCODE, RCODE, and flags to be set in the outgoing answer + SECTION <type1> + <RR sets> + SECTION <type2> + <RR sets> + ENTRY_END + + +.. _`entry matching`: + +Entry matching +^^^^^^^^^^^^^^ +Entries present in Deckard scenario define values *expected* in DNS messages. The *expected* values are compared with values in messages *received* from the network. Entry matches only if all specified elements match. + +.. code-block:: + + MATCH <match element list> + +*<match element list>* is a space-separated list of elements in *expected* and *received* messages to be compared. Supported elements are: + +============ ========================================================================================= +element DNS message fields and additional rules +============ ========================================================================================= +opcode ``OPCODE`` as `defined in IANA registry <https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-5>`_ + + - *expected* message ``OPCODE`` is defined by ``REPLY`` keyword + +qtype RR type in question section [qmatch]_ +qname name in question section (case insensitive) [qmatch]_ +qcase name in question section (case sensitive) [qmatch]_ +subdomain name in question section of the *received* message is a subdomain of the name in *expected* question section + (case insensitive, exact match accepted) [qmatch]_ +flags all `defined flags <https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-12>`_ + in message header: ``QR AA TC RD RA AD CD`` + + - it does not match on ``DO`` flag which resides in EDNS header flags + - *expected* message flags are defined by ``REPLY`` keyword + +rcode extended response code (``RCODE`` value + `combined from message header and EDNS header <https://tools.ietf.org/html/rfc6891#section-6.1.3>`_) + + - *expected* message ``RCODE`` is defined by ``REPLY`` keyword + +question equivalent to ``qtype qname`` +answer whole ANSWER section [sectmatch]_ +authority whole AUTHORITY section [sectmatch]_ +additional whole ADDITIONAL section [sectmatch]_ +edns EDNS `version <https://tools.ietf.org/html/rfc6891#section-6.1.3>`_ and + EDNS `payload <https://tools.ietf.org/html/rfc6891#section-6.1.2>`_ size +nsid `NSID <https://tools.ietf.org/html/rfc5001>`_ presence and value +all equivalent to ``opcode qtype qname flags rcode answer authority additional`` +============ ========================================================================================= + +.. [qmatch] *Expected* values are defined by QUESTION section in the entry. If the *expected* QUESTION section is empty, the conditions is ignored. Only values from the first (qname, qclass, qtype) tuple are checked. Question matching is case insensitive (except for ``qcase``). + +.. [sectmatch] Number of records must match. Owner names are case-insensitive and TTL is ignored. RR data are compared according to type-specific rules. Each RR present in the *expected* message must be present in the *received* message and vice versa. + + +.. _`entry adjusting`: + +Entry adjusting +^^^^^^^^^^^^^^^ +.. code-block:: + + ADJUST <adjust element list> + +An entry used as a template to prepare an answer to an incoming query might be preprocessed. +Adjust element list defines what fields will be modified: + +============= =========================================================================================== +element modification to the DNS message +============= =========================================================================================== +copy_id query id + query domain name will be copied from incoming message [copy_id_bug]_ +copy_query whole question section will be copied from incoming message +raw_id query id will be copied into the first two bytes of RAW answer +do_not_answer no response will be sent at all +============= =========================================================================================== + +.. [copy_id_bug] https://gitlab.labs.nic.cz/knot/deckard/issues/9 + + +.. _`entry flags`: + +Entry flags +^^^^^^^^^^^ +.. code-block:: + + REPLY <RCODE flags> + +*<RCODE flags>* is space-separated RCODE and list of flags in the entry. Usage of these flags depend on entry context. + +Supported values: + + - NOERROR, FORMERR, SERVFAIL, NXDOMAIN, NOTIMP, REFUSED, YXDOMAIN, YXRRSET, NXRRSET, NOTAUTH, NOTZONE, BADVERS - standard rcodes + - QR, AA, TC, RD, RA, AD, CD - i.e. standard dns flags + - DO - enable 'DNSSEC desired' flag + +.. warning:: The keyword ``REPLY`` has different meaning depending on the ``ENTRY`` context. + + +Entry RR sections +^^^^^^^^^^^^^^^^^ +An entry might specify content of DNS message sections QUESTION, ANSWER, AUTHORITY, and ADDITIONAL. Syntax is of resource records is the same as in zone file. Format: + +.. code-block:: + + SECTION QUESTION + <owner name> [class] <RR type> ; QUESTION is special + SECTION <ANSWER/AUTHORITY/ADDITIONAL> + <owner name> [TTL] [class] <RR type> <RR data> ; same as in zone file + ... + <owner name> [TTL] [class] <RR type> <RR data> + +Example: + +.. code-block:: + + SECTION QUESTION + www.example.com. IN A + SECTION ANSWER + www.example.com. IN A 10.20.30.40 + SECTION AUTHORITY + example.com. IN NS ns.example.com. + SECTION ADDITIONAL + ns.example.com. IN A 1.2.3.4 + + +Default values for DNS messages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +========== =========================================================================================== +feature default value +========== =========================================================================================== +EDNS version 0 with buffer size 4096 B +REPLY QUERY, NOERROR +========== =========================================================================================== + + +Entry with RAW data +^^^^^^^^^^^^^^^^^^^ +An entry might have special section named ``RAW``. This section is used only for sending raw, +potentially invalid DNS messages. The section contains a single-line data interpreted as hexadecimal string. +Data decoded from this string will be sent to binary under test without any changes. + +Deckard does not expect any answer to RAW queries, so ``STEP CHECK_ANSWER`` is not needed. +Main intent of this behavior is to check if binary under test is +able to process valid queries after getting series badly formed packets. + +It is also possible to use ``RAW`` in conjuction with ``SECTION`` for the +purpose of responding with raw data to a query that matches the ``SECTION``. +It's possible to modify raw data to use query's ID by using ``ADJUST raw_id``. + +One ``ENTRY`` can contain only one ``RAW`` section. + +Example + +.. code-block:: + + ENTRY_BEGIN + RAW + b5c9ca3d50104320f4120000000000000000 + ENTRY_END + + + +Mock answers (``RANGE``) +------------------------ +When Deckard receives a query *from binary under test*, it searches for mock answers. +A set of mock answers for particular set of IP addresses and ID range is described using ``RANGE`` +block starting with ``RANGE_BEGIN`` keyword. The ``RANGE`` contains mock DNS messages represented +as ENTRY_ blocks which specify `entry matching`_ conditions along with `entry adjusting`_ actions and `entry flags`_ specification. + +Format: + +.. code-block:: + + ; comment before the range, e.g. K.ROOT-SERVERS.NET. + RANGE_BEGIN 0 100 ; this RANGE is valid for STEP IDs <0, 100> + ADDRESS 193.0.14.129 ; IP address simulated by this range + ;ADDRESS 192.0.2.222 ; multiple IP addresses are allowed + + ENTRY_BEGIN ; first ENTRY in this range + MATCH opcode qtype qname ; use this entry only if all these match the query + ADJUST copy_id ; adjust message ID before senting the answer + REPLY QR NOERROR ; answer with RCODE NOERROR and QR flag set + SECTION QUESTION + . IN NS ; MATCH qname qtype are compared with this value + SECTION ANSWER ; all this will be copied verbatim to the answer + . IN NS K.ROOT-SERVERS.NET. + SECTION ADDITIONAL + K.ROOT-SERVERS.NET. IN A 193.0.14.129 + ENTRY_END + + ENTRY_BEGIN ; second ENTRY in this range + ... + ENTRY_END + + RANGE_END + +When Deckard receives a query *from binary under test*, it searches for an eligible range. When an eligible range is found, it searches inside the range to find a mock answer. In detail, it works like this: + +#. Deckard searches for an eligible ``RANGE`` block. Following two conditions must be fulfilled: + + - current ``STEP ID`` is inside ID range specified by ``RANGE_BEGIN`` keyword. + - target IP address of the query is in set of IP addresses specified using ``ADDRESS`` keywords + +#. If an eligible range is found, Deckard examines all entries in the range and evaluate all ``MATCH`` conditions associated with entries. +#. An entry where all MATCH conditions are fulfilled is used as template for the mock answer. (See `entry matching`_.) +#. Mock answer is modified according to ``ADJUST`` and ``REPLY`` keywords. (See `entry adjusting`_ actions and `entry flags`_ specification.) +#. The modified answer message is sent to the binary under test. + +Valid scenario must specify answers for all queries generated by the binary under test. The test will fail if no answer is found in the eligible range or if no eligible range is defined. + +.. note:: Behavior of the binary under test, including queries it generates, depends on its configuration. For example enabling or disabling query name minimization will change minimal set of queries which a test scenario has to describe using ``RANGE`` blocks. + +.. tip:: It is recommended to construct scenarios that support multiple configurations and possibly software implementations. This leads to higher number of entries in ``RANGE`` blocks but provides robustness against changes in particular implementation. E.g. a scenario for DNS resolver testing can be developed using multiple DNS resolver implementations and combine entries for all of them inside single scenario. With this approach a small change in a resolver implementation will likely not require further changes to the scenario. + + +Configuration (``CONFIG_END``) +------------------------------ +Configuration block affects behavior of the binary under test. Deckard transforms configuration block into configuration for the binary under test. + +Format is list of "key: value" pairs, one pair per line. There is no explicit start keyword, configuration block starts immediately at scenario file begin and ends with keyword ``CONFIG_END``. + +.. code-block:: + + ; config options + query-minimization: on + stub-addr: 193.0.14.129 ; K.ROOT-SERVERS.NET. + trust-anchor: ". 3600 IN DS 10000 13 4 ABCDEF0123456789" + val-override-date: "1442323400" + CONFIG_END + +========================== ======= ===================================================================== +config option default meaning +========================== ======= ===================================================================== +do-not-query-localhost on on = queries cannot be sent to 127.0.0.1/8 or ::1/128 addresses +domain-insecure (none) domain name specifying DNS sub-tree with explicitly disabled DNSSEC validation +force-ipv6 off use a IPv6 address as ``stub-addr`` +harden-glue on additional checks on glue addresses +query-minimization on RFC 7816 query algorithm enabled; default inherited from QMIN environment variable +stub-addr (none) IP address for resolver priming queries (RFC 8109) +trust-anchor (none) owner name with its DS records (this option can be repeated multiple times) +val-override-date (none) system time reported to binary under the test; format ``YYYYMMDDHHMMSS``, so ``20120420235959`` means ``Fri Apr 20 23:59:59 2012`` +val-override-timestamp (none) system time reported to binary under the test: format POSIX timestamp +========================== ======= ===================================================================== + +Examples +-------- +See `scenatio example <scenario_example.rst>`_. The example there is a bit terse but still valid. diff --git a/tests/integration/deckard/doc/user_guide.rst b/tests/integration/deckard/doc/user_guide.rst new file mode 100644 index 0000000..a8bc23b --- /dev/null +++ b/tests/integration/deckard/doc/user_guide.rst @@ -0,0 +1,286 @@ +.. sectnum:: + +How to use Deckard +================== +.. contents:: + +Deckard runs one or more binaries in isolated network which is described by so-called *scenario*. +There are four components in play: + +- Deckard itself (test orchestrator) +- binary under test (your own) +- configuration for the binary (generated by Deckard from *template* and *YaML configuration*, i.e. ``.j2`` and ``.yaml`` files) +- environment description and test data (Deckard *scenario*, i.e. ``.rpl`` file) + +It is easy to run tests if everything is already prepared and running tests gets harder +as number of components you have to prepare yourself raises. + +Let's start with the easiest case: + +First run +--------- +Easiest way to run Deckard is using one of the prepared Shell scripts in Deckard repository (``{kresd,unbound,pdns}_run.sh`` for Knot Resolver, Unbound and PowerDNS Recursor respectively). + +Deckard uses `pytest` to generate and run the tests as well as collect the results. +Output is therefore generated by `pytest` as well (``.`` for passed test, ``F`` for failed test and ``s`` for skipped test) and will look something like this: + +.. code-block:: + + $ ./kresd_run.sh + deckard_pytest.py::test_passes_qmin_on[Scenario(path='sets/resolver/black_data.rpl', + qmin=False, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': + ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3] + SKIPPED [ 0%] + + […many lines later…] + + deckard_pytest.py::test_passes_qmin_off[Scenario(path='sets/resolver/world_mx_nic_www.rpl', + qmin=None, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], + 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3] PASSED [100%] + + ======================= 275 passed, 97 skipped in 316.61s (0:05:16) ======================= + +.. note:: There is a lot of tests skipped because we run them with query minimization both on and off and some of the scenarios work only with query minimization on (or off respectively). For details see `Scenario guide#Configuration <scenario_guide.rst#configuration-config-end>`_. + +Command line arguments +---------------------- +As mentioned above we use `py.test` to run the tests so all possible command line arguments for the ``*run.sh`` scripts can be seen by running ``py.test -h`` in the root of Deckard repository. + +Here is a list of the most useful ones: + +- ``-n number`` – runs the testing in parallel with ``number`` of processes (this requires `pytest-xdist` to be installed) +- ``-k EXPRESSION`` – only run tests which match the given substring expression (e.g. ``./kresd_run -k "world_"`` will only run the scenarios with `world_` in their file name. +- ``--collectonly`` – only print the names of selected tests, no tests will be run +- ``--log-level DEBUG`` – print all debug information for failed tests +- ``--scenarios path`` – specifies where to look for `.rpl` files (``sets/resolver`` is the default) + +YaML configuration +------------------ +All ``*_run.sh`` scripts internally call the ``run.sh`` script and pass command line arguments to it. For example: + +.. code-block:: + + # running ./kresd_run.sh -n 4 -k "iter_" will result in running + ./run.sh --config configs/kresd.yaml -n 4 -k "iter_" + +As you can see, path to YaML configuration file is passed to ``run.sh``. You can edit one of the prepared ones stored in `configs/` or write your own. + +Commented contents of ``kresd.yaml`` follows: + +.. code-block:: yaml + + programs: + - name: kresd # path to binary under test + binary: kresd + additional: # list additional parameters for binary under test (e.g. path to configuration files) + - --noninteractive + conncheck: True # wait until TCP port 53 accepts connections (enabled by default) + templates: + - template/kresd.j2 # list of Jinja2_ template files to generate configuration files + configs: + - config # list of names of configuration files to be generated from Jinja2_ templates + noclean: True # optional, do not remove working dir after a successful test + +- 'configs' files will be generated from respective files in 'templates' list +- i.e. the first file in 'configs' list is the result of processing of the first file from 'templates' list and so on +- generated files are stored in a new working directory created by Deckard for each binary + +Most often it is sufficient to use these files for basic configuration changes. Read next section for details about config file templates. + +Running multiple binaries +^^^^^^^^^^^^^^^^^^^^^^^^^ +You can specify multiple programs to run in the YaML configuration. Deckard executes all binaries using parameters from the file. This is handy for testing interoperability of multiple binaries, e.g. when one program is configured as DNS recursor and other program is using it as forwarder. + +The YAML file contains **ordered** list of binaries and their parameters. Deckard will send queries to the binary listed first. + +.. code-block:: yaml + + programs: + - name: forwarding # name of this Knot Resolver instance + binary: kresd # kresd is first so it will receive queries from Deckard + additional: [] + templates: + - template/kresd_fwd.j2 # uses variable PROGRAMS['recursor']['address'] + configs: + - config + - name: recursor # name of this Unbound instance + binary: unbound + additional: + - -d + - -c + - unbound.conf + templates: + - template/unbound.j2 + - template/hints_zone.j2 # uses variable ROOT_ADDR + configs: + - unbound.conf + - hints.zone + - ta.keys + +In this setup it is necessary to configure one binary to contact the other. IP addresses assigned by Deckard at run-time are accessible using ``PROGRAMS`` `template variables`_ and symbolic names assigned to binaries in the YAML file. For example, template ``kresd_fwd.j2`` can use IP address of binary named ``recursor`` like this: + +.. code-block:: lua + + policy.add(policy.all(policy.FORWARD("{{PROGRAMS['recursor']['address']}}"))) + +When all preparations are finished, run Deckard using following syntax: + +.. code-block:: bash + + $ ./run.sh --config path/to/config.yaml + +.. note:: You can run multiple configs in one test instance. Just be aware that ``--scenarios`` must be provided for each config. + +.. code-block:: + + # This will run scenarios from `scenarios1` folder with configuration from `config1.yaml` and `scenarios2.yaml` with `config2.yaml` respectively. + $ ./run.sh --config path/to/config1.yaml --scenarios path/to/scenarios1 --config path/to/config2.yaml --scenarios path/to/scenarios2 + + + + +Using an existing scenarios with custom configuration template +-------------------------------------------------------------- + +It some cases it is necessary to modify or create new template files. Typically this is needed when: + +- there are no templates for particular binary (e.g. if you want to test a brand new program) +- an existing template hardcodes some configuration and you want to change it + +Deckard uses the Jinja2_ templating engine (like Ansible or Salt) and supplies several variables that you can use in templates. For simplicity you can imagine that all occurrences of ``{{variable}}`` in template are replaced with value of the *variable*. See Jinja2_ documentation for further details. + +Here is an example of template for Unbound: + +.. code-block:: jinja + + server: + directory: "" # do not leave current working directory + chroot: "" + pidfile: "" + username: "" + + interface: {{SELF_ADDR}} # Deckard will assign an address + interface-automatic: no + access-control: ::0/0 allow # accept queries from Deckard + + do-daemonize: no # log to stdout & stderr + use-syslog: no + verbosity: 3 # be verbose, it is handy for debugging + val-log-level: 2 + log-queries: yes + + {% if QMIN == "false" %} # Jinja2 condition + qname-minimisation: no # a constant inside condition + {% else %} + qname-minimisation: yes + {% endif %} + harden-glue: no # hardcoded constant, use a variable instead! + + root-hints: "hints.zone" # reference to other files in working directory + trust-anchor-file: "ta.keys" # use separate template to generate these + +This configuration snippet refers to files ``hints.zone`` and ``ta.keys`` which need to be generated as well. Each file uses own template file. An template for ``hints.zone`` might look like this: + +.. code-block:: jinja + + # this is hints file which directs resolver to query + # fake root server simulated by Deckard + . 3600000 NS K.ROOT-SERVERS.NET. + # IP address version depends on scenario setting, handle IPv4 & IPv6 + {% if ':' in ROOT_ADDR %} + K.ROOT-SERVERS.NET. 3600000 AAAA {{ROOT_ADDR}} + {% else %} + K.ROOT-SERVERS.NET. 3600000 A {{ROOT_ADDR}} + {% endif %} + +Templates can use any of following variables: + +.. _`template variables`: + +List of variables for templates +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +- ``DAEMON_NAME`` - user-specified symbolic name of particular binary under test, e.g. ``recursor`` + +Addresses: + +- ``ROOT_ADDR`` - fake root server hint (an address declared in a RANGE) +- ``FORWARD_ADDR`` - IP address where resolver should forward all queries (an address declared in a RANGE) +- ``SELF_ADDR`` - address assigned to the binary under test + +- port is not expressed, must be 53 +- IP version depends on settings in particular scenario +- templates must handle IPv4 and IPv6 as well + +Path variables: + +- ``INSTALL_DIR`` - path to directory containing file ``deckard.py`` +- ``WORKING_DIR`` - working directory for binary under test, each binary gets its own directory + +DNS specifics: + +- ``DO_NOT_QUERY_LOCALHOST`` [bool]_ - allows or disallows querying local addresses +- ``HARDEN_GLUE`` [bool]_ - enables or disables additional checks on glue addresses +- ``QMIN`` [bool]_ - enables or disables query minimization respectively +- ``TRUST_ANCHORS`` - list of trust anchors in form of a DS records, see `scenario guide <doc/scenario_guide.rst>`_ +- ``NEGATIVE_TRUST_ANCHORS`` - list of domain names with explicitly disabled DNSSEC validation + +Cross references: +- ``PROGRAMS`` - dictionary of dictionaries with parameters for each binary under test + - it is handy for cases where configuration for one binary under test has to refer to another binary under test, e.g. ``PROGRAMS['recursor']['address']`` and ``PROGRAMS['forwarder']['address']``. + + +.. [bool] boolean expressed as string ``true``/``false`` + +It's okay if you don't use all of the variables, but expect some tests to fail. E.g. if you don't set the ``TRUST_ANCHORS``, +then the DNSSEC tests will not work properly. + + +Debugging scenario execution +---------------------------- +Output from a failed test looks like this: + +.. code-block:: + + $ ./kresd_run.sh + =========================================== FAILURES =========================================== + _____ test_passes_qmin_off[Scenario(path='sets/resolver/val_ta_sentinel.rpl', qmin=False)] _____ + [...] + E ValueError: val_ta_sentinel.rpl step 212 char position 15875, "rcode": expected 'SERVFAIL', + E got 'NOERROR' in the response: + E id 54873 + E opcode QUERY + E rcode NOERROR + E flags QR RD RA AD + E edns 0 + E payload 4096 + E ;QUESTION + E _is-ta-bd19.test. IN A + E ;ANSWER + E _is-ta-bd19.test. 5 IN A 192.0.2.1 + E ;AUTHORITY + E ;ADDITIONAL + + pydnstest/scenario.py:888: ValueError + +In this example, the test step ``212`` in scenario ``sets/resolver/val_ta_sentinel.rpl`` is failing with query-minimisation off. The binary under test did not produce expected answer, so either the test scenario or binary is wrong. If we were debugging this example, we would have to open file ``val_ta_sentinel.rpl`` on character postition ``15875`` and use our brains :-). + +Tips: + +- details about scenario format are in `the scenario guide <scenario_guide.rst>`_ +- network traffic from each binary is logged in PCAP format to a file in working directory +- standard output and error from each binary is logged into log file in working directory +- working directory can be explicitly specified in environment variable ``DECKARD_DIR` +- command line argument ``--log-level DEBUG`` forces extra verbose logging, including logs from all binaries and packets handled by Deckard +- environment variable ``DECKARD_NOCLEAN`` instructs Deckard not to remove working directories after successful tests +- environment variable ``DECKARD_WRAPPER`` is prepended to all commands to be executed, intended usage is to run binary under test with ``valgrind`` or ``rr record`` + + +Writting own scenarios +---------------------- +See `the scenario guide <scenario_guide.rst>`_. + + + + + +.. _`Jinja2`: http://jinja.pocoo.org/ |