diff options
Diffstat (limited to 'doc/userguide/devguide/extending/app-layer')
13 files changed, 742 insertions, 0 deletions
diff --git a/doc/userguide/devguide/extending/app-layer/app-layer-frames.rst b/doc/userguide/devguide/extending/app-layer/app-layer-frames.rst new file mode 100644 index 0000000..a810f55 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/app-layer-frames.rst @@ -0,0 +1,223 @@ +******************************* +Application Layer Frame Support +******************************* + +.. contents:: Table of Contents + +Baseline +======== + +- `Suricata rules format <https://docs.suricata.io/en/latest/rules/intro.html>`_ + +General Concepts +================ + +Frame support was introduced with Suricata 7.0. Up until 6.0.x, Suricata's architecture and state of parsers meant that the network traffic available to the detection engine was just a stream of data, without detail about higher level parsers. + +.. note:: For Suricata, *Frame* is a generic term that can represent any unit of network data we are interested in, which could be comprised of one or several records of other, lower level protocol(s). Frames work as "stream annotations", allowing Suricata to tell the detection engine what type of record exists at a specific offset in the stream. + +The normal pipeline of detection in Suricata implied that: + +- Certain rules could be quite costly performance-wise. This happened because the same stream could be inspected several times for different rules, since for certain signatures the detection is done when Suricata is still inspecting a lower level stream, not the application layer protocol (e.g., *TCP* traffic, in place of *SMB* one); +- Rules could be difficult and tedious to write (and read), requiring that writers went in byte-detail to express matching on specific payload patterns. + +What the Frame support offers is the ability to "point" to a specific portion of stream and identify what type of traffic Suricata is looking at. Then, as the engine reassembles the stream, one can have "read access" to that portion of the stream, aggregating concepts like what type of application layer protocol that is, and differentiating between ``header``, ``data`` or even protocol versions (*SMB1*, *SMB2*...). + +The goal of the stream *Frame* is to expose application layer protocol `PDUs <https://en.wikipedia.org/wiki/Protocol_data_unit>`_ and other such arbitrary elements to the detection engine directly, instead of relying on Transactions. The main purpose is to bring *TCP data* processing times down by specialising/ filtering down traffic detection. + +Adding Frame Support to a Parser +================================ + +The application layer parser exposes frames it supports to the detect engine, by tagging them as they're parsed. The rest works automatically. + +In order to allow the engine to identify frames for records of a given application layer parser, thought must be given as to which frames make sense for the specific protocol you are handling. Some parsers may have clear ``header`` and ``data`` fields that form its *protocol data unit* (pdu). For others, the distinction might be between ``request`` and ``response``, only. Whereas for others it may make sense to have specific types of data. This is better understood by seeing the different types of frame keywords, which vary on a per-protocol basis. + +It is also important to keep follow naming conventions when defining Frame Types. While a protocol may have strong naming standards for certain structures, do compare those with what Suricata already has registered: + +- ``hdr``: used for the record header portion +- ``data``: is used for the record data portion +- ``pdu``: unless documented otherwise, means the whole record, comprising ``hdr`` and ``data`` +- ``request``: a message from a client to a server +- ``response``: a message from a server to a client + +Basic steps +~~~~~~~~~~~ + +Once the frame types that make sense for a given protocol are defined, the basic steps for adding them are: + +- create an enum with the frame types; +- identify the parsing function(s) where application layer records are parsed; +- identify the correct moment to register the frames; +- use the Frame API calls directly or build upon them and use your functions to register the frames; +- register the relevant frame callbacks when registering the parser. + +Once these are done, you can enable frame eve-output to confirm that your frames are being properly registered. It is important to notice that some hard coded limits could influence what you see on the logs (max size of log output; type of logging for the payload, cf. https://redmine.openinfosecfoundation.org/issues/4988). + +If all the steps are successfully followed, you should be able to write a rule using the *frame* keyword and the frame types you have registered with the application layer parser. + +Using the *SMB* parser as example, before frame support, a rule would look like:: + + alert tcp ... flow:to_server; content:"|ff|SMB"; content:"some smb 1 issue"; + +With frame support, one is able to do:: + + alert smb ... flow:to_server; frame:smb1.data; content:"some smb 1 issue"; + +Implementation Examples & API Callbacks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Though the steps are the same, there are a few differences when implementing frame support in Rust or in C. The following sections elaborate on that, as well as on the process itself. (Note that the code snippets have omitted portions of code that weren't so relevant to this document). + +Rust +---- + +This section shows how Frame support is added in Rust, using examples from the `SIP parser <https://github.com/OISF/suricata/blob/master/rust/src/sip/sip.rs>`_, and the `telnet parser <https://github.com/OISF/suricata/blob/master/rust/src/telnet/telnet.rs>`_. + +**Define the frame types**. The frame types are defined as an enum. In Rust, make sure to derive from the ``AppLayerFrameType``: + +.. literalinclude:: ../../../../../rust/src/sip/sip.rs + :caption: rust/src/sip/sip.rs + :language: rust + :start-after: // app-layer-frame-documentation tag start: FrameType enum + :end-before: // app-layer-frame-documentation tag end: FrameType enum + +**Frame registering**. Some understanding of the parser will be needed in order to find where the frames should be registered. It makes sense that it will happen when the input stream is being parsed into records. See when some pdu and request frames are created for SIP: + +.. literalinclude:: ../../../../../rust/src/sip/sip.rs + :caption: rust/src/sip/sip.rs + :language: rust + :start-after: // app-layer-frame-documentation tag start: parse_request + :end-before: // app-layer-frame-documentation tag end: parse_request + :dedent: 4 + +.. note:: when to create PDU frames + + The standard approach we follow for frame registration is that a frame ``pdu`` will always be created, regardless of parser status (in practice, before the parser is called). The other frames are then created when and if only the parser succeeds. + +**Use the Frame API or build upon them as needed**. These are the frame registration functions highlighted above: + +.. literalinclude:: ../../../../../rust/src/sip/sip.rs + :caption: rust/src/sip/sip.rs + :language: rust + :start-after: // app-layer-frame-documentation tag start: function to add frames + :end-before: // app-layer-frame-documentation tag end: function to add frames + +**Register relevant frame callbacks.** As these are inferred from the ``#[derive(AppLayerFrameType)]`` statement, all that is needed is: + +.. literalinclude:: ../../../../../rust/src/sip/sip.rs + :caption: rust/src/sip/sip.rs + :language: rust + :start-at: get_frame_id_by_name + :end-at: ffi_name_from_id), + :dedent: 8 + +.. note:: on frame_len + + For protocols which search for an end of frame char, like telnet, indicate unknown length by passing ``-1``. Once the length is known, it must be updated. For those where length is a field in the record (e.g. *SIP*), the frame is set to match said length, even if that is bigger than the current input + +The telnet parser has examples of using the Frame API directly for registering telnet frames, and also illustrates how that is done when length is not yet known: + +.. literalinclude:: ../../../../../rust/src/telnet/telnet.rs + :caption: rust/src/telnet/telnet.rs + :language: rust + :start-after: // app-layer-frame-documentation tag start: parse_request + :end-before: // app-layer-frame-documentation tag end: parse_request + :lines: 1-3, 22-49 + :dedent: 4 + +We then update length later on (note especially lines 3 and 10): + +.. literalinclude:: ../../../../../rust/src/telnet/telnet.rs + :caption: rust/src/telnet/telnet.rs + :language: rust + :start-after: // app-layer-frame-documentation tag start: update frame_len + :end-before: // app-layer-frame-documentation tag end: update frame_len + :linenos: + :dedent: 12 + +The Frame API calls parameters represent: + +- ``flow``: dedicated data type, carries specific flow-related data +- ``stream_slice``: dedicated data type, carries stream data, shown further bellow +- ``frame_start``: a pointer to the start of the frame buffer in the stream (``cur_i`` in the SMB code snippet) +- ``frame_len``: what we expect the frame length to be (the engine may need to wait until it has enough data. See what is done in the telnet snippet request frames registering) +- ``frame_type``: type of frame it's being registering (defined in an enum, as shown further above) + +``StreamSlice`` contains the input data to the parser, alongside other Stream-related data important in parsing context. Definition is found in *applayer.rs*: + +.. literalinclude:: ../../../../../rust/src/applayer.rs + :caption: rust/src/applayer.rs + :language: rust + :start-at: pub struct StreamSlice + :end-before: impl StreamSlice + + +C code +------ + +Implementing Frame support in C involves a bit more manual work, as one cannot make use of the Rust derives. Code snippets from the *HTTP* parser: + +Defining the frame types with the enum means: + +.. literalinclude:: ../../../../../src/app-layer-htp.c + :caption: src/app-layer-htp.c + :start-after: /* app-layer-frame-documentation tag start: HttpFrameTypes + :end-before: /* app-layer-frame-documentation tag end: HttpFrameTypes + :lines: 1-16 + +The HTTP parser uses the Frame registration functions from the C API (``app-layer-frames.c``) directly for registering request Frames. Here we also don't know the length yet. The ``0`` indicates flow direction: ``toserver``, and ``1`` would be used for ``toclient``: + +.. literalinclude:: ../../../../../src/app-layer-htp.c + :caption: src/app-layer-htp.c + :start-after: /* app-layer-frame-documentation tag start: frame registration http request + :end-before: /* app-layer-frame-documentation tag end: frame registration http request + :dedent: 4 + +Updating ``frame->len`` later: + +.. literalinclude:: ../../../../../src/app-layer-htp.c + :caption: src/app-layer-htp.c + :start-after: /* app-layer-frame-documentation tag start: updating frame->len + :end-before: /* app-layer-frame-documentation tag end: updating frame->len + :dedent: 4 + +Register relevant callbacks (note that the actual functions will also have to be written, for C): + +.. literalinclude:: ../../../../../src/app-layer-htp.c + :caption: src/app-layer-htp.c + :language: c + :start-after: /* app-layer-frame-documentation tag start: registering relevant callbacks + :end-before: /* app-layer-frame-documentation tag end: registering relevant callbacks + :dedent: 8 + +.. note:: The ``GetFrameIdByName`` functions can be "probed", so they should not generate any output or that could be misleading (for instance, Suricata generating a log message stating that a valid frame type is unknown). + +Visual context +============== + +``input`` and ``input_len`` are used to calculate the proper offset, for storing the frame. The stream buffer slides forward, so frame offsets/frames have to be updated. The `relative offset` (``rel_offset``) reflects that: + +.. code-block:: c + + Start: + [ stream ] + [ frame ...........] + rel_offset: 2 + len: 19 + + Slide: + [ stream ] + [ frame .... .] + rel_offset: -10 + len: 19 + + Slide: + [ stream ] + [ frame ........... ] + rel_offset: -16 + len: 19 + +The way the engine handles stream frames can be illustrated as follows: + +.. image:: img/StreamFrames.png + :scale: 80 diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/DnsUnidirectionalTransactions.msc b/doc/userguide/devguide/extending/app-layer/diagrams/DnsUnidirectionalTransactions.msc new file mode 100644 index 0000000..43dd653 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/DnsUnidirectionalTransactions.msc @@ -0,0 +1,19 @@ +# MSC Sequence Diagram Example: DNS Query Transaction + +msc { + # Chart Options + arcgradient = "10"; + + # Entities + a [ label = "Client" ], b [ label = "Server" ]; + + # Message Flow + a =>> b [ label = "DNS Request" ]; + --- [ label = "Transaction 1 Completed" ]; + |||; + b =>> a [ label = "DNS Response" ]; + --- [ label = "Transaction 2 Completed" ]; + + |||; + ||| [label="[ generated with Mscgen ]", textcolor="gray"]; +} diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/DnsUnidirectionalTransactions.png b/doc/userguide/devguide/extending/app-layer/diagrams/DnsUnidirectionalTransactions.png Binary files differnew file mode 100644 index 0000000..611ae31 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/DnsUnidirectionalTransactions.png diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/HTTP2BidirectionalTransaction.msc b/doc/userguide/devguide/extending/app-layer/diagrams/HTTP2BidirectionalTransaction.msc new file mode 100644 index 0000000..3c3484f --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/HTTP2BidirectionalTransaction.msc @@ -0,0 +1,20 @@ +# MSC Sequence Diagram for an HTTP2 Transaction, which is bidirectional in Suricata + +msc { + + # Chart options + arcgradient = "10"; + + # Entities + a [ label = "Client" ], b [ label = "Server" ]; + + # Message flow + a =>> b [ label = "Request" ]; + b =>> a [ label = "Response" ]; + |||; + --- [ label = "Transaction Completed" ]; + |||; + ||| [label="[ generated with Mscgen ]", textcolor="gray"]; +} + +# Reference: https://tools.ietf.org/html/rfc7540#section-8.1 diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/HTTP2BidirectionalTransaction.png b/doc/userguide/devguide/extending/app-layer/diagrams/HTTP2BidirectionalTransaction.png Binary files differnew file mode 100644 index 0000000..02de5dc --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/HTTP2BidirectionalTransaction.png diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/TemplateTransaction.msc b/doc/userguide/devguide/extending/app-layer/diagrams/TemplateTransaction.msc new file mode 100644 index 0000000..3a5a308 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/TemplateTransaction.msc @@ -0,0 +1,18 @@ +# MSC Sequence Diagram Example: Template transaction + +msc { + # Chart Options + arcgradient = "10"; + + # Entities + a [ label = "Client" ], b [ label = "Server" ]; + + # Message Flow + a =>> b [ label = "Request ('12:HelloWorld!')" ]; + b =>> a [ label = "Response ('3:Bye')" ]; + |||; + --- [ label = "Transaction Completed" ]; + + |||; + ||| [label="[ generated with Mscgen ]", textcolor="gray"]; +} diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/TemplateTransaction.png b/doc/userguide/devguide/extending/app-layer/diagrams/TemplateTransaction.png Binary files differnew file mode 100644 index 0000000..89daed4 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/TemplateTransaction.png diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/TlsHandshake.msc b/doc/userguide/devguide/extending/app-layer/diagrams/TlsHandshake.msc new file mode 100644 index 0000000..34a025d --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/TlsHandshake.msc @@ -0,0 +1,34 @@ +# MSC Sequence Diagram Example: TLS Handshake Transaction + +msc { + # Chart Options + arcgradient = "10"; + + # Entities + a [ label = "Client" ], b [ label = "Server" ]; + + # Message Flow + # TLS_STATE_IN_PROGRESS = 0, + a abox b [ label = "TLS_STATE_IN_PROGRESS" ]; + a =>> b [ label = "ClientHello" ]; + b =>> a [ label = "ServerHello" ]; + b =>> a [ label = "ServerCertificate" ]; + b =>> a [ label = "ServerHello Done" ]; + + a =>> b [ label = "ClientCertificate" ]; + # TLS_STATE_CERT_READY = 1, + a abox b [ label = "TLS_STATE_CERT_READY" ]; + a =>> b [ label = "ClientKeyExchange" ]; + + a =>> b [ label = "Finished" ]; + b =>> a [ label = "Finished" ]; + # TLS_HANDSHAKE_DONE = 2, + a abox b [ label = "TLS_HANDSHAKE_DONE" ]; + ...; + # TLS_STATE_FINISHED = 3 + a abox b [ label = "TLS_STATE_FINISHED" ]; + --- [ label = "Transaction Completed" ]; + + |||; + ||| [label="[ generated with Mscgen ]", textcolor="gray"]; +} diff --git a/doc/userguide/devguide/extending/app-layer/diagrams/TlsHandshake.png b/doc/userguide/devguide/extending/app-layer/diagrams/TlsHandshake.png Binary files differnew file mode 100644 index 0000000..4be7b04 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/diagrams/TlsHandshake.png diff --git a/doc/userguide/devguide/extending/app-layer/img/StreamFrames.png b/doc/userguide/devguide/extending/app-layer/img/StreamFrames.png Binary files differnew file mode 100644 index 0000000..f145c98 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/img/StreamFrames.png diff --git a/doc/userguide/devguide/extending/app-layer/index.rst b/doc/userguide/devguide/extending/app-layer/index.rst new file mode 100644 index 0000000..c392e58 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/index.rst @@ -0,0 +1,9 @@ +App-Layer +========= + +.. toctree:: + :maxdepth: 2 + + app-layer-frames.rst + parser.rst + transactions.rst diff --git a/doc/userguide/devguide/extending/app-layer/parser.rst b/doc/userguide/devguide/extending/app-layer/parser.rst new file mode 100644 index 0000000..262ed89 --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/parser.rst @@ -0,0 +1,98 @@ +******* +Parsers +******* + +Callbacks +========= + +The API calls callbacks that are registered at the start of the program. + +The function prototype is: + +.. code-block:: c + + typedef AppLayerResult (*AppLayerParserFPtr)(Flow *f, void *protocol_state, + AppLayerParserState *pstate, + const uint8_t *buf, uint32_t buf_len, + void *local_storage, const uint8_t flags); + +Examples +-------- + +A C example: + +.. code-block:: c + + static AppLayerResult HTPHandleRequestData(Flow *f, void *htp_state, + AppLayerParserState *pstate, + const uint8_t *input, uint32_t input_len, + void *local_data, const uint8_t flags); + +In Rust, the callbacks are similar. + +.. code-block:: rust + + #[no_mangle] + pub extern "C" fn rs_dns_parse_response_tcp(_flow: *const core::Flow, + state: *mut std::os::raw::c_void, + _pstate: *mut std::os::raw::c_void, + input: *const u8, + input_len: u32, + _data: *const std::os::raw::c_void, + _flags: u8) + -> AppLayerResult + + +Return Types +============ + +Parsers return the type `AppLayerResult`. + +There are 3 possible results: + - `APP_LAYER_OK` - parser consumed the data successfully + - `APP_LAYER_ERROR` - parser encountered a unrecoverable error + - `APP_LAYER_INCOMPLETE(c,n)` - parser consumed `c` bytes, and needs `n` more before being called again + +Rust parsers follow the same logic, but can return + - `AppLayerResult::ok()` + - `AppLayerResult::err()` + - `AppLayerResult::incomplete(c,n)` + +For `i32` and `bool`, Rust parsers can also use `.into()`. + +APP_LAYER_OK / AppLayerResult::ok() +----------------------------------- + +When a parser returns "OK", it signals to the API that all data has been consumed. The parser will be called again when more data is available. + +APP_LAYER_ERROR / AppLayerResult::err() +--------------------------------------- + +Returning "ERROR" from the parser indicates to the API that the parser encountered an unrecoverable error and the processing of the protocol should stop for the rest of this flow. + +.. note:: This should not be used for recoverable errors. For those events should be set. + +APP_LAYER_INCOMPLETE / AppLayerResult::incomplete() +--------------------------------------------------- + +Using "INCOMPLETE" a parser can indicate how much more data is needed. Many protocols use records that have the size as one of the first parameters. When the parser receives a partial record, it can read this value and then tell the API to only call the parser again when enough data is available. + +`consumed` is used how much of the current data has been processed +`needed` is the number of bytes that the parser needs on top of what was consumed. + +Example:: + + [ 32 record 1 ][ 32 record 2 ][ 32 r.. ] + 0 31 32 63 64 72 + ^ ^ + consumed: 64 ---------------/ | + needed: 32 -------------------/ + +.. note:: "INCOMPLETE" is only supported for TCP + +The parser will be called again when the `needed` data is available OR when the stream ends. In the latter case the data will be incomplete. It's up to the parser to decide what to do with it in this case. + +Supporting incomplete data +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In some cases it may be preferable to actually support dealing with incomplete records. For example protocols like SMB and NFS can use very large records during file transfers. Completely queuing these before processing could be a waste of resources. In such cases the "INCOMPLETE" logic could be used for just the record header, while the record data is streamed into the parser. diff --git a/doc/userguide/devguide/extending/app-layer/transactions.rst b/doc/userguide/devguide/extending/app-layer/transactions.rst new file mode 100644 index 0000000..357bdcd --- /dev/null +++ b/doc/userguide/devguide/extending/app-layer/transactions.rst @@ -0,0 +1,321 @@ +************ +Transactions +************ + +.. contents:: Table of Contents + +General Concepts +================ + +For Suricata, transactions are an abstraction that help with detecting and logging. An example of a complete transaction is +a pair of messages in the form of a request (from client to server) and a response (from server to client) in HTTP. + +In order to know when to log an event for a given protocol, the engine tracks the progress of each transaction - that +is, when is it complete, or when it reaches a key intermediate state. They aid during the detection phase, +when dealing with protocols that can have large PDUs (protocol data units), like TCP, in controlling state for partial rule matching -- in case of rules that mention more than one field. + +Transactions are implemented and stored in the per-flow state. The engine interacts with them using a set of callbacks the parser registers. + +How the engine uses transactions +================================ + +Logging +~~~~~~~ + +Suricata controls when logging should happen based on transaction completeness. For simpler protocols, such as ``dns`` +or ``ntp``, that will most +likely happen once per transaction, by the time of its completion. In other cases, like with HTTP, this may happen at intermediary states. + +In ``OutputTxLog``, the engine will compare current state with the value defined for the logging to happen, per flow +direction (``logger->tc_log_progress``, ``logger->ts_log_progress``). If state is less than that value, the engine skips to +the next logger. Code snippet from: suricata/src/output-tx.c: + +.. code-block:: c + + static TmEcode OutputTxLog(ThreadVars *tv, Packet *p, void *thread_data) + { + . + . + . + if ((ts_eof && tc_eof) || last_pseudo) { + SCLogDebug("EOF, so log now"); + } else { + if (logger->LogCondition) { + int r = logger->LogCondition(tv, p, alstate, tx, tx_id); + if (r == FALSE) { + SCLogDebug("conditions not met, not logging"); + goto next_logger; + } + } else { + if (tx_progress_tc < logger->tc_log_progress) { + SCLogDebug("progress not far enough, not logging"); + goto next_logger; + } + + if (tx_progress_ts < logger->ts_log_progress) { + SCLogDebug("progress not far enough, not logging"); + goto next_logger; + } + } + } + . + . + . + } + +Rule Matching +~~~~~~~~~~~~~ + +Transaction progress is also used for certain keywords to know what is the minimum state before we can expect a match: until that, Suricata won't even try to look for the patterns. + +As seen in ``DetectAppLayerMpmRegister2`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet, +``HTTP2StateDataClient``, ``HTTP2StateDataServer`` and ``0`` are the values passed to the functions - in the last +example, for ``FTPDATA``, +the existence of a transaction implies that a file is being transferred. Hence the ``0`` value. + + +.. code-block:: c + + void DetectFiledataRegister(void) + { + . + . + DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOSERVER, 2, + PrefilterMpmFiledataRegister, NULL, + ALPROTO_HTTP2, HTTP2StateDataClient); + DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOCLIENT, 2, + PrefilterMpmFiledataRegister, NULL, + ALPROTO_HTTP2, HTTP2StateDataServer); + . + . + DetectAppLayerInspectEngineRegister2("file_data", + ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateDataServer, + DetectEngineInspectFiledata, NULL); + DetectAppLayerInspectEngineRegister2( + "file_data", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0, DetectEngineInspectFiledata, NULL); + . + . + } + +Progress Tracking +================= + +As a rule of thumb, transactions will follow a request-response model: if a transaction has had a request and a response, it is complete. + +But if a protocol has situations where a request or response won’t expect or generate a message from its counterpart, +it is also possible to have uni-directional transactions. In such cases, transaction is set to complete at the moment of +creation. + +For example, DNS responses may be considered as completed transactions, because they also contain the request data, so +all information needed for logging and detection can be found in the response. + +In addition, for file transfer protocols, or similar ones where there may be several messages before the file exchange +is completed (NFS, SMB), it is possible to create a level of abstraction to handle such complexity. This could be achieved by adding phases to the model implemented by the protocol (e.g., protocol negotiation phase (SMB), request parsed (HTTP), and so on). + +This is controlled by implementing progress states. In Suricata, those will be enums that are incremented as the parsing +progresses. A state will start at 0. The higher its value, the closer the transaction would be to completion. Due to how +the engine tracks detection across states, there is an upper limit of 48 to the state progress (it must be < 48). + +The engine interacts with transactions' state using a set of callbacks the parser registers. State is defined per flow direction (``STREAM_TOSERVER`` / ``STREAM_TOCLIENT``). + +In Summary - Transactions and State +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Initial State value: ``0``. +- Simpler scenarios: State is simply a bool. ``1`` represents transaction completion, per direction. +- Complex Transaction State in Suricata: ``enum`` (Rust: ``i32``). Completion is indicated by the highest enum value (some examples are: SSH, HTTP, HTTP2, DNS, SMB). + +Examples +======== + +This section shares some examples from Suricata codebase, to help visualize how Transactions work and are handled by the engine. + +Enums +~~~~~ + +Code snippet from: rust/src/ssh/ssh.rs: + +.. code-block:: rust + + pub enum SSHConnectionState { + SshStateInProgress = 0, + SshStateBannerWaitEol = 1, + SshStateBannerDone = 2, + SshStateFinished = 3, + } + +From src/app-layer-ftp.h: + +.. code-block:: c + + enum { + FTP_STATE_IN_PROGRESS, + FTP_STATE_PORT_DONE, + FTP_STATE_FINISHED, + }; + +From src/app-layer-ssl.h: + +.. code-block:: c + + enum { + TLS_STATE_IN_PROGRESS = 0, + TLS_STATE_CERT_READY = 1, + TLS_HANDSHAKE_DONE = 2, + TLS_STATE_FINISHED = 3 + }; + +API Callbacks +~~~~~~~~~~~~~ + +In Rust, this is done via the RustParser struct. As seen in rust/src/applayer.rs: + +.. code-block:: rust + + /// Rust parser declaration + pub struct RustParser { + . + . + . + /// Progress values at which the tx is considered complete in a direction + pub tx_comp_st_ts: c_int, + pub tx_comp_st_tc: c_int, + . + . + . + } + +In C, the callback API is: + +.. code-block:: c + + void AppLayerParserRegisterStateProgressCompletionStatus( + AppProto alproto, const int ts, const int tc) + +Simple scenario described, in Rust: + +rust/src/dhcp/dhcp.rs: + +.. code-block:: rust + + tx_comp_st_ts: 1, + tx_comp_st_tc: 1, + +For SSH, this looks like this: + +rust/src/ssh/ssh.rs: + +.. code-block:: rust + + tx_comp_st_ts: SSHConnectionState::SshStateFinished as i32, + tx_comp_st_tc: SSHConnectionState::SshStateFinished as i32, + +In C, callback usage would be as follows: + +src/app-layer-dcerpc.c: + +.. code-block:: c + + AppLayerParserRegisterStateProgressCompletionStatus(ALPROTO_DCERPC, 1, 1); + +src/app-layer-ftp.c: + +.. code-block:: c + + AppLayerParserRegisterStateProgressCompletionStatus( + ALPROTO_FTP, FTP_STATE_FINISHED, FTP_STATE_FINISHED); + +Sequence Diagrams +~~~~~~~~~~~~~~~~~ + +A DNS transaction in Suricata can be considered unidirectional: + +.. image:: diagrams/DnsUnidirectionalTransactions.png + :width: 600 + :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "DNS Request". After that, there is a dotted line labeled "Transaction Completed". + +An HTTP2 transaction is an example of a bidirectional transaction, in Suricata (note that, while HTTP2 may have multiple streams, those are mapped to transactions in Suricata. They run in parallel, scenario not shown in this Sequence Diagram - which shows one transaction, only): + +.. image:: diagrams/HTTP2BidirectionalTransaction.png + :width: 600 + :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "Request" and below that an arrow going from Server to Client labeled "Response". Below those arrows, a dotted line indicates that the transaction is completed. + +A TLS Handshake is a more complex example, where several messages are exchanged before the transaction is considered completed: + +.. image:: diagrams/TlsHandshake.png + :width: 600 + :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "ClientHello" and below that an arrow going from Server to Client labeled "ServerHello". Below those arrows, several more follow from Server to Client and vice-versa, before a dotted line indicates that the transaction is finally completed. + +Template Protocol +~~~~~~~~~~~~~~~~~ + +Suricata has a template protocol for educational purposes, which has simple bidirectional transactions. + +A completed transaction for the template looks like this: + +.. image:: diagrams/TemplateTransaction.png + :width: 600 + :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "Request". An arrow below that first one goes from Server to Client. + +Following are the functions that check whether a transaction is considered completed, for the Template Protocol. Those are called by the Suricata API. Similar functions exist for each protocol, and may present implementation differences, based on what is considered a transaction for that given protocol. + +In C: + +.. code-block:: c + + static int TemplateGetStateProgress(void *txv, uint8_t direction) + { + TemplateTransaction *tx = txv; + + SCLogNotice("Transaction progress requested for tx ID %"PRIu64 + ", direction=0x%02x", tx->tx_id, direction); + + if (direction & STREAM_TOCLIENT && tx->response_done) { + return 1; + } + else if (direction & STREAM_TOSERVER) { + /* For the template, just the existence of the transaction means the + * request is done. */ + return 1; + } + + return 0; + } + +And in Rust: + +.. code-block:: rust + + pub extern "C" fn rs_template_tx_get_alstate_progress( + tx: *mut std::os::raw::c_void, + _direction: u8, + ) -> std::os::raw::c_int { + let tx = cast_pointer!(tx, TemplateTransaction); + + // Transaction is done if we have a response. + if tx.response.is_some() { + return 1; + } + return 0; + } + +Work In Progress changes +======================== + +Currently we are working to have files be part of the transaction instead of the per-flow state, as seen in https://redmine.openinfosecfoundation.org/issues/4444. + +Another work in progress is to limit the number of transactions per flow, to prevent Denial of Service (DoS) by quadratic complexity - a type of attack that may happen to protocols which can have multiple transactions at the same time - such as HTTP2 so-called streams (see https://redmine.openinfosecfoundation.org/issues/4530). + +Common words and abbreviations +============================== + +- al, applayer: application layer +- alproto: application layer protocol +- alstate: application layer state +- engine: refers to Suricata core detection logic +- flow: a bidirectional flow of packets with the same 5-tuple elements (protocol, source ip, destination ip, source port, destination port. Vlans can be added as well) +- PDU: Protocol Data Unit +- rs: rust +- tc: to client +- ts: to server +- tx: transaction |
