diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-09-19 04:14:33 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-09-19 04:14:33 +0000 |
commit | 9f153fbfec0fb9c9ce38e749a7c6f4a5e115d4e9 (patch) | |
tree | 2784370cda9bbf2da9114d70f05399c0b229d28c /doc/wsug_src/wsug_mate.adoc | |
parent | Adding debian version 4.2.6-1. (diff) | |
download | wireshark-9f153fbfec0fb9c9ce38e749a7c6f4a5e115d4e9.tar.xz wireshark-9f153fbfec0fb9c9ce38e749a7c6f4a5e115d4e9.zip |
Merging upstream version 4.4.0.
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
-rw-r--r-- | doc/wsug_src/wsug_mate.adoc (renamed from docbook/wsug_src/wsug_mate.adoc) | 1571 |
1 files changed, 827 insertions, 744 deletions
diff --git a/docbook/wsug_src/wsug_mate.adoc b/doc/wsug_src/wsug_mate.adoc index 6de78c70..9df35551 100644 --- a/docbook/wsug_src/wsug_mate.adoc +++ b/doc/wsug_src/wsug_mate.adoc @@ -17,7 +17,7 @@ was written to help troubleshooting gateways and other systems where a "use" involves more protocols. However, MATE can be used as well to analyze other issues regarding an interaction between packets like response times, incompleteness of transactions, presence/absence of certain attributes in a -group of PDUs and more. +group of Protocol Data Units (PDUs) and more. MATE is a Wireshark plugin that allows the user to specify how different frames are related to each other. To do so, MATE extracts data from the frames' @@ -47,24 +47,25 @@ that timeout) These are the steps to try out MATE: -* Run Wireshark and check if the plugin is installed correct (MATE should -appear in Help->About->Plugins) -* Get a configuration file e.g., tcp.mate (see {wireshark-wiki-url}Mate/Examples[Mate/Examples] +* Run Wireshark and check if the plugin is installed (MATE should +appear in Help->About Wireshark:Plugins) +* Get a configuration file e.g., <<File_tcp_mate,tcp.mate>> (see {wireshark-wiki-url}Mate/Examples[Mate/Examples] for more) and place it somewhere on your harddisk. -* Go to Preferences->Protocols->MATE and set the config filename to the file -you want to use (you don't have to restart Wireshark) +* Go to Edit->Preferences...->Protocols->MATE and set the Configuration Filename to the file +you want to use and restart Wireshark. * Load a corresponding capture file (e.g., {wireshark-wiki-url}uploads/27707187aeb30df68e70c8fb9d614981/http.cap[http.cap]) and see if MATE has added some new display filter fields, something like: `mate tcp_pdu:1->tcp_ses:1` or, at prompt: `path_to/wireshark -o "mate.config: tcp.mate" -r http.cap`. -If anything went well, your packet details might look something like this: +If everything went well, your packet details might look something like this: +.Packet Details - MATE TCP Session (tcp.mate) image::images/ws-mate-tcp-output.png[] [#ChMateManual] -=== MATE Manual +=== MATE Overview ==== Introduction @@ -81,45 +82,53 @@ will use the term "PDU" to refer to the objects created by MATE containing the relevant information extracted from the frame; I'll use "frame" to refer to the "raw" information extracted by the various dissectors that pre-analyzed the frame. -For every PDU, MATE checks if it belongs to an existing "Group of PDUs" (Gop). -If it does, it assigns the PDU to that Gop and moves any new relevant attributes -to the Gop's attribute list. How and when do PDUs belong to Gops is described +For every PDU, MATE checks if it belongs to an existing "Group of PDUs" (GOP). +If it does, it assigns the PDU to that GOP and moves any new relevant attributes +to the GOP's attribute list. How and when do PDUs belong to GOPs is described in the configuration file as well. -Every time a Gop is assigned a new PDU, MATE will check if it matches the -conditions to make it belong to a "Group of Groups" (Gog). Naturally the -conditions that make a Gop belong to a Gog are taken from the configuration +Every time a GOP is assigned a new PDU, MATE will check if it matches the +conditions to make it belong to a "Group of Groups" (GOG). Naturally the +conditions that make a GOP belong to a GOG are taken from the configuration file as well. Once MATE is done analyzing the frame it will be able to create a "protocol" -tree for each frame based on the PDUs, the Gops they belong to and naturally any -Gogs the former belongs to. +tree for each frame based on the PDUs, the GOPs they belong to and naturally any +GOGs the former belongs to. How to tell MATE what to extract, how to group it and then how to relate those groups is made using AVPs and AVPLs. -Information in MATE is contained in Attribute/Value Pairs (AVPs). AVPs are made +Information in MATE is contained in Attribute Value Pairs (AVPs). AVPs are made of two strings: the name and the value. AVPs are used in the configuration and there they have an operator as well. There are various ways AVPs can be matched against each other using those operators. -AVPs are grouped into AVP Lists (AVPLs). PDUs, Gops and Gogs have an AVPL each. +AVPs are grouped into AVP Lists (AVPLs). PDUs, GOPs and GOGs have an AVPL each. Their AVPLs will be matched in various ways against others coming from the configuration file. MATE will be instructed how to extract AVPs from frames in order to create a PDU with an AVPL. It will be instructed as well, how to match that AVPL against the AVPLs of other similar PDUs in order to relate them. In MATE the relationship -between PDUs is a Gop, it has an AVPL as well. MATE will be configured with other -AVPLs to operate against the Gop's AVPL to relate Gops together into Gogs. +between PDUs is a GOP, it has an AVPL as well. MATE will be configured with other +AVPLs to operate against the GOP's AVPL to relate GOPs together into GOGs. A good understanding on how AVPs and AVPLs work is fundamental to understand how MATE works. +===== About MATE + +MATE was originally written by Luis Ontanon, a Telecommunications systems +troubleshooter, as a way to save time filtering out the packets of a single call +from huge capture files using just the calling number. Later he used the time he +had saved to make it flexible enough to work with protocols other than the ones +he was directly involved with. + [#AVP] -==== Attribute Value Pairs +==== Attribute Value Pairs (AVP) -Information used by MATE to relate different frames is contained in Attribute/ +Information used by MATE to relate different frames is contained in Attribute Value Pairs (AVPs). AVPs are made of two strings - the name and the value. When AVPs are used in the configuration, an operator is defined as well. There are various ways AVPs can be matched against each other using those operators. @@ -129,16 +138,20 @@ various ways AVPs can be matched against each other using those operators. another_name= "1234 is the value" ---- -The name is a string used to refer to a "kind" of an AVP. Two AVPs won't match +The name is a string used to refer to a "type" of an AVP. Two AVPs won't match unless their names are identical. + +The name must start with a lowercase letter (a-z) and can contain only alphanumeric characters +(a-zA-Z0-9) and the special characters "_", "-", and ".". The name ends with an operator. + You should not use uppercase characters in names, or names that start with “.” or “_”. Capitalized names are reserved for configuration parameters (we'll call them keywords); nothing forbids you from using capitalized strings for other things as well but it probably would be confusing. I'll avoid using capitalized words for anything but the keywords in this document, the reference manual, the examples and the base library. Names that start with a “.” would be very confusing as well -because in the old grammar, AVPL transformations use names starting with a “.” to +because in the old grammar, AVPL transforms use names starting with a “.” to indicate they belong to the replacement AVPL. The value is a string that is either set in the configuration (for configuration @@ -146,17 +159,16 @@ AVPs) or by Wireshark while extracting interesting fields from a frame's tree. The values extracted from fields use the same representation as they do in filter strings except that no quotes are used. -The name can contain only alphanumeric characters, "_", and ".". The name ends -with an operator. - -The value will be dealt with as a string even if it is a number. If there are +*The value will be dealt with as a string even if it is a number.* If there are any spaces in the value, the value must be between quotes "". +Values that are also keywords such as True and False should also be wrapped +in quotes ("True", "False"). ---- - ip_addr=10.10.10.11, - tcp_port=1234, - binary_data=01:23:45:67:89:ab:cd:ef, - parameter12=0x23aa, + ip_addr=10.10.10.11 + tcp_port=1234 + binary_data=01:23:45:67:89:ab:cd:ef + parameter12=0x23aa parameter_with_spaces="this value has spaces" ---- @@ -165,7 +177,7 @@ Remember two AVPs won't match unless their names are identical. In MATE, match operations are always made between the AVPs extracted from frames (called data AVPs) and the configuration's AVPs. -Currently defined MATE's AVP match operators are: +Currently defined MATE AVP match operators are: * <<Equal,Equal>> _=_ will match if the string given completely matches the data AVP's value string @@ -186,20 +198,22 @@ higher than the string given * <<Exists,Exists>> _?_ (the ? can be omitted) will match as far as a data AVP of the given name exists -==== AVP lists +==== AVP lists (AVPL) An AVPL is a set of diverse AVPs that can be matched against other AVPLs. Every -PDU, Gop and Gog has an AVPL that contains the information regarding it. The -rules that MATE uses to group Pdus and Gops are AVPL operations. +PDU, GOP and GOG has an AVPL that contains the information regarding it. The +rules that MATE uses to group PDUs and GOPs are AVPL operations. -There will never be two identical AVPs in a given AVPL. However, we can have +*There will never be two identical AVPs in a given AVPL.* However, we can have more than one AVP with the same name in an AVPL as long as their values are different. Some AVPL examples: ---- - ( addr=10.20.30.40, addr=192.168.0.1, tcp_port=21, tcp_port=32534, user_cmd=PORT, data_port=12344, data_addr=192.168.0.1 ) - ( addr=10.20.30.40, addr=192.168.0.1, channel_id=22:23, message_type=Setup, calling_number=1244556673 ) + ( addr=10.20.30.40, addr=192.168.0.1, tcp_port=21, tcp_port=32534, user_cmd=PORT, + data_port=12344, data_addr=192.168.0.1 ) + ( addr=10.20.30.40, addr=192.168.0.1, channel_id=22:23, message_type=Setup, + calling_number=1244556673 ) ( addr=10.20.30.40, addr=192.168.0.1, ses_id=01:23:45:67:89:ab:cd:ef ) ( user_id=pippo, calling_number=1244556673, assigned_ip=10.23.22.123 ) ---- @@ -207,74 +221,74 @@ Some AVPL examples: In MATE there are two types of AVPLs: * data AVPLs that contain information extracted from frames. -* operation AVPLs that come from the configuration and are used to tell MATE how +* configuration AVPLs that come from the configuration and are used to tell MATE how to relate items based on their data AVPLs. -Data AVPLs can be operated against operation AVPLs in various ways: +Data AVPLs can be operated against configuration AVPLs in various ways: * <<Loose,Loose Match>>: Will match if at least one of the AVPs of each AVPL -match. If it matches it will return an AVPL containing all AVPs from the operand -AVPL that did match the operator's AVPs. -* <<Every,"Every" Match>>: Will match if none of the AVPs of the operator AVPL -fails to match a present AVP in the operand AVPL, even if not all of the -operator's AVPs have a match. If it matches it will return an AVPL containing -all AVPs from the operand AVPL that did match one AVP in the operator AVPL. -* <<Strict,Strict Match>>: Will match if and only if every one of the operator's -AVPs have at least one match in the operand AVPL. If it matches it will return -an AVPL containing the AVPs from the operand that matched. +match. If it matches it will return an AVPL containing all AVPs from the data +AVPL that did match the configuration AVPs. +* <<Every,"Every" Match>>: Will match if none of the AVPs of the configuration AVPL +fails to match a present AVP in the data AVPL, even if not all of the +configuration AVPs have a match. If it matches it will return an AVPL containing +all AVPs from the data AVPL that did match one AVP in the configuration AVPL. +* <<Strict,Strict Match>>: Will match if and only if every one of the configuration +AVPs have at least one match in the data AVPL. If it matches it will return +an AVPL containing the AVPs from the data AVPL that matched. * There's also a <<Merge,Merge>> operation that is to be performed between AVPLs -where all the AVPs that don't exist in the operand AVPL but exist in the operand -will be added to the operand AVPL. -* Other than that, there are <<Transform,Transformations>> - a combination +where all the AVPs that don't exist in the data AVPL but exist in the configuration +will be added to the data AVPL. +* Other than that, there are <<Transform,Transforms>> - a combination of a match AVPL and an AVPL to merge. -==== MATE Analysis +=== MATE Frame Analysis MATE's analysis of a frame is performed in three phases: -* In the first phase, MATE attempts to extract a MATE Pdu from the frame's -protocol tree. MATE will create a Pdu if MATE's config has a _Pdu_ declaration +* In the first phase, MATE attempts to extract a MATE PDU from the frame's +protocol tree. MATE will create a PDU if MATE's config has a _Pdu_ declaration whose _Proto_ is contained in the frame. -* In the second phase, if a Pdu has been extracted from the frame, MATE will try -to group it to other Pdus into a Gop (Group of Pdus) by matching the key -criteria given by a _Gop_ declaration. If there is no Gop yet with the key -criteria for the Pdu, MATE will try to create a new Gop for it if it matches the -_Start_ criteria given in the Gop declaration. +* In the second phase, if a PDU has been extracted from the frame, MATE will try +to group it to other PDUs into a GOP (Group of PDUs) by matching the key +criteria given by a _Gop_ declaration. If there is no GOP yet with the key +criteria for the PDU, MATE will try to create a new GOP for it if it matches the +_Start_ criteria given in the _Gop_ declaration. -* In the third phase, if there's a Gop for the Pdu, MATE will try to group this -Gop with other Gops into a Gog (Group of Groups) using the criteria given by the -_Member_ criteria of a Gog declaration. +* In the third phase, if there's a GOP for the PDU, MATE will try to group this +GOP with other GOPs into a GOG (Group of Groups) using the criteria given by the +_Member_ criteria of a _Gog_ declaration. +.MATE Analysis (PDU->GOP->GOG) flowchart image::images/ws-mate-analysis.png[] The extraction and matching logic comes from MATE's configuration; MATE's -configuration file is declared by the _mate.config_ preference. By default it is +configuration file is specified by the _mate.config_ preference. By default it is an empty string which means: do not configure MATE. The config file tells MATE what to look for in frames; How to make PDUs out of -it; How will PDUs be related to other similar PDUs into Gops; And how Gops -relate into Gogs. +it; How will PDUs be related to other similar PDUs into GOPs; And how GOPs +relate into GOGs. The MATE configuration file is a list of declarations. There are 4 types of -declarations: _Transform_, _Pdu_, _Gop_ and _Gog_. +declarations: _Transform_, _Pdu_, _Gop_, and _Gog_. A _Transform_ block must be +before any of the other block declarations that may use it. -===== Mate's PDU's +==== Create PDUs (Phase 1) MATE will look in the tree of every frame to see if there is useful data to extract, and if there is, it will create one or more PDU objects containing the useful information. -The first part of MATE's analysis is the "PDU extraction"; there are various -"Actions" that are used to instruct MATE what has to be extracted from the -current frame's tree into MATE's PDUs. +The first part of MATE's analysis is the "PDU extraction". -====== PDU data extraction +===== PDU data extraction -MATE will make a Pdu for each different proto field of Proto type present in the +MATE will make a PDU for each different proto field of _Proto_ type present in the frame. MATE will fetch from the field's tree those fields that are defined in the <<Pdu>> declaration whose initial offset in the frame is within the -boundaries of the current Proto and those of the given Transport and Payload +boundaries of the current _Proto_ and those of the given _Transport_ and _Payload_ statements. ---- @@ -284,18 +298,16 @@ Pdu dns_pdu Proto dns Transport ip { Extract dns_resp From dns.flags.response; }; ---- -MATE will make a Pdu for each different proto field of Proto type present in the -frame. MATE will fetch from the field's tree those fields that are defined in -the <<Pdu>> AVPL whose initial offset in the frame is within the boundaries of -the current Proto and those of the various assigned Transports. +.Wireshark window - fields for PDU extraction image::images/ws-mate-dns_pane.png[] -Once MATE has found a _Proto_ field for which to create a Pdu from the frame it +Once MATE has found a _Proto_ field for which to create a PDU from the frame it will move backwards in the frame looking for the respective _Transport_ fields. After that it will create AVPs named as each of those given in the rest of the AVPL for every instance of the fields declared as its values. +.Frame fields mapped to PDU attributes image::images/ws-mate-dns_pdu.png[] Sometimes we need information from more than one _Transport_ protocol. In that @@ -303,61 +315,68 @@ case MATE will check the frame looking backwards to look for the various _Transport_ protocols in the given stack. MATE will choose only the closest transport boundary per "protocol" in the frame. -This way we'll have all Pdus for every _Proto_ that appears in a frame match its +This way we'll have all PDUs for every _Proto_ that appears in a frame match its relative transports. ---- Pdu isup_pdu Proto isup Transport mtp3/ip { - Extract m3pc From mtp3.dpc; - Extract m3pc From mtp3.opc; - Extract cic From isup.cic; - Extract addr From ip.addr; - Extract isup_msg From isup.message_type; + Extract addr From ip.addr; + + Extract m3pc From mtp3.dpc; + Extract m3pc From mtp3.opc; + + Extract cic From isup.cic; + Extract isup_msg From isup.message_type; }; ---- +.Frame containing multiple PDUs image::images/ws-mate-isup_over_mtp3_over_ip.png[] -This allows to assign the right _Transport_ to the Pdu avoiding duplicate +This allows to assign the right _Transport_ to the PDU avoiding duplicate transport protocol entries (in case of tunneled ip over ip for example). ---- Pdu ftp_pdu Proto ftp Transport tcp/ip { - Extract addr From ip.addr; - Extract port From tcp.port; - Extract ftp_cmd From ftp.command; + Extract addr From ip.addr; + Extract port From tcp.port; + Extract ftp_cmd From ftp.command; }; ---- +.Frame with encapsulated (tunneled) fields image::images/ws-mate-ftp_over_gre.png[] Other than the mandatory _Transport_ there is also an optional _Payload_ statement, which works pretty much as _Transport_ but refers to elements after the _Proto_'s range. It is useful in those cases where the payload protocol -might not appear in a Pdu but nevertheless the Pdu belongs to the same category. +might not appear in a PDU but nevertheless the PDU belongs to the same category. ---- Pdu mmse_over_http_pdu Proto http Transport tcp/ip { - Payload mmse; + Payload mmse; + + Extract addr From ip.addr; + Extract port From tcp.port; + + Extract content From http.content_type; + Extract host From http.host; + Extract http_rq From http.request; + Extract method From http.request.method; + Extract resp From http.response.code; - Extract addr From ip.addr; - Extract port From tcp.port; - Extract method From http.request.method; - Extract content From http.content_type; - Extract http_rq From http.request; - Extract resp From http.response.code; - Extract host From http.host; - Extract trx From mmse.transaction_id; - Extract msg_type From mmse.message_type; - Extract notify_status From mmse.status; - Extract send_status From mmse.response_status; + Extract msg_type From mmse.message_type; + Extract notify_status From mmse.status; + Extract send_status From mmse.response_status; + Extract trx From mmse.transaction_id; }; ---- +.Extract from Payload fields image::images/ws-mate-mmse_over_http.png[] -====== Conditions on which to create PDUs +===== Conditions on which to create PDUs There might be cases in which we won't want MATE to create a PDU unless some of its extracted attributes meet or do not meet some criteria. For that we use the @@ -367,8 +386,8 @@ _Criteria_ statements of the _Pdu_ declarations. Pdu isup_pdu Proto isup Transport mtp3/ip { ... - // MATE will create isup_pdu PDUs only when there is not a point code '1234' - Criteria Reject Strict (m3pc=1234); + // MATE will create isup_pdu PDUs only when there is not a point code '1234' + Criteria Reject Strict (m3pc=1234); }; Pdu ftp_pdu Proto ftp Transport tcp/ip { @@ -379,81 +398,82 @@ Pdu ftp_pdu Proto ftp Transport tcp/ip { }; ---- -The _Criteria_ statement is given an action (_Accept_ or _Reject_), a match mode +The _Criteria_ statement is given an action (_Accept_ or _Reject_), a match type (_Strict_, _Loose_ or _Every_) and an AVPL against which to match the currently extracted one. -====== Transforming the attributes of a PDU +===== Transforming the attributes of a PDU -Once the fields have been extracted into the Pdu's AVPL, MATE will apply any -declared transformation to it. The way transforms are applied and how they work +Once the fields have been extracted into the PDU's AVPL, MATE will apply any +declared _Transform_ to it. The way transforms are applied and how they work is described later on. However, it's useful to know that once the AVPL for the -Pdu is created, it may be transformed before being analyzed. That way we can +PDU is created, it may be transformed before being analyzed. That way we can massage the data to simplify the analysis. -====== MATE's PDU tree +===== MATE's PDU tree -Every successfully created Pdu will add a MATE tree to the frame dissection. If -the Pdu is not related to any Gop, the tree for the Pdu will contain just the -Pdu's info, if it is assigned to a Gop, the tree will also contain the Gop items, -and the same applies for the Gog level. +Every successfully created PDU will add a MATE tree to the frame dissection. If +the PDU is not related to any GOP, the tree for the PDU will contain just the +PDU's info. If it is assigned to a GOP, the tree will also contain the GOP items, +and the same applies for the GOG level. ---- mate dns_pdu:1 dns_pdu: 1 dns_pdu time: 3.750000 dns_pdu Attributes - dns_resp: 0 - dns_id: 36012 + dns_resp: False + dns_id: 0x8cac addr: 10.194.4.11 addr: 10.194.24.35 ---- -The Pdu's tree contains some filterable fields +The PDU's tree contains some filterable fields -* _mate.dns_pdu_ will contain the number of the "dns_pdu" Pdu +* _mate.dns_pdu_ will contain the number of the "dns_pdu" PDU * _mate.dns_pdu.RelativeTime_ will contain the time passed since the beginning of the capture in seconds -* the tree will contain the various attributes of the Pdu as well, these will +* the tree will contain the various attributes of the PDU as well, these will all be strings (to be used in filters as "10.0.0.1", not as 10.0.0.1) ** mate.dns_pdu.dns_resp ** mate.dns_pdu.dns_id ** mate.dns_pdu.addr -===== Grouping Pdus together (Gop) +==== Grouping PDUs together (GOP) (Phase 2) -Once MATE has created the Pdus it passes to the Pdu analysis phase. During the -PDU analysis phase MATE will try to group Pdus of the same type into 'Groups of -Pdus' (aka *Gop*s) and copy some AVPs from the Pdu's AVPL to the Gop's AVPL. +Once MATE has created the PDUs it passes to the PDU analysis phase. During the +PDU analysis phase MATE will try to group PDUs of the same type into 'Groups of +PDUs' (aka *GOP*++s++) and copy some AVPs from the PDU's AVPL to the GOP's AVPL. +.Grouping PDUs (GOP) flowchart image::images/ws-mate-pdu_analysis.png[] -====== What can belong to a Gop +===== What can belong to a GOP -Given a Pdu, the first thing MATE will do is to check if there is any Gop -declaration in the configuration for the given Pdu type. If so, it will use its -_Match_ AVPL to match it against the Pdu's AVPL; if they don't match, the -analysis phase is done. If there is a match, the AVPL is the Gop's candidate key -which will be used to search the Gop's index for the Gop to which to assign -the current PDU. If there is no such Gop and this Pdu does not match the -_Start_ criteria of a Gop declaration for the Pdu type, the Pdu will remain +Given a PDU, the first thing MATE will do is to check if there is any GOP +declaration in the configuration for the given PDU type. If so, it will use its +_Match_ AVPL to match it against the PDU's AVPL; if they don't match, the +analysis phase is done. If there is a match, the AVPL is the GOP's candidate key +which will be used to search the index of GOPs for the GOP to which to assign +the current PDU. If there is no such GOP and this PDU does not match the +_Start_ criteria of a _Gop_ declaration for the PDU type, the PDU will remain unassigned and only the analysis phase will be done. ---- -Gop ftp_ses On ftp_pdu Match (addr, addr, port, port); -Gop dns_req On dns_pdu Match (addr, addr, dns_id); -Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic); +Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {...}; +Gop dns_req On dns_pdu Match (addr, addr, dns_id) {...}; +Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {...}; ---- -====== Start of a Gop +===== Start of a GOP -If there was a match, the candidate key will be used to search the Gop's index -to see if there is already a Gop matching the Gop's key the same way. If there -is such a match in the Gops collection, and the PDU doesn't match the _Start_ -AVPL for its kind, the PDU will be assigned to the matching Gop. If it is a -_Start_ match, MATE will check whether or not that Gop has been already -stopped. If the Gop has been stopped, a new Gop will be created and will replace -the old one in the Gop's index. +If there was a match, the candidate key will be used to search the index of GOPs +to see if there is already a GOP matching the GOP's key the same way. If there +is such a match in the GOPs collection, and the PDU doesn't match the _Start_ +AVPL for its type, the PDU will be assigned to the matching GOP. If it is a +_Start_ match, MATE will check whether or not that GOP has been already +stopped. If the GOP has been stopped, a new GOP will be created and will replace +the old one in the index of GOPs. ---- Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) { @@ -461,7 +481,7 @@ Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) { }; Gop dns_req On dns_pdu Match (addr, addr, dns_id) { - Start (dns_resp=0); + Start (dns_resp="True"); }; Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) { @@ -469,14 +489,14 @@ Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) { }; ---- -If no _Start_ is given for a Gop, a Pdu whose AVPL matches an existing Gog's -key will act as the start of a Gop. +If no _Start_ is given for a GOP, a PDU whose AVPL matches an existing GOP's +key will act as the start of a GOP. -====== What goes into the Gop's AVPL +===== What goes into the GOP's AVPL -Once we know a Gop exists and the Pdu has been assigned to it, MATE will copy -into the Gop's AVPL all the attributes matching the key plus any AVPs of the -Pdu's AVPL matching the _Extra_ AVPL. +Once we know a GOP exists and the PDU has been assigned to it, MATE will copy +into the GOP's AVPL all the attributes matching the key plus any AVPs of the +PDU's AVPL matching the _Extra_ AVPL. ---- Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) { @@ -490,24 +510,24 @@ Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) { }; ---- -====== End of a Gop +===== End of a GOP -Once the Pdu has been assigned to the Gop, MATE will check whether or not the -Pdu matches the _Stop_, if it happens, MATE will mark the Gop as stopped. Even -after stopped, a Gop may get assigned new Pdus matching its key, unless such -Pdu matches _Start_. If it does, MATE will instead create a new Gop starting -with that Pdu. +Once the PDU has been assigned to the GOP, MATE will check whether or not the +PDU matches the _Stop_, if it happens, MATE will mark the GOP as stopped. Even +after stopped, a GOP may get assigned new PDUs matching its key, unless such +PDU matches _Start_. If it does, MATE will instead create a new GOP starting +with that PDU. ---- Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) { Start (ftp_cmd=USER); - Stop (ftp_cmd=QUIT); // The response to the QUIT command will be assigned to the same Gop + Stop (ftp_cmd=QUIT); // The response to the QUIT command will be assigned to the same GOP Extra (pasv_prt, pasv_addr); }; Gop dns_req On dns_pdu Match (addr, addr, dns_id) { - Start (dns_resp=0); - Stop (dns_resp=1); + Start (dns_resp="False"); + Stop (dns_resp="True"); }; Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) { @@ -517,92 +537,102 @@ Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) { }; ---- -If no _Stop_ criterium is stated for a given Gop, the Gop will be stopped as -soon as it is created. However, as with any other Gop, Pdus matching the Gop's -key will still be assigned to the Gop unless they match a _Start_ condition, -in which case a new Gop using the same key will be created. +If no _Stop_ criterium is stated for a given GOP, the GOP will be stopped as +soon as it is created. However, as with any other GOP, PDUs matching the GOP's +key will still be assigned to the GOP unless they match a _Start_ condition, +in which case a new GOP using the same key will be created. To group multiple +PDUs that match the _Start_, add a bogus _Stop_ such as +---- +Gop frame_ses On frame_pdu Match (frame_time) { + Start (frame_time); + Stop (frame_time="FOO"); +}; +---- -===== Gop's tree +===== GOP's tree -For every frame containing a Pdu that belongs to a Gop, MATE will create a tree -for that Gop. +For every frame containing a PDU that belongs to a GOP, MATE will create a tree +for that GOP. The example below represents the tree created by the _dns_pdu_ and _dns_req_ examples. ---- ... -mate dns_pdu:6->dns_req:1 +MATE dns_pdu:6->dns_req:1 dns_pdu: 6 dns_pdu time: 2.103063 dns_pdu time since beginning of Gop: 2.103063 - dns_req: 1 - dns_req Attributes - dns_id: 36012 - addr: 10.194.4.11 - addr: 10.194.24.35 - dns_req Times - dns_req start time: 0.000000 - dns_req hold time: 2.103063 - dns_req duration: 2.103063 - dns_req number of PDUs: 2 - Start PDU: in frame 1 - Stop PDU: in frame 6 (2.103063 : 2.103063) dns_pdu Attributes - dns_resp: 1 - dns_id: 36012 + dns_resp: True + dns_id: 0x8cac + addr: 10.194.4.11 + addr: 10.194.24.35 + dns_req: 1 + GOP Key: addr=10.194.4.11; addr=10.194.24.35; dns_id=0x8cac; + dns_req Attributes + dns_id: 0x8cac addr: 10.194.4.11 addr: 10.194.24.35 + dns_req Times + dns_req start time: 0.000000 + dns_req hold time: 2.103063 + dns_req duration: 2.103063 + dns_req number of PDUs: 2 + Start PDU: in frame 1 + Stop PDU: in frame 6 (2.103063 : 2.103063) + ---- -Other than the pdu's tree, this one contains information regarding the -relationship between the Pdus that belong to the Gop. That way we have: +Other than the PDU's tree, this one contains information regarding the +relationship between the PDUs that belong to the GOP. That way we have: -* mate.dns_req which contains the id of this dns_req Gop. This will be present -in frames that belong to dns_req Gops. +* mate.dns_req which contains the id of this dns_req GOP. This will be present +in frames that belong to dns_req GOPs. * mate.dns_req.dns_id and mate.dns_req.addr which represent the values of the -attributes copied into the Gop. -* the timers of the Gop +attributes copied into the GOP. +* the timers of the GOP ** mate.dns_req.StartTime time (in seconds) passed since beginning of capture -until Gop's start. -** mate.dns_req.Time time passed between the start Pdu and the stop Pdu assigned -to this Gop (only created if a Stop criterion has been declared for the Gop and -a matching Pdu has arrived). -** mate.dns_req.Duration time passed between the start Pdu and the last Pdu -assigned to this Gop. -* mate.dns_req.NumOfPdus the number of Pdus that belong to this Gop -** a filterable list of frame numbers of the pdus of this Gop +until GOP's start. +** mate.dns_req.Time time passed between the start PDU and the stop PDU assigned +to this GOP (only created if a Stop criterion has been declared for the GOP and +a matching PDU has arrived). +** mate.dns_req.Duration time passed between the start PDU and the last PDU +assigned to this GOP. +* mate.dns_req.NumOfPdus the number of PDUs that belong to this GOP +** mate.dns_req.Pdu a filterable list of frame numbers of the PDUs of this GOP -====== Gop's timers +===== GOP's timers -Note that there are two "timers" for a Gop: +Note that there are two "timers" for a GOP: -* *Time*, which is defined only for Gops that have been Stopped, and gives the -time passed between the _Start_ and the _Stop_ Pdus. -* *Duration*, which is defined for every Gop regardless of its state, and give -the time passed between its _Start_ Pdu and the last Pdu that was assigned to -that Gop. +* *Time*, which is defined only for GOPs that have been Stopped, and gives the +time passed between the _Start_ and the _Stop_ PDUs. +* *Duration*, which is defined for every GOP regardless of its state, and give +the time passed between its _Start_ PDU and the last PDU that was assigned to +that GOP. So: -* we can filter for Pdus that belong to Gops that have been Stopped with +* we can filter for PDUs that belong to GOPs that have been Stopped with *mate.xxx.Time* -* we can filter for Pdus that belong to unstopped Gops with *mate.xxx && -mate.xxx.Time* -* we can filter for Pdus that belong to stopped Gops using *mate.xxx.Duration* -* we can filter for Pdus that belong to Gops that have taken more (or less) time +* we can filter for PDUs that belong to unstopped GOPs with *mate.xxx && +!mate.xxx.Time* +* we can filter for PDUs that belong to stopped GOPs using *mate.xxx.Duration* +* we can filter for PDUs that belong to GOPs that have taken more (or less) time that 0.5s to complete with *mate.xxx.Time > 0.5* (you can try these also as color filters to find out when response times start to grow) -===== Grouping Gops together (Gog) +==== Grouping GOPs together (GOG) (Phase 3) -When Gops are created, or whenever their AVPL changes, Gops are (re)analyzed to -check if they match an existent group of groups (Gog) or can create a new one. -The Gop analysis is divided into two phases. In the first phase, the still -unassigned Gop is checked to verify whether it belongs to an already existing -Gog or may create a new one. The second phase eventually checks the Gog and -registers its keys in the Gogs index. +When GOPs are created, or whenever their AVPL changes, GOPs are (re)analyzed to +check if they match an existent group of groups (GOG) or can create a new one. +The GOP analysis is divided into two phases. In the first phase, the still +unassigned GOP is checked to verify whether it belongs to an already existing +GOG or may create a new one. The second phase eventually checks the GOG and +registers its keys in the index of GOGs. +.Grouping GOPs (GOG) flowchart image::images/ws-mate-gop_analysis.png[] There are several reasons for the author to believe that this feature needs to @@ -610,41 +640,41 @@ be reimplemented, so probably there will be deep changes in the way this is done in the near future. This section of the documentation reflects the version of MATE as of Wireshark 0.10.9; in future releases this will change. -====== Declaring a Group Of Groups +===== Declaring a Group Of Groups (GOG) -The first thing we have to do configuring a Gog is to tell MATE that it exists. +The first thing we have to do configuring a GOG is to tell MATE that it exists. ---- -Gog web_use { +Gog http_use { ... }; ---- -====== Telling MATE what could be a Gog member +===== Telling MATE what could be a GOG member -Then we have to tell MATE what to look for a match in the candidate Gops. +Then we have to tell MATE what to look for a match in the candidate GOPs. ---- -Gog web_use { +Gog http_use { Member http_ses (host); Member dns_req (host); }; ---- -====== Getting interesting data into the Gop +===== Getting interesting data into the GOG Most often, also other attributes than those used for matching would be -interesting. In order to copy from Gop to Gog other interesting attributes, we -might use _Extra_ like we do for Gops. +interesting. In order to copy from GOP to GOG other interesting attributes, we +might use _Extra_ like we do for GOPs. ---- -Gog web_use { +Gog http_use { ... Extra (cookie); }; ---- -====== Gog's tree +===== GOG's tree ---- mate http_pdu:4->http_req:2->http_use:1 @@ -669,11 +699,16 @@ mate http_pdu:4->http_req:2->http_use:1 We can filter on: -* *mate.http_use.Duration* time elapsed between the first frame of a Gog and the last one assigned to it. -* the attributes passed to the Gog +* the timers of the GOG +** *mate.http_use.StartTime* time (in seconds) passed since beginning of capture until GOG's start. +** *mate.http_use.Duration* time elapsed between the first frame of a GOG and the last one assigned to it. +* the attributes passed to the GOG ** *mate.http_use.host* +* *mate.http_use.NumOfGops* the number of GOPs that belong to this GOG +* *mate.http_use.GopStart* the start frame of a GOP +* *mate.http_use.GopStop* the stop frame of a GOP -===== AVPL Transforms +==== Adjust data (AVPL Transforms) A Transform is a sequence of Match rules optionally completed with modification of the match result by an additional AVPL. Such modification may be an Insert @@ -681,26 +716,26 @@ of the match result by an additional AVPL. Such modification may be an Insert AVPL before it is processed further. They come to be very helpful in several cases. -====== Syntax +===== Syntax -AVPL Transformations are declared in the following way: +AVPL Transforms are declared in the following way: ---- - Transform name { - Match [Strict|Every|Loose] match_avpl [Insert|Replace] modify_avpl ; - ... - }; +Transform name { + Match [Strict|Every|Loose] match_avpl [Insert|Replace] modify_avpl; + ... +}; ---- -The *name* is the handle to the AVPL transformation. It is used to refer to the +The *name* is the handle to the AVPL transform. It is used to refer to the transform when invoking it later. The _Match_ declarations instruct MATE what and how to match against the data AVPL and how to modify the data AVPL if the match succeeds. They will be executed in the order they appear in the config file whenever they are invoked. -The optional match mode qualifier (_Strict_, _Every_, or _Loose_) is used -to choose the match mode as explained above; _Strict_ is a default value which +The optional match type qualifier (_Strict_, _Every_, or _Loose_) is used +to choose the <<Match,Match type>>; _Strict_ is the default value which may be omitted. The optional modification mode qualifier instructs MATE how the modify AVPL @@ -708,17 +743,28 @@ should be used: * the default value _Insert_ (which may be omitted) causes the _modify_avpl_ to be *merged* to the existing data AVPL, -* the _Replace_ causes all the matching AVPs from the data AVPL to be +* _Replace_ causes all the matching AVPs from the data AVPL to be *replaced* by the _modify_avpl_. The _modify_avpl_ may be an empty one; this comes useful in some cases for both _Insert_ and _Replace_ modification modes. +---- +Transform rm_client_from_http_resp1 { + Match (http_rq); //first match wins so the request won't get the not_rq attribute inserted + Match Every (addr) Insert (not_rq); //this line won't be evaluated if the first one matched so not_rq won't be inserted to requests +}; + +Transform rm_client_from_http_resp2 { + Match (not_rq, client) Replace (); //replace "client and not_rq" with nothing +}; +---- + Examples: ---- - Transform insert_name_and { - Match Strict (host=10.10.10.10, port=2345) Insert (name=JohnDoe); +Transform insert_name_and { + Match Strict (host=10.10.10.10, port=2345) Insert (name=JohnDoe); }; ---- @@ -727,7 +773,7 @@ port=2345 ---- Transform insert_name_or { - Match Loose (host=10.10.10.10, port=2345) Insert (name=JohnDoe); + Match Loose (host=10.10.10.10, port=2345) Insert (name=JohnDoe); }; ---- @@ -736,7 +782,7 @@ port=2345 ---- Transform replace_ip_address { - Match (host=10.10.10.10) Replace (host=192.168.10.10); + Match (host=10.10.10.10) Replace (host=192.168.10.10); }; ---- @@ -744,7 +790,7 @@ replaces the original host=10.10.10.10 by host=192.168.10.10 ---- Transform add_ip_address { - Match (host=10.10.10.10) (host=192.168.10.10); + Match (host=10.10.10.10) (host=192.168.10.10); }; ---- @@ -753,7 +799,7 @@ host=10.10.10.10 in it too ---- Transform replace_may_be_surprising { - Match Loose (a=aaaa, b=bbbb) Replace (c=cccc, d=dddd); + Match Loose (a=aaaa, b=bbbb) Replace (c=cccc, d=dddd); }; ---- @@ -765,10 +811,10 @@ intact, * (a=aaaa, b=bbbb) gets transformed to (c=cccc, d=dddd) because both a=aaaa and b=bbbb did match. -====== Usage +===== Usage -Once declared, Transforms can be added to the declarations of PDUs, Gops or -Gogs. This is done by adding the _Transform name_list_ statement to the +Once declared, Transforms can be added to the declarations of PDUs, GOPs or +GOGs. This is done by adding the _Transform name_list_ statement to the declaration: ---- @@ -781,15 +827,16 @@ Pdu my_proto_pdu Proto my_proto Transport ip { * In case of PDU, the list of transforms is applied against the PDU's AVPL after its creation. -* In case of Gop and Gog, the list of transforms is applied against their +* In case of GOP and GOG, the list of transforms is applied against their respective AVPLs when they are created and every time they change. ===== Operation +.Applying Transform flowchart image::images/ws-mate-transform.png[] -* A list of previously declared Transforms may be given to every Item (Pdu, Gop, -or Gog), using the Transform statement. +* A list of previously declared Transforms may be given to every Item (_Pdu_, _Gop_, +or _Gog_), using the _Transform_ statement. * Every time the AVPL of an item changes, it will be operated against *all* the Transforms on the list given to that item. The Transforms on the list are applied left to right. @@ -799,75 +846,83 @@ tried or until one of them succeeds. MATE's Transforms can be used for many different things, like: -====== Multiple Start/Stop conditions for a Gop +===== Multiple Start/Stop conditions for a GOP -Using _Transforms_ we can add more than one start or stop condition to a Gop. +Using _Transforms_ we can add more than one start or stop condition to a GOP. ---- Transform start_cond { - Match (attr1=aaa,attr2=bbb) (msg_type=start); - Match (attr3=www,attr2=bbb) (msg_type=start); - Match (attr5^a) (msg_type=stop); - Match (attr6$z) (msg_type=stop); + Match (attr1=aaa,attr2=bbb) (msg_type=start); + Match (attr3=www,attr2=bbb) (msg_type=start); + Match (attr5^a) (msg_type=stop); + Match (attr6$z) (msg_type=stop); }; Pdu pdu ... { - ... - Transform start_cond; + ... + Transform start_cond; } Gop gop ... { - Start (msg_type=start); - Stop (msg_type=stop); - ... + Start (msg_type=start); + Stop (msg_type=stop); + ... } ---- -====== Marking Gops and Gogs to filter them easily +===== Marking GOPs and GOGs to filter them easily ---- Transform marks { - Match (addr=10.10.10.10, user=john) (john_at_host); - Match (addr=10.10.10.10, user=tom) (tom_at_host); + Match (addr=10.10.10.10, user=john) (john_at_host); + Match (addr=10.10.10.10, user=tom) (tom_at_host); } ... Gop my_gop ... { - ... - Transform marks; + ... + Transform marks; } ---- -After that we can use a display filter *mate.gop.john_at_host* or -*mate.gop.tom_at_host* +After that we can use a display filter *mate.my_gop.john_at_host* or +*mate.my_gop.tom_at_host* -====== Adding direction knowledge to MATE +===== Adding (Insert) direction knowledge to MATE ---- Transform direction_as_text { - Match (src=192.168.0.2, dst=192.168.0.3) Replace (direction=from_2_to_3); - Match (src=192.168.0.3, dst=192.168.0.2) Replace (direction=from_3_to_2); + Match (src=192.168.0.2, dst=192.168.0.3) Insert (direction=from_2_to_3); + Match (src=192.168.0.3, dst=192.168.0.2) Insert (direction=from_3_to_2); }; Pdu my_pdu Proto my_proto Transport tcp/ip { - Extract src From ip.src; - Extract dst From ip.dst; - Extract addr From ip.addr; - Extract port From tcp.port; - Extract start From tcp.flags.syn; - Extract stop From tcp.flags.fin; - Extract stop From tcp.flags.rst; - Transform direction_as_text; + Extract src From ip.src; + Extract dst From ip.dst; + Extract addr From ip.addr; + Extract port From tcp.port; + Extract start From tcp.flags.syn; + Extract stop From tcp.flags.fin; + Extract stop From tcp.flags.rst; + Transform direction_as_text; } Gop my_gop On my_pdu Match (addr,addr,port,port) { - ... - Extra (direction); + ... + Extra (direction); } ---- -====== NAT +The original example (below) would delete _src_ and _dst_ then add _direction_. +---- +Transform direction_as_text { + Match (src=192.168.0.2, dst=192.168.0.3) Replace (direction=from_2_to_3); + Match (src=192.168.0.3, dst=192.168.0.2) Replace (direction=from_3_to_2); +}; +---- + +===== NAT NAT can create problems when tracing, but we can easily work around it by Transforming the NATed IP address and the Ethernet address of the router into @@ -875,34 +930,26 @@ the non-NAT address: ---- Transform denat { - Match (addr=192.168.0.5, ether=01:02:03:04:05:06) Replace (addr=123.45.67.89); - Match (addr=192.168.0.6, ether=01:02:03:04:05:06) Replace (addr=123.45.67.90); - Match (addr=192.168.0.7, ether=01:02:03:04:05:06) Replace (addr=123.45.67.91); + Match (addr=192.168.0.5, ether=01:02:03:04:05:06) Replace (addr=123.45.67.89); + Match (addr=192.168.0.6, ether=01:02:03:04:05:06) Replace (addr=123.45.67.90); + Match (addr=192.168.0.7, ether=01:02:03:04:05:06) Replace (addr=123.45.67.91); } Pdu my_pdu Proto my_proto transport tcp/ip/eth { - Extract ether From eth.addr; - Extract addr From ip.addr; - Extract port From tcp.port; - Transform denat; + Extract ether From eth.addr; + Extract addr From ip.addr; + Extract port From tcp.port; + Transform denat; } ---- -==== About MATE - -MATE was originally written by Luis Ontanon, a Telecommunications systems -troubleshooter, as a way to save time filtering out the packets of a single call -from huge capture files using just the calling number. Later he used the time he -had saved to make it flexible enough to work with protocols other than the ones -he was directly involved with. - [#ChMateConfigurationTutorial] === MATE's configuration tutorial -We'll show a MATE configuration that first creates Gops for every DNS and HTTP -request, then it ties the Gops together in a Gop based on the host. Finally, -we'll separate into different Gogs request coming from different users. +We'll show a MATE configuration that first creates GOPs for every DNS and HTTP +request, then it ties the GOPs together in a GOG based on the host. Finally, +we'll separate into different GOGs request coming from different users. With this MATE configuration loaded we can: @@ -915,78 +962,78 @@ to isolate DNS and HTTP packets related to a visit of a certain user. that take more than 1.5 seconds to complete. The complete config file is available on the Wireshark Wiki: -https://gitlab.com/wireshark/wireshark/-/wikis/Mate/Tutorial +{wireshark-wiki-url}Mate/Tutorial Note: This example uses _dns.qry.name_ which is defined since Wireshark -version 0.10.9. Supposing you have a mate plugin already installed you can test +version 0.10.9. Supposing you have a MATE plugin already installed you can test it with the current Wireshark version. -==== A Gop for DNS requests +==== A GOP for DNS requests -First we'll tell MATE how to create a Gop for each DNS request/response. +First we'll tell MATE how to create a GOP for each DNS request/response. -MATE needs to know what makes a DNS PDU. We describe it this using a Pdu +MATE needs to know what makes a DNS PDU. We describe it using a _Pdu_ declaration: ---- Pdu dns_pdu Proto dns Transport ip { - Extract addr From ip.addr; - Extract dns_id From dns.id; - Extract dns_resp From dns.flags.response; + Extract addr From ip.addr; + Extract dns_id From dns.id; + Extract dns_resp From dns.flags.response; }; ---- -Using _Proto dns_ we tell MATE to create Pdus every time it finds _dns_. Using +Using _Proto dns_ we tell MATE to create PDUs every time it finds _dns_. Using _Transport ip_ we inform MATE that some of the fields we are interested are in the _ip_ part of the frame. Finally, we tell MATE to import _ip.addr_ as _addr_, _dns.id_ as _dns_id_ and _dns.flags.response_ as _dns_resp_. Once we've told MATE how to extract _dns_pdus_ we'll tell it how to match -requests and responses and group them into a Gop. For this we'll use a _Gop_ -declaration to define the Gop, and then, _Start_ and _Stop_ statements to -tell it when the Gop starts and ends. +requests and responses and group them into a GOP. For this we'll use a _Gop_ +declaration to define the GOP, and then, _Start_ and _Stop_ statements to +tell it when the GOP starts and ends. ---- Gop dns_req On dns_pdu Match (addr,addr,dns_id) { - Start (dns_resp=0); - Stop (dns_resp=1); + Start (dns_resp="False"); + Stop (dns_resp="True"); }; ---- -Using the *Gop* declaration we tell MATE that the *Name* of the Gop is _dns_req_, -that _dns_pdus_ can become members of the Gop, and what is the key used to match -the Pdus to the Gop. +Using the *Gop* declaration we tell MATE that the *Name* of the GOP is _dns_req_, +that _dns_pdus_s can become members of the GOP, and what is the key used to match +the PDUs to the GOP. -The key for this Gop is _"addr, addr, dns_id"_. That means that in order to -belong to the same Gop, _dns_pdus_ have to have both addresses and the +The key for this GOP is _"addr, addr, dns_id"_. That means that in order to +belong to the same GOP, _dns_pdus_ have to have both addresses and the _request id_ identical. We then instruct MATE that a _dns_req_ starts whenever -a _dns_pdu_ matches _"dns_resp=0"_ and that it stops when another _dns_pdu_ -matches _"dns_resp=1"_. +a _dns_pdu_ matches _"dns_resp=++"++False++"++"_ and that it stops when another _dns_pdu_ +matches _"dns_resp=++"++True++"++"_. At this point, if we open a capture file using this configuration, we are able to use a display filter *mate.dns_req.Time > 1* to see only the packets of DNS requests that take more than one second to complete. We can use a display filter *mate.dns_req && ! mate.dns_req.Time* to find -requests for which no response was given. *mate.xxx.Time* is set only for Gops +requests for which no response was given. *mate.xxx.Time* is set only for GOPs that have being stopped. -==== A Gop for HTTP requests +==== A GOP for HTTP requests -This other example creates a Gop for every HTTP request. +This other example creates a GOP for every HTTP request. ---- Pdu http_pdu Proto http Transport tcp/ip { - Extract addr From ip.addr; - Extract port From tcp.port; - Extract http_rq From http.request.method; - Extract http_rs From http.response; - DiscardPduData true; + Extract addr From ip.addr; + Extract port From tcp.port; + Extract http_rq From http.request.method; + Extract http_rs From http.response; + DiscardPduData true; }; Gop http_req On http_pdu Match (addr, addr, port, port) { - Start (http_rq); - Stop (http_rs); + Start (http_rq); + Stop (http_rs); }; ---- @@ -997,64 +1044,64 @@ response header takes more than one second to come * filtering with *mate.http_req.Duration > 1.5* will show those request that take more than 1.5 seconds to complete. -You have to know that *mate.xxx.Time* gives the time in seconds between the pdu -matching the GopStart and the Pdu matching the GopStop (yes, you can create +You have to know that *mate.xxx.Time* gives the time in seconds between the PDU +matching the GOP *Start* clause and the PDU matching the GOP *Stop* clause (yes, you can create timers using this!). On the other hand, *mate.xxx.Duration* gives you the time -passed between the GopStart and the last pdu assigned to that Gop regardless -whether it is a stop or not. After the GopStop, Pdus matching the Gop's Key will -still be assigned to the same Gop as far as they don't match the GopStart, in -which case a new Gop with the same key will be created. +passed between the GOP *Start* and the last PDU assigned to that GOP regardless +whether it is a *Stop* or not. After the GOP *Stop*, PDUs matching the GOP's Key will +still be assigned to the same GOP as far as they don't match the GOP *Start*, in +which case a new GOP with the same key will be created. -==== Getting DNS and HTTP together into a Gog +==== Getting DNS and HTTP together into a GOG -We'll tie together to a single Gog all the http packets belonging to requests -and responses to a certain host and the dns request and response used to resolve -its domain name using the Pdu and Gop definitions of the previous examples +We'll tie together to a single GOG all the HTTP packets belonging to requests +and responses to a certain host and the DNS request and response used to resolve +its domain name using the _Pdu_ and _Gop_ definitions of the previous examples To be able to group DNS and HTTP requests together, we need to import into the -Pdus and Gops some part of information that both those protocols share. Once the -Pdus and Gops have been defined, we can use _Extract_ (for Pdus) and -_Extract_ (for Gops) statements to tell MATE what other protocol fields are to -be added to Pdus' and Gops' AVPLs. We add the following statements to the +PDUs and GOPs some part of information that both those protocols share. Once the +PDUs and GOPs have been defined, we can use _Extract_ (for PDUs) and +_Extract_ (for GOPs) statements to tell MATE what other protocol fields are to +be added to PDU's and GOP's AVPLs. We add the following statements to the appropriate declarations: ---- -Extract host From http.host; // to Pdu http_pdu as the last Extract in the list -Extra (host); // to Gop http_req after the Stop + Extract host From http.host; // to Pdu http_pdu as the last Extract in the list + Extra (host); // to Gop http_req after the Stop -Extract host From dns.qry.name; // to Pdu dns_pdu as the last Extract in the list -Extra (host); // to Gop dns_req after the Stop + Extract host From dns.qry.name; // to Pdu dns_pdu as the last Extract in the list + Extra (host); // to Gop dns_req after the Stop ---- Here we've told MATE to import _http.host_ into _http_pdu_ and _dns.qry.name_ into _dns_pdu_ as _host_. We also have to tell MATE to copy the _host_ -attribute from the Pdus to the Gops, we do this using _Extra_. +attribute from the PDUs to the GOPs - we do this using _Extra_. -Once we have all the data we need in Pdus and Gops, we tell MATE what makes -different Gops belong to a certain Gog. +Once we have all the data we need in PDUs and GOPs, we tell MATE what makes +different GOPs belong to a certain GOG. ---- Gog http_use { - Member http_req (host); - Member dns_req (host); - Expiration 0.75; + Member http_req (host); + Member dns_req (host); + Expiration 0.75; }; ---- -Using the _Gog_ declaration, we tell MATE to define a Gog type _Named_ -_http_use_ whose expiration is 0.75 seconds after all the Gops that belong to it -had been stopped. After that time, an eventual new Gop with the same key match -will create a new Gog instead of been added to the previous Gog. +Using the _Gog_ declaration, we tell MATE to define a GOG type named +_http_use_ whose expiration is 0.75 seconds after all the GOPs that belong to it +had been stopped. After that time, an eventual new GOP with the same key match +will create a new GOG instead of been added to the previous GOG. Using the _Member_ statements, we tell MATE that *http_req*s with the same -*host* belong to the same Gog, same thing for *dns_req*s. +*host* belong to the same GOG, same thing for *dns_req*s. -So far we have instructed mate to group every packet related to sessions towards +So far we have instructed MATE to group every packet related to sessions towards a certain host. At this point if we open a capture file and: * a display filter *mate.http_use.Duration > 5* will show only those requests that have taken more than 5 seconds to complete starting from the DNS request -and ending with the last packet of the http responses. +and ending with the last packet of the HTTP responses. * a display filter *mate.http_use.host == "www.w3c.org"* will show all the packets (both DNS and HTTP) related to the requests directed to www.w3c.org @@ -1065,60 +1112,60 @@ packets (both DNS and HTTP) related to the requests directed to www.w3c.org This configuration works fine if used for captures taken at the client's side but deeper in the network we'd got a real mess. Requests from many users get -mixed together into _http_uses_. Gogs are created and stopped almost randomly -(depending on the timing in which Gops start and stop). How do we get requests +mixed together into _http_uses_. GOGs are created and stopped almost randomly +(depending on the timing in which GOPs start and stop). How do we get requests from individual users separated from each other? MATE has a tool that can be used to resolve this kind of grouping issues. This -tool are the _Transforms_. Once defined, they can be applied against Pdus, -Gops and Gogs and they might replace or insert more attributes based on what's -there. We'll use them to create an attribute named client, using which we'll +tool are the _Transforms_. Once defined, they can be applied against PDUs, +GOPs and GOGs and they might replace or insert more attributes based on what's +there. We'll use them to create an attribute named *client*, using which we'll separate different requests. -For DNS we need the ip.src of the request moved into the Gop only from the DNS +For DNS we need the ip.src of the request moved into the GOP only from the DNS request. So we first tell MATE to import ip.src as client: ---- -Extract client From ip.src; + Extract client From ip.src; ---- -Next, we tell MATE to replace ( *dns_resp=1, client* ) with just *dns_resp=1* in -the Pdu. That way, we'll keep the attribute *client* only in the DNS request -Pdus (i.e., packets coming from the client).To do so, we have to add a -_Transform_ declaration (in this case, with just one clause) before the Pdu +Next, we tell MATE to replace ( *dns_resp="True", client* ) with just *dns_resp="True"* in +the PDU. That way, we'll keep the attribute *client* only in the DNS request +PDUs (i.e., packets coming from the client).To do so, we have to add a +_Transform_ declaration (in this case, with just one clause) before the _Pdu_ declaration which uses it: ---- Transform rm_client_from_dns_resp { - Match (dns_resp=1, client) Replace (dns_resp=1); + Match (dns_resp="True", client) Replace (dns_resp="True"); }; ---- Next, we invoke the transform by adding the following line after the _Extract_ -list of the dns_pdu Pdu: +list of the dns_pdu PDU: ---- - Transform rm_client_from_dns_resp; + Transform rm_client_from_dns_resp; ---- HTTP is a little trickier. We have to remove the attribute carrying ip.src from both the response and the "continuations" of the response, but as there is nothing to filter on for the continuations, we have to add a fake attribute -first. And then we have to remove client when the fake attribute appears. +first. And then we have to remove *client* when the fake attribute appears. This is possible due to the fact that the _Match_ clauses in the _Transform_ are executed one by one until one of them succeeds. First, we declare another two _Transforms_: ---- Transform rm_client_from_http_resp1 { - Match (http_rq); //first match wins so the request won't get the not_rq attribute inserted - Match Every (addr) Insert (not_rq); //this line won't be evaluated if the first one matched so not_rq won't be inserted to requests + Match (http_rq); //first match wins so the request won't get the not_rq attribute inserted + Match Every (addr) Insert (not_rq); //this line won't be evaluated if the first one matched so not_rq won't be inserted to requests }; Transform rm_client_from_http_resp2 { - Match (not_rq, client) Replace (); //replace "client and not_rq" with nothing (will happen only in the response and eventual parts of it) + Match (not_rq, client) Replace (); //replace "client and not_rq" with nothing (will happen only in the response and eventual parts of it) }; ---- @@ -1126,8 +1173,8 @@ Next, we add another _Extract_ statement to the _http_pdu_ declaration, and apply both _Transforms_ declared above in a proper order: ---- - Extract client From ip.src; - Transform rm_client_from_http_resp1, rm_client_from_http_resp2; + Extract client From ip.src; + Transform rm_client_from_http_resp1, rm_client_from_http_resp2; ---- In MATE, all the _Transform_s listed for an item will be evaluated, while @@ -1137,24 +1184,24 @@ first sequence before adding the _not_rq_ attribute. Then we apply the second _Transform_ which removes both _not_rq_ and _client_ if both are there. Yes, _Transform_s are cumbersome, but they are very useful. -Once we got all what we need in the Pdus, we have to tell MATE to copy the -attribute _client_ from the Pdus to the respective Gops, by adding client to -_Extra_ lists of both Gop declarations: +Once we got all what we need in the PDUs, we have to tell MATE to copy the +attribute _client_ from the PDUs to the respective GOPs, by adding client to +_Extra_ lists of both _Gop_ declarations: ---- -Extra (host, client); + Extra (host, client); ---- -On top of that, we need to modify the old declarations of Gop key to new ones -that include both _client_ and _host_. So we change the Gog *Member* +On top of that, we need to modify the old declarations of GOP key to new ones +that include both _client_ and _host_. So we change the _Gog_ *Member* declarations the following way: ---- - Member http_req (host, client); - Member dns_req (host, client); + Member http_req (host, client); + Member dns_req (host, client); ---- -Now we got it, every "usage" gets its own Gog. +Now we got it, every "usage" gets its own GOG. [#ChMateConfigurationExamples] @@ -1164,22 +1211,29 @@ The following is a collection of various configuration examples for MATE. Many of them are useless because the "conversations" facility does a better job. Anyway they are meant to help users understanding how to configure MATE. -==== TCP session +[#File_tcp_mate] +==== TCP session (tcp.mate) -The following example creates a GoP out of every TCP session. +The following example creates a GOP out of every TCP session. ---- +Transform add_tcp_stop { + Match (tcp_flags_reset="True") Insert (tcp_stop="True"); + Match (tcp_flags_fin="True") Insert (tcp_stop="True"); +}; + Pdu tcp_pdu Proto tcp Transport ip { Extract addr From ip.addr; Extract port From tcp.port; Extract tcp_start From tcp.flags.syn; - Extract tcp_stop From tcp.flags.reset; - Extract tcp_stop From tcp.flags.fin; + Extract tcp_flags_reset From tcp.flags.reset; + Extract tcp_flags_fin From tcp.flags.fin; + Transform add_tcp_stop; }; Gop tcp_ses On tcp_pdu Match (addr, addr, port, port) { - Start (tcp_start=1); - Stop (tcp_stop=1); + Start (tcp_start="True"); + Stop (tcp_stop="True"); }; Done; @@ -1187,54 +1241,54 @@ Done; This probably would do fine in 99.9% of the cases but 10.0.0.1:20->10.0.0.2:22 and 10.0.0.1:22->10.0.0.2:20 would both fall into the same gop if they happen to overlap in time. -* filtering with *mate.tcp_ses.Time > 1* will give all the sessions that last less than one second +* filtering with *mate.tcp_ses.Time > 1* will give all the sessions that last more than one second * filtering with *mate.tcp_ses.NumOfPdus < 5* will show all tcp sessions that have less than 5 packets. * filtering with *mate.tcp_ses.Id == 3* will show all the packets for the third tcp session MATE has found -==== a Gog for a complete FTP session +==== a GOG for a complete FTP session -This configuration allows to tie a complete passive ftp session (including the -data transfer) in a single Gog. +This configuration allows to tie a complete passive FTP session (including the +data transfer) in a single GOG. ---- Pdu ftp_pdu Proto ftp Transport tcp/ip { - Extract ftp_addr From ip.addr; - Extract ftp_port From tcp.port; - Extract ftp_resp From ftp.response.code; - Extract ftp_req From ftp.request.command; - Extract server_addr From ftp.passive.ip; - Extract server_port From ftp.passive.port; - - LastPdu; + Extract ftp_addr From ip.addr; + Extract ftp_port From tcp.port; + Extract ftp_resp From ftp.response.code; + Extract ftp_req From ftp.request.command; + Extract server_addr From ftp.passive.ip; + Extract server_port From ftp.passive.port; + + LastPdu true; }; Pdu ftp_data_pdu Proto ftp-data Transport tcp/ip{ - Extract server_addr From ip.src; - Extract server_port From tcp.srcport; + Extract server_addr From ip.src; + Extract server_port From tcp.srcport; }; -Gop ftp_data On ftp_data_pdu (server_addr, server_port) { - Start (server_addr); +Gop ftp_data On ftp_data_pdu Match (server_addr, server_port) { + Start (server_addr); }; -Gop ftp_ctl On ftp_pdu (ftp_addr, ftp_addr, ftp_port, ftp_port) { - Start (ftp_resp=220); - Stop (ftp_resp=221); - Extra (server_addr, server_port); +Gop ftp_ctl On ftp_pdu Match (ftp_addr, ftp_addr, ftp_port, ftp_port) { + Start (ftp_resp=220); + Stop (ftp_resp=221); + Extra (server_addr, server_port); }; Gog ftp_ses { - Member ftp_ctl (ftp_addr, ftp_addr, ftp_port, ftp_port); - Member ftp_data (server_addr, server_port); + Member ftp_ctl (ftp_addr, ftp_addr, ftp_port, ftp_port); + Member ftp_data (server_addr, server_port); }; Done; ---- Note: not having anything to distinguish between ftp-data packets makes this -config to create one Gop for every ftp-data packet instead of each transfer. -Pre-started Gops would avoid this. +config to create one GOP for every ftp-data packet instead of each transfer. +Pre-started GOPs would avoid this. ==== using RADIUS to filter SMTP traffic of a specific user @@ -1285,46 +1339,46 @@ Done; ---- Filtering the capture file with *mate.user_mail.username == "theuser"* will -filter the radius packets and smtp traffic for _"theuser"_. +filter the RADIUS packets and SMTP traffic for _"theuser"_. ==== H323 Calls -This configuration will create a Gog out of every call. +This configuration will create a GOG out of every call. ---- Pdu q931 Proto q931 Transport ip { - Extract addr From ip.addr; - Extract call_ref From q931.call_ref; - Extract q931_msg From q931.message_type; - Extract calling From q931.calling_party_number.digits; - Extract called From q931.called_party_number.digits; - Extract guid From h225.guid; - Extract q931_cause From q931.cause_value; + Extract addr From ip.addr; + Extract call_ref From q931.call_ref; + Extract q931_msg From q931.message_type; + Extract calling From q931.calling_party_number.digits; + Extract called From q931.called_party_number.digits; + Extract guid From h225.guid; + Extract q931_cause From q931.cause_value; }; Gop q931_leg On q931 Match (addr, addr, call_ref) { - Start (q931_msg=5); - Stop (q931_msg=90); - Extra (calling, called, guid, q931_cause); + Start (q931_msg=5); + Stop (q931_msg=90); + Extra (calling, called, guid, q931_cause); }; Pdu ras Proto h225.RasMessage Transport ip { - Extract addr From ip.addr; - Extract ras_sn From h225.requestSeqNum; - Extract ras_msg From h225.RasMessage; - Extract guid From h225.guid; + Extract addr From ip.addr; + Extract ras_sn From h225.requestSeqNum; + Extract ras_msg From h225.RasMessage; + Extract guid From h225.guid; }; Gop ras_req On ras Match (addr, addr, ras_sn) { - Start (ras_msg {0|3|6|9|12|15|18|21|26|30} ); - Stop (ras_msg {1|2|4|5|7|8|10|11|13|14|16|17|19|20|22|24|27|28|29|31}); - Extra (guid); + Start (ras_msg {0|3|6|9|12|15|18|21|26|30} ); + Stop (ras_msg {1|2|4|5|7|8|10|11|13|14|16|17|19|20|22|24|27|28|29|31}); + Extra (guid); }; Gog call { - Member ras_req (guid); - Member q931_leg (guid); - Extra (called,calling,q931_cause); + Member ras_req (guid); + Member q931_leg (guid); + Extra (called,calling,q931_cause); }; Done; @@ -1339,10 +1393,10 @@ with this we can: ==== MMS With this example, all the components of an MMS send or receive will be tied -into a single Gog. Note that this example uses the _Payload_ clause because +into a single GOG. Note that this example uses the _Payload_ clause because MMS delivery uses MMSE over either HTTP or WSP. As it is not possible to relate the retrieve request to a response by the means of MMSE only (the request is -just an HTTP GET without any MMSE), a Gop is made of HTTP Pdus but MMSE data +just an HTTP GET without any MMSE), a GOP is made of HTTP PDUs but MMSE data need to be extracted from the bodies. ---- @@ -1351,61 +1405,61 @@ need to be extracted from the bodies. ## tested against any capture file due to lack of the latter. Transform rm_client_from_http_resp1 { - Match (http_rq); - Match Every (addr) Insert (not_rq); + Match (http_rq); + Match Every (addr) Insert (not_rq); }; Transform rm_client_from_http_resp2 { - Match (not_rq,ue) Replace (); + Match (not_rq,ue) Replace (); }; Pdu mmse_over_http_pdu Proto http Transport tcp/ip { - Payload mmse; - Extract addr From ip.addr; - Extract port From tcp.port; - Extract http_rq From http.request; - Extract content From http.content_type; - Extract resp From http.response.code; - Extract method From http.request.method; - Extract host From http.host; - Extract content From http.content_type; - Extract trx From mmse.transaction_id; - Extract msg_type From mmse.message_type; - Extract notify_status From mmse.status; - Extract send_status From mmse.response_status; - Transform rm_client_from_http_resp1, rm_client_from_http_resp2; + Payload mmse; + Extract addr From ip.addr; + Extract port From tcp.port; + Extract http_rq From http.request; + Extract content From http.content_type; + Extract resp From http.response.code; + Extract method From http.request.method; + Extract host From http.host; + Extract content From http.content_type; + Extract trx From mmse.transaction_id; + Extract msg_type From mmse.message_type; + Extract notify_status From mmse.status; + Extract send_status From mmse.response_status; + Transform rm_client_from_http_resp1, rm_client_from_http_resp2; }; Gop mmse_over_http On mmse_over_http_pdu Match (addr, addr, port, port) { - Start (http_rq); - Stop (http_rs); - Extra (host, ue, resp, notify_status, send_status, trx); + Start (http_rq); + Stop (http_rs); + Extra (host, ue, resp, notify_status, send_status, trx); }; Transform mms_start { - Match Loose() Insert (mms_start); + Match Loose() Insert (mms_start); }; Pdu mmse_over_wsp_pdu Proto wsp Transport ip { - Payload mmse; - Extract trx From mmse.transaction_id; - Extract msg_type From mmse.message_type; - Extract notify_status From mmse.status; - Extract send_status From mmse.response_status; - Transform mms_start; + Payload mmse; + Extract trx From mmse.transaction_id; + Extract msg_type From mmse.message_type; + Extract notify_status From mmse.status; + Extract send_status From mmse.response_status; + Transform mms_start; }; Gop mmse_over_wsp On mmse_over_wsp_pdu Match (trx) { - Start (mms_start); - Stop (never); - Extra (ue, notify_status, send_status); + Start (mms_start); + Stop (never); + Extra (ue, notify_status, send_status); }; Gog mms { - Member mmse_over_http (trx); - Member mmse_over_wsp (trx); - Extra (ue, notify_status, send_status, resp, host, trx); - Expiration 60.0; + Member mmse_over_http (trx); + Member mmse_over_wsp (trx); + Extra (ue, notify_status, send_status, resp, host, trx); + Expiration 60.0; }; ---- @@ -1413,13 +1467,13 @@ Gog mms { === MATE's configuration library -The MATE library (will) contains GoP definitions for several protocols. Library +The MATE library (will) contains GOP definitions for several protocols. Library protocols are included in your MATE config using: +_Action=Include; Lib=proto_name;_+. For Every protocol with a library entry, we'll find defined what from the PDU is -needed to create a GoP for that protocol, eventually any criteria and the very -essential GoP definition (i.e., __GopDef__, _GopStart_ and _GopStop_). +needed to create a GOP for that protocol, eventually any criteria and the very +essential GOP definition (i.e., __Gop__, _Start_ and _Stop_). [NOTE] ==== @@ -1432,20 +1486,36 @@ these in the new format. ===== TCP -It will create a GoP for every TCP session, If it is used it should be the last +It will create a GOP for every TCP session. If it is used it should be the last one in the list. And every other proto on top of TCP should be declared with -_Stop=TRUE;_ so the a TCP PDU is not created where we got already one going on. +_LastPdu=TRUE;_ so that a TCP PDU is not created where another pdu type exists. ---- - Action=PduDef; Name=tcp_pdu; Proto=tcp; Transport=ip; addr=ip.addr; port=tcp.port; tcp_start=tcp.flags.syn; tcp_stop=tcp.flags.fin; tcp_stop=tcp.flags.reset; - Action=GopDef; Name=tcp_session; On=tcp_pdu; addr; addr; port; port; - Action=GopStart; For=tcp_session; tcp_start=1; - Action=GopStop; For=tcp_session; tcp_stop=1; +Transform add_tcp_stop { + Match (tcp_flags_reset="True") Insert (tcp_stop="True"); + Match (tcp_flags_fin="True") Insert (tcp_stop="True"); +}; + +Pdu tcp_pdu Proto tcp Transport ip { + Extract addr From ip.addr; + Extract port From tcp.port; + Extract tcp_start From tcp.flags.syn; + Extract tcp_flags_reset From tcp.flags.reset; + Extract tcp_flags_fin From tcp.flags.fin; + Transform add_tcp_stop; +}; + +Gop tcp_ses On tcp_pdu Match (addr, addr, port, port) { + Start (tcp_start="True"); + Stop (tcp_stop="True"); +}; + +Done; ---- ===== DNS -will create a GoP containing every request and its response (eventually +will create a GOP containing every request and its response (eventually retransmissions too). ---- @@ -1458,7 +1528,7 @@ Action=GopStop; For=dns_req; dns_rsp=1; ===== RADIUS -A Gop for every transaction. +A GOP for every transaction. ---- Action=PduDef; Name=radius_pdu; Proto=radius; Transport=udp/ip; addr=ip.addr; port=udp.port; radius_id=radius.id; radius_code=radius.code; @@ -1482,7 +1552,7 @@ Action=GopExtra; For=rtsp_ses; rtsp_ses; rtsp_url; ==== VoIP/Telephony -Most protocol definitions here will create one Gop for every Call Leg unless +Most protocol definitions here will create one GOP for every Call Leg unless stated. ===== ISUP @@ -1529,7 +1599,7 @@ Action=GopStop; For=sip; sip_method=BYE; ===== MEGACO -Will create a Gop out of every transaction. +Will create a GOP out of every transaction. To "tie" them to your call's GoG use: _Action=GogKey; Name=your_call; On=mgc_tr; addr!mgc_addr; megaco_ctx;_ @@ -1547,7 +1617,7 @@ Action=GopExtra; For=mgc_tr; term^DS1; megaco_ctx!Choose one; === MATE's reference manual -==== Attribute Value Pairs +==== Attribute Value Pairs (AVP) MATE uses AVPs for almost everything: to keep the data it has extracted from the frames' trees as well as to keep the elements of the configuration. @@ -1555,11 +1625,11 @@ frames' trees as well as to keep the elements of the configuration. These "pairs" (actually tuples) are made of a name, a value and, in case of configuration AVPs, an operator. Names and values are strings. AVPs with operators other than '=' are used only in the configuration and are used for -matching AVPs of Pdus, GoPs and GoGs in the analysis phase. +matching AVPs of PDUs, GOPs and GOGs in the analysis phase. ===== Name -The name is a string used to refer to a class of AVPs. Two attributes won't +The name is a string used to refer to a type of AVP. Two attributes won't match unless their names are identical. Capitalized names are reserved for keywords (you can use them for your elements if you want but I think it's not the case). MATE attribute names can be used in Wireshark's display filters the @@ -1573,7 +1643,7 @@ AVPs) or by MATE while extracting interesting fields from a dissection tree and/or manipulating them later. The values extracted from fields use the same representation as they do in filter strings. -===== Operators +==== AVP Operators (=,!,{},^,$,~,<,>,?) Currently only match operators are defined (there are plans to (re)add transform attributes but some internal issues have to be solved before that). The match @@ -1604,7 +1674,7 @@ higher than the configuration value string. what the value string is. [#Equal] -====== Equal AVP Operator +===== Equal AVP Operator (=) This operator tests whether the values of the operator and the operand AVP are equal. @@ -1614,7 +1684,7 @@ attrib=aaa *matches* attrib=aaa + attrib=aaa *does not match* attrib=bbb [#NotEqual] -====== Not equal AVP operator +===== Not equal AVP operator (!) This operator matches if the value strings of two AVPs are not equal. @@ -1623,7 +1693,7 @@ attrib=aaa matches attrib!bbb + attrib=aaa does not match attrib!aaa [#OneOf] -====== "One of" AVP operator +===== "One of" AVP operator ({}) The "one of" operator matches if the data AVP value is equal to one of the values listed in the "one of" AVP. @@ -1634,7 +1704,7 @@ attrib=2 matches attrib{1|2|3} + attrib=4 does not match attrib{1|2|3} [#StartsWith] -====== "Starts with" AVP operator +===== "Starts with" AVP operator (^) The "starts with" operator matches if the first characters of the data AVP value are identical to the configuration AVP value. @@ -1647,7 +1717,7 @@ attrib=abcd does not match attrib^bcd + attrib=abc does not match attrib^abcd + [#EndsWith] -====== "Ends with" operator +===== "Ends with" operator ($) The ends with operator will match if the last bytes of the data AVP value are equal to the configuration AVP value. @@ -1658,7 +1728,7 @@ attrib=yz does not match attrib$xyz + attrib=abc...wxyz does not match attrib$abc [#Contains] -====== Contains operator +===== Contains operator (~) The "contains" operator will match if the data AVP value contains a string identical to the configuration AVP value. @@ -1670,7 +1740,7 @@ attrib=abcde matches attrib~cde + attrib=abcde does not match attrib~xyz [#LowerThan] -====== "Lower than" operator +===== "Lower than" operator (<) The "lower than" operator will match if the data AVP value is semantically lower than the configuration AVP value. @@ -1687,7 +1757,7 @@ BUGS It should check whether the values are numbers and compare them numerically [#HigherThan] -====== "Higher than" operator +===== "Higher than" operator (>) The "higher than" operator will match if the data AVP value is semantically higher than the configuration AVP value. @@ -1705,7 +1775,7 @@ BUGS It should check whether the values are numbers and compare them numerically [#Exists] -====== Exists operator +===== Exists operator (?) The exists operator will always match as far as the two operands have the same name. @@ -1716,14 +1786,15 @@ attrib=abc matches attrib? + attrib=abc matches attrib (this is just an alternative notation of the previous example) + obviously attrib=abc does not match other_attrib? + -==== Attribute/Value Pair List (AVPL) -Pdus, GoPs and GoGs use an AVPL to contain the tracing information. An AVPL is +==== Attribute Value Pair List (AVPL) +PDUs, GOPs and GOGs use an AVPL to contain the tracing information. An AVPL is an unsorted set of <<AVP,AVPs>> that can be matched against other AVPLs. -===== Operations between AVPLs +[#Match] +==== Operations between AVPLs (Match) There are three types of match operations that can be performed between AVPLs. -The Pdu's/GoP's/GoG's AVPL will be always one of the operands; the AVPL operator +The PDU's/GOP's/GOG's AVPL will be always one of the operands; the AVPL operator (match type) and the second operand AVPL will always come from the <<Config,configuration>>. Note that a diverse AVP match operator may be specified for each AVP in the @@ -1733,7 +1804,7 @@ An AVPL match operation returns a result AVPL. In <<Transform,Transform>>s, the result AVPL may be replaced by another AVPL. The replacement means that the existing data AVPs are dropped and the replacement AVPL from the <<Config,configuration>> is <<Merge,Merged>> to the data AVPL of the -Pdu/GoP/GoG. +PDU/GOP/GOG. * <<Loose,Loose Match>>: Will match if at least one of the AVPs of the two operand AVPLs match. If it matches, it returns a result AVPL containing all AVPs @@ -1748,15 +1819,15 @@ configuration AVPL has at least one match in the data AVPL. If it matches, it returns a result AVPL containing those AVPs from the data AVPL that matched. [#Loose] -====== Loose Match +===== Loose Match A loose match between AVPLs succeeds if at least one of the data AVPs matches at least one of the configuration AVPs. Its result AVPL contains all the data AVPs that matched. -Loose matches are used in Extra operations against the <<Pdu,Pdu>>'s AVPL to -merge the result into <<Gop,Gop>>'s AVPL, and against <<Gop,Gop>>'s AVPL to -merge the result into <<Gog,Gog>>'s AVPL. They may also be used in +Loose matches are used in Extra operations against the <<Pdu,PDU>>'s AVPL to +merge the result into <<Gop,GOP>>'s AVPL, and against <<Gop,GOP>>'s AVPL to +merge the result into <<Gog,GOG>>'s AVPL. They may also be used in <<Criteria,Criteria>> and <<Transform,Transform>>s. [NOTE] @@ -1775,7 +1846,7 @@ Loose Match Examples (attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Loose (attr_a=xxx; attr_c=ccc) ==> No Match! [#Every] -====== Every Match +===== Every Match An "every" match between AVPLs succeeds if none of the configuration's AVPs that have a counterpart in the data AVPL fails to match. Its result AVPL contains all @@ -1800,13 +1871,13 @@ https://gitlab.com/wireshark/wireshark/-/issues/12184[issue 12184]. (attr_a=aaa; attr_b=bbb; attr_c=xxx) Match Every (attr_a=xxx, attr_c=ccc) ==> No Match! [#Strict] -====== Strict Match +===== Strict Match A Strict match between AVPLs succeeds if and only if every AVP in the configuration AVPL has at least one counterpart in the data AVPL and none of the AVP matches fails. The result AVPL contains all the data AVPs that matched. -These are used between Gop keys (key AVPLs) and Pdu AVPLs. They may also be used +These are used between GOP keys (key AVPLs) and PDU AVPLs. They may also be used in <<Criteria,Criteria>> and <<Transform,Transform>>s. Examples @@ -1820,118 +1891,94 @@ Examples (attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Strict (attr_a?, attr_c?, attr_d?) ==> No Match! [#Merge] -====== AVPL Merge +==== AVPL Merge An AVPL may be merged into another one. That would add to the latter every AVP from the former that does not already exist there. This operation is done -* between the result of a key match and the Gop's or Gog's AVPL, -* between the result of an Extra match and the Gop's or Gog's AVPL, -* between the result of a <<Transform,Transform>> match and Pdu's/Gop's AVPL. If +* between the result of a key match and the GOP's or GOG's AVPL, +* between the result of an Extra match and the GOP's or GOG's AVPL, +* between the result of a <<Transform,Transform>> match and PDU's/GOP's AVPL. If the operation specified by the Match clause is Replace, the result AVPL of the match is removed from the item's AVPL before the modify_avpl is merged into it. Examples -(attr_a=aaa, attr_b=bbb) Merge (attr_a=aaa, attr_c=xxx) former becomes (attr_a=aaa, attr_b=bbb, attr_c=xxx) +(attr_a=aaa, attr_b=bbb) "merge" (attr_a=aaa, attr_c=xxx) former becomes (attr_a=aaa, attr_b=bbb, attr_c=xxx) -(attr_a=aaa, attr_b=bbb) Merge (attr_a=aaa, attr_a=xxx) former becomes (attr_a=aaa, attr_a=xxx, attr_b=bbb) +Can't have multiple "attr_a" with same value "aaa" -(attr_a=aaa, attr_b=bbb) Merge (attr_c=xxx, attr_d=ddd) former becomes (attr_a=aaa, attr_b=bbb, attr_c=xxx, attr_d=ddd) +(attr_a=aaa, attr_b=bbb) "merge" (attr_a=aaa, attr_a=xxx) former becomes (attr_a=aaa, attr_a=xxx, attr_b=bbb) -[#Transform] -====== Transforms +Multiple "attr_a" with different values "aaa" and "xxx" -A Transform is a sequence of Match rules optionally followed by an instruction -how to modify the match result using an additional AVPL. Such modification may -be an Insert (merge) or a Replace. The syntax is as follows: +(attr_a=aaa, attr_b=bbb) "merge" (attr_c=xxx, attr_d=ddd) former becomes (attr_a=aaa, attr_b=bbb, attr_c=xxx, attr_d=ddd) ----- -Transform name { - Match [Strict|Every|Loose] match_avpl [[Insert|Replace] modify_avpl] ; // may occur multiple times, at least once -}; ----- - -For examples of Transforms, check the <<ChMateManual,Manual>> page. - -TODO: migrate the examples here? - -The list of Match rules inside a Transform is processed top to bottom; -the processing ends as soon as either a Match rule succeeds or all have been -tried in vain. - -Transforms can be used as helpers to manipulate an item's AVPL before the item -is processed further. An item declaration may contain a Transform clause -indicating a list of previously declared Transforms. Regardless whether the -individual transforms succeed or fail, the list is always executed completely -and in the order given, i.e., left to right. - -In MATE configuration file, a Transform must be declared before declaring any -item which uses it. +All AVP names are unique so resulting AVPL contains all AVPs from both AVPLs [#Config] -=== Configuration AVPLs +=== Configuration Reference (mate.config) [#Pdu] -==== Pdsu's configuration actions +==== PDU declaration block The following configuration AVPLs deal with PDU creation and data extraction. -===== Pdu declaration block header +===== _Pdu_ declaration block header In each frame of the capture, MATE will look for source _proto_name_'s PDUs in the order in which the declarations appear in its configuration and will create -Pdus of every type it can from that frame, unless specifically instructed that -some Pdu type is the last one to be looked for in the frame. If told so for a -given type, MATE will extract all Pdus of that type and the previously declared +PDUs of every type it can from that frame, unless specifically instructed that +some PDU type is the last one to be looked for in the frame. If told so for a +given type, MATE will extract all PDUs of that type and the previously declared types it finds in the frame but not those declared later. -The complete declaration of a Pdu looks as below; the mandatory order of the +The complete declaration of a _Pdu_ looks as below; the mandatory order of the diverse clauses is as shown. ---- - Pdu name Proto proto_name Transport proto1[/proto2/proto3[/...]]] { - Payload proto; //optional, no default value - Extract attribute From proto.field ; //may occur multiple times, at least once - Transform (transform1[, transform2[, ...]]); //optional - Criteria [{Accept|Reject}] [{Strict|Every|Loose} match_avpl]; - DropUnassigned {true|false}; //optional, default=false - DiscardPduData {true|false}; //optional, default=false - LastExtracted {true|false}; //optional, default=false - }; +Pdu name Proto proto_name Transport {proto1[/proto2/proto3[/...]|mate}; { + Payload proto; //optional, no default value + Extract attribute From proto.field ; //may occur multiple times, at least once + Transform transform1[, transform2[, ...]]; //optional + Criteria {Accept|Reject} {Strict|Every|Loose} match_avpl; //optional + DropUnassigned {TRUE|FALSE}; //optional, default=FALSE + DiscardPduData {TRUE|FALSE}; //optional, default=FALSE + LastPdu {TRUE|FALSE}; //optional, default=FALSE +}; ---- -===== Pdu name +====== Pdu name -The _name_ is a mandatory attribute of a Pdu declaration. It is chosen +The _name_ is a mandatory attribute of a _Pdu_ declaration. It is chosen arbitrarily, except that each _name_ may only be used once in MATE's configuration, regardless the class of an item it is used for. The _name_ is -used to distinguish between different types of Pdus, Gops, and Gogs. The _name_ -is also used as part of the filterable fields' names related to this type of Pdu +used to distinguish between different types of PDUs, GOPs, and GOGs. The _name_ +is also used as part of the filterable fields' names related to this type of PDU which MATE creates. -However, several Pdu declarations may share the same _name_. In such case, all +However, several _Pdu_ declarations may share the same _name_. In such case, all of them are created from each source PDU matching their _Proto_, _Transport_, and _Payload_ clauses, while the bodies of their declarations may be totally different from each other. Together with the _Accept_ (or _Reject_) clauses, -this feature is useful when it is necessary to build the Pdu's AVPL from +this feature is useful when it is necessary to build the PDU's AVPL from different sets of source fields depending on contents (or mere presence) of other source fields. ====== Proto and Transport clauses Every instance of the protocol _proto_name_ PDU in a frame will generate one -Pdu with the AVPs extracted from fields that are in the _proto_name_'s range +PDU with the AVPs extracted from fields that are in the _proto_name_'s range and/or the ranges of underlying protocols specified by the _Transport_ list. -It is a mandatory attribute of a Pdu declaration. The _proto_name_ is the name +It is a mandatory attribute of a _Pdu_ declaration. The _proto_name_ is the name of the protocol as used in Wireshark display filter. -The Pdu's _Proto_, and its _Transport_ list of protocols separated by / tell -MATE which fields of a frame can get into the Pdu's AVPL. In order that MATE +The PDU's _Proto_, and its _Transport_ list of protocols separated by / tell +MATE which fields of a frame can get into the PDU's AVPL. In order that MATE would extract an attribute from a frame's protocol tree, the area representing the field in the hex display of the frame must be within the area of either the -_Proto_ or its relative _Transport_ s. _Transport_ s are chosen moving backwards +_Proto_ or its relative _Transport_++s++. _Transport_++s++ are chosen moving backwards from the protocol area, in the order they are given. _Proto http Transport tcp/ip_ does what you'd expect it to - it selects the @@ -1943,24 +1990,24 @@ too doesn't work so far. Once we've selected the _Proto_ and _Transport_ ranges, MATE will fetch those protocol fields belonging to them whose extraction is declared using the -_Extract_ clauses for the Pdu type. The _Transport_ list is also mandatory, +_Extract_ clauses for the PDU type. The _Transport_ list is also mandatory, if you actually don't want to use any transport protocol, use _Transport mate_. (This didn't work until 0.10.9). -====== Payload clause +===== Payload clause -Other than the Pdu's _Proto_ and its _Transport_ protocols, there is also a +Other than the PDU's _Proto_ and its _Transport_ protocols, there is also a _Payload_ attribute to tell MATE from which ranges of _Proto_'s payload to -extract fields of a frame into the Pdu. In order to extract an attribute from a +extract fields of a frame into the PDU. In order to extract an attribute from a frame's tree the highlighted area of the field in the hex display must be within -the area of the _Proto_'s relative payload(s). _Payload_ s are chosen moving +the area of the _Proto_'s relative payload(s). _Payload_++s++ are chosen moving forward from the protocol area, in the order they are given. _Proto http Transport tcp/ip Payload mmse_ will select the first mmse range after the current http range. Once we've selected the _Payload_ ranges, MATE will fetch those protocol fields belonging to them whose extraction is declared -using the _Extract_ clauses for the Pdu type. +using the _Extract_ clauses for the PDU type. -====== Extract clause +===== Extract clause Each _Extract_ clause tells MATE which protocol field value to extract as an AVP value and what string to use as the AVP name. The protocol fields are referred @@ -1968,294 +2015,330 @@ to using the names used in Wireshark display filters. If there is more than one such protocol field in the frame, each instance that fulfills the criteria stated above is extracted into its own AVP. The AVP names may be chosen arbitrarily, but to be able to match values originally coming from different -Pdus (e.g., hostname from DNS query and a hostname from HTTP GET request) later +PDUs (e.g., hostname from DNS query and a hostname from HTTP GET request) later in the analysis, identical AVP names must be assigned to them and the dissectors must provide the field values in identical format (which is not always the case). -====== Transform clause +===== Transform clause -The _Transform_ clause specifies a list of previously declared _Transform_ s to -be performed on the Pdu's AVPL after all protocol fields have been extracted to +The _Transform_ clause specifies a list of previously declared _Transform_++s++ to +be performed on the PDU's AVPL after all protocol fields have been extracted to it. The list is always executed completely, left to right. On the contrary, the list of Match clauses inside each individual _Transform_ is executed only until the first match succeeds. [#Criteria] -====== Criteria clause +===== Criteria clause -This clause tells MATE whether to use the Pdu for analysis. It specifies a match -AVPL, an AVPL match type (_Strict_, _Every_, or _Loose_) and the action to be +This clause tells MATE whether to use the PDU for analysis. It specifies a match +AVPL, an AVPL <<Match,Match type>> (_Strict_, _Every_, or _Loose_) and the action to be performed (_Accept_ or _Reject_) if the match succeeds. Once every attribute has been extracted and eventual transform list has been executed, and if the -_Criteria_ clause is present, the Pdu's AVPL is matched against the match AVPL; -if the match succeeds, the action specified is executed, i.e., the Pdu is +_Criteria_ clause is present, the PDU's AVPL is matched against the match AVPL; +if the match succeeds, the action specified is executed, i.e., the PDU is accepted or rejected. The default behaviors used if the respective keywords are omitted are _Strict_ and _Accept_. Accordingly, if the clause is omitted, all -Pdus are accepted. +PDUs are accepted. -====== DropUnassigned clause +===== DropUnassigned clause -If set to _TRUE_, MATE will destroy the Pdu if it cannot assign it to a Gop. +If set to _TRUE_, MATE will destroy the PDU if it cannot assign it to a GOP. If set to _FALSE_ (the default if not given), MATE will keep them. -====== DiscardPduData clause +===== DiscardPduData clause -If set to _TRUE_, MATE will delete the Pdu's AVPL once it has analyzed it and -eventually extracted some AVPs from it into the Gop's AVPL. This is useful to +If set to _TRUE_, MATE will delete the PDU's AVPL once it has analyzed it and +eventually extracted some AVPs from it into the GOP's AVPL. This is useful to save memory (of which MATE uses a lot). If set to _FALSE_ (the default if not -given), MATE will keep the Pdu attributes. +given), MATE will keep the PDU attributes. -====== LastExtracted clause +===== LastPdu clause If set to _FALSE_ (the default if not given), MATE will continue to look for -Pdus of other types in the frame. If set to _TRUE_, it will not try to create -Pdus of other types from the current frame, yet it will continue to try for the +PDUs of other types in the frame. If set to _TRUE_, it will not try to create +PDUs of other types from the current frame, yet it will continue to try for the current type. [#Gop] -===== Gop's configuration actions +==== GOP declaration block -====== Gop declaration block header +===== _Gop_ declaration block header -Declares a Gop type and its prematch candidate key. +Declares a Gop type and its candidate key. ---- - Gop name On pduname Match key { - Start match_avpl; // optional - Stop match_avpl; // optional - Extra match_avpl; // optional - Transform transform_list; // optional - Expiration time; // optional - IdleTimeout time; // optional - Lifetime time; // optional - DropUnassigned [TRUE|FALSE]; //optional - ShowTree [NoTree|PduTree|FrameTree|BasicTree]; //optional - ShowTimes [TRUE|FALSE]; //optional, default TRUE - }; +Gop name On pduname Match key { + Start match_avpl; // optional + Stop match_avpl; // optional + Extra match_avpl; // optional + Transform transform_list; // optional + Expiration time; // optional + IdleTimeout time; // optional + Lifetime time; // optional + DropUnassigned [TRUE|FALSE]; //optional + ShowTree [NoTree|PduTree|FrameTree|BasicTree]; //optional + ShowTimes [TRUE|FALSE]; //optional, default TRUE +}; ---- ====== Gop name -The _name_ is a mandatory attribute of a Gop declaration. It is chosen +The _name_ is a mandatory attribute of a _Gop_ declaration. It is chosen arbitrarily, except that each _name_ may only be used once in MATE's configuration, regardless the class of an item it is used for. The _name_ is -used to distinguish between different types of Pdus, Gops, and Gogs. The _name_ +used to distinguish between different types of PDUs, GOPs, and GOGs. The _name_ is also used as part of the filterable fields' names related to this type of -Gop which MATE creates. +GOP which MATE creates. ====== On clause -The _name_ of Pdus which this type of Gop is supposed to be groupping. It is +The _name_ of PDUs which this type of GOP is supposed to be grouping. It is mandatory. ====== Match clause -Defines what AVPs form up the _key_ part of the Gop's AVPL (the Gop's _key_ AVPL -or simply the Gop's _key_). All Pdus matching the _key_ AVPL of an active Gop -are assigned to that Gop; a Pdu which contains the AVPs whose attribute names -are listed in the Gop's _key_ AVPL, but they do not strictly match any active -Gop's _key_ AVPL, will create a new Gop (unless a _Start_ clause is given). -When a Gop is created, the elements of its key AVPL are copied from the creating -Pdu. +Defines what AVPs form up the _key_ part of the GOP's AVPL (the GOP's _key_ AVPL +or simply the GOP's _key_). All PDUs matching the _key_ AVPL of an active GOP +are assigned to that GOP; a PDU which contains the AVPs whose attribute names +are listed in the GOP's _key_ AVPL, but they do not strictly match any active +GOP's _key_ AVPL, will create a new GOP (unless a _Start_ clause is given). +When a GOP is created, the elements of its key AVPL are copied from the creating +PDU. -====== Start clause +===== Start clause -If given, it tells MATE what match_avpl must a Pdu's AVPL match, in addition to -matching the Gop's _key_, in order to start a Gop. If not given, any Pdu whose -AVPL matches the Gop's _key_ AVPL will act as a start for a Gop. The Pdu's AVPs -matching the match_avpl are not automatically copied into the Gop's AVPL. +If given, it tells MATE what match_avpl must a PDU's AVPL match, in addition to +matching the GOP's _key_, in order to start a GOP. If not given, any PDU whose +AVPL matches the GOP's _key_ AVPL will act as a start for a GOP. The PDU's AVPs +matching the match_avpl are not automatically copied into the GOP's AVPL. -====== Stop clause +===== Stop clause -If given, it tells MATE what match_avpl must a Pdu's AVPL match, in addition to -matching the Gop's key, in order to stop a Gop. If omitted, the Gop is -"auto-stopped" - that is, the Gop is marked as stopped as soon as it is created. -The Pdu's AVPs matching the match_avpl are not automatically copied into the -Gop's AVPL. +If given, it tells MATE what match_avpl must a PDU's AVPL match, in addition to +matching the GOP's _key_, in order to stop a GOP. If omitted, the GOP is +"auto-stopped" - that is, the GOP is marked as stopped as soon as it is created. +The PDU's AVPs matching the match_avpl are not automatically copied into the +GOP's AVPL. -====== Extra clause +===== Extra clause -If given, tells MATE which AVPs from the Pdu's AVPL are to be copied into the -Gop's AVPL in addition to the Gop's key. +If given, tells MATE which AVPs from the PDU's AVPL are to be copied into the +GOP's AVPL in addition to the GOP's key. -====== Transform clause +===== Transform clause -The _Transform_ clause specifies a list of previously declared _Transform_ s to -be performed on the Gop's AVPL after the AVPs from each new Pdu, specified by -the key AVPL and the _Extra_ clause's match_avpl, have been merged into it. +The _Transform_ clause specifies a list of previously declared _Transform_++s++ to +be performed on the GOP's AVPL after the AVPs from each new PDU, specified by +the _key_ AVPL and the _Extra_ clause's match_avpl, have been merged into it. The list is always executed completely, left to right. On the contrary, the list of _Match_ clauses inside each individual _Transform_ is executed only until the first match succeeds. -====== Expiration clause +===== Expiration clause -A (floating) number of seconds after a Gop is _Stop_ ped during which further -Pdus matching the _Stop_ ped Gop's key but not the _Start_ condition will still -be assigned to that Gop. The default value of zero has an actual meaning of -infinity, as it disables this timer, so all Pdus matching the _Stop_ ped Gop's -key will be assigned to that Gop unless they match the _Start_ condition. +A (floating) number of seconds after a GOP is _Stop_ ped during which further +PDUs matching the _Stop_ ped GOP's key but not the _Start_ condition will still +be assigned to that GOP. The default value of zero has an actual meaning of +infinity, as it disables this timer, so all PDUs matching the _Stop_ ped GOP's +key will be assigned to that GOP unless they match the _Start_ condition. -====== IdleTimeout clause +===== IdleTimeout clause -A (floating) number of seconds elapsed from the last Pdu assigned to the Gop -after which the Gop will be considered released. The default value of zero has -an actual meaning of infinity, as it disables this timer, so the Gop won't be -released even if no Pdus arrive - unless the _Lifetime_ timer expires. +A (floating) number of seconds elapsed from the last PDU assigned to the GOP +after which the GOP will be considered released. The default value of zero has +an actual meaning of infinity, as it disables this timer, so the GOP won't be +released even if no PDUs arrive - unless the _Lifetime_ timer expires. -====== Lifetime clause +===== Lifetime clause -A (floating) of seconds after the Gop _Start_ after which the Gop will be +A (floating) of seconds after the GOP _Start_ after which the GOP will be considered released regardless anything else. The default value of zero has an actual meaning of infinity. -====== DropUnassigned clause +===== DropUnassigned clause -Whether or not a Gop that has not being assigned to any Gog should be discarded. -If _TRUE_, the Gop is discarded right after creation. If _FALSE_, the default, -the unassigned Gop is kept. Setting it to _TRUE_ helps save memory and speed up +Whether or not a GOP that has not being assigned to any GOG should be discarded. +If _TRUE_, the GOP is discarded right after creation. If _FALSE_, the default, +the unassigned GOP is kept. Setting it to _TRUE_ helps save memory and speed up filtering. -====== TreeMode clause +===== TreeMode clause -Controls the display of Pdus subtree of the Gop: +Controls the display of PDUs subtree of the GOP: * _NoTree_: completely suppresses showing the tree -* _PduTree_: the tree is shown and shows the Pdus by Pdu Id -* _FrameTree_: the tree is shown and shows the Pdus by the frame number in which +* _PduTree_: the tree is shown and shows the PDUs by PDU Id +* _FrameTree_: the tree is shown and shows the PDUs by the frame number in which they are * _BasicTree_: needs investigation -====== ShowTimes clause +===== ShowTimes clause -Whether or not to show the times subtree of the Gop. If _TRUE_, the default, -the subtree with the timers is added to the Gop's tree. If _FALSE_, the subtree +Whether or not to show the times subtree of the GOP. If _TRUE_, the default, +the subtree with the timers is added to the GOP's tree. If _FALSE_, the subtree is suppressed. [#Gog] -===== Gog's configuration actions +==== GOG declaration block -====== Gop declaration block header +===== _Gog_ declaration block header -Declares a Gog type and its prematch candidate key. +Declares a Gog type and its candidate key. ---- - Gog name { - Member gopname (key); // mandatory, at least one - Extra match_avpl; // optional - Transform transform_list; // optional - Expiration time; // optional, default 2.0 - GopTree [NoTree|PduTree|FrameTree|BasicTree]; // optional - ShowTimes [TRUE|FALSE]; // optional, default TRUE - }; +Gog name { + Member gopname (key); // mandatory, at least one + Extra match_avpl; // optional + Transform transform_list; // optional + Expiration time; // optional, default 2.0 + GopTree [NoTree|PduTree|FrameTree|BasicTree]; // optional + ShowTimes [TRUE|FALSE]; // optional, default TRUE +}; ---- -====== Gop name +====== Gog name -The _name_ is a mandatory attribute of a Gog declaration. It is chosen +The _name_ is a mandatory attribute of a _Gog_ declaration. It is chosen arbitrarily, except that each _name_ may only be used once in MATE's configuration, regardless the class of an item it is used for. The _name_ is -used to distinguish between different types of Pdus, Gops, and Gogs. The _name_ +used to distinguish between different types of PDUs, GOPs, and GOGs. The _name_ is also used as part of the filterable fields' names related to this type of -Gop which MATE creates. +GOG which MATE creates. -====== Member clause +===== Member clause -Defines the _key_ AVPL for the Gog individually for each Gop type _gopname_. -All _gopname_ type Gops whose _key_ AVPL matches the corresponding _key_ AVPL -of an active Gog are assigned to that Gog; a Gop which contains the AVPs whose -attribute names are listed in the Gog's corresponding _key_ AVPL, but they do -not strictly match any active Gog's _key_ AVPL, will create a new Gog. When a -Gog is created, the elements of its _key_ AVPL are copied from the creating Gop. +Defines the _key_ AVPL for the GOG individually for each GOP type _gopname_. +All _gopname_ type GOPs whose _key_ AVPL matches the corresponding _key_ AVPL +of an active GOG are assigned to that GOG; a GOP which contains the AVPs whose +attribute names are listed in the GOG's corresponding _key_ AVPL, but they do +not strictly match any active GOG's _key_ AVPL, will create a new GOG. When a +GOG is created, the elements of its _key_ AVPL are copied from the creating GOP. Although the _key_ AVPLs are specified separately for each of the Member -_gopname_ s, in most cases they are identical, as the very purpose of a Gog is -to group together Gops made of Pdus of different types. +_gopname_++s++, in most cases they are identical, as the very purpose of a GOG is +to group together GOPs made of PDUs of different types. -====== Extra clause +===== Extra clause -If given, tells MATE which AVPs from any of the Gop's AVPL are to be copied -into the Gog's AVPL in addition to the Gog's key. +If given, tells MATE which AVPs from any of the GOP's AVPL are to be copied +into the GOG's AVPL in addition to the GOG's key. -====== Expiration clause +===== Expiration clause -A (floating) number of seconds after all the Gops assigned to a Gog have been -released during which new Gops matching any of the session keys should still be -assigned to the existing Gog instead of creating a new one. Its value can range +A (floating) number of seconds after all the GOPs assigned to a GOG have been +released during which new GOPs matching any of the session keys should still be +assigned to the existing GOG instead of creating a new one. Its value can range from 0.0 to infinite. Defaults to 2.0 seconds. -====== Transform clause +===== Transform clause -The _Transform_ clause specifies a list of previously declared _Transform_ s to -be performed on the Gog's AVPL after the AVPs from each new Gop, specified by +The _Transform_ clause specifies a list of previously declared _Transform_++s++ to +be performed on the GOG's AVPL after the AVPs from each new GOP, specified by the _key_ AVPL and the _Extra_ clause's match_avpl, have been merged into it. The list is always executed completely, left to right. On the contrary, the list of _Match_ clauses inside each individual _Transform_ is executed only until the first match succeeds. -====== TreeMode clause +===== TreeMode clause -Controls the display of Gops subtree of the Gog: +Controls the display of GOPs subtree of the GOG: * _NoTree_: completely suppresses showing the tree * _BasicTree_: needs investigation * _FullTree_: needs investigation -====== ShowTimes clause +===== ShowTimes clause -Whether or not to show the times subtree of the Gog. If _TRUE_, the default, -the subtree with the timers is added to the Gog's tree. If _FALSE_, the subtree +Whether or not to show the times subtree of the GOG. If _TRUE_, the default, +the subtree with the timers is added to the GOG's tree. If _FALSE_, the subtree is suppressed. -===== Settings Config AVPL +[#Transform] +==== Transform declaration block + +A Transform is a sequence of Match rules optionally followed by an instruction +how to modify the match result using an additional AVPL. Such modification may +be an Insert (merge) or a Replace. The syntax is as follows: + +---- +Transform name { + Match [Strict|Every|Loose] match_avpl [[Insert|Replace] modify_avpl] ; // may occur multiple times, at least once +}; +---- + +For examples of Transforms, check the <<ChMateManual,Manual>> page. + +TODO: migrate the examples here? + +The list of Match rules inside a Transform is processed top to bottom; +the processing ends as soon as either a Match rule succeeds or all have been +tried in vain. + +Transforms can be used as helpers to manipulate an item's AVPL before the item +is processed further. An item declaration may contain a Transform clause +indicating a list of previously declared Transforms. Regardless whether the +individual transforms succeed or fail, the list is always executed completely +and in the order given, i.e., left to right. + +In MATE configuration file, a Transform must be declared before declaring any +item which uses it. + +==== Settings configuration AVPL + +[NOTE] +==== +The *Settings* parameters have been moved to other configuration parameters +or deprecated. Leave for now until rest of document is updated for current syntax. +==== The *Settings* config element is used to pass to MATE various operational parameters. the possible parameters are -====== GogExpiration +===== GogExpiration -How long in seconds after all the gops assigned to a gog have been released new -gops matching any of the session keys should create a new gog instead of being +How long in seconds after all the GOPs assigned to a GOG have been released new +GOPs matching any of the session keys should create a new GOG instead of being assigned to the previous one. Its value can range from 0.0 to infinite. Defaults to 2.0 seconds. -====== DiscardPduData +===== DiscardPduData -Whether or not the AVPL of every Pdu should be deleted after it was being +Whether or not the AVPL of every PDU should be deleted after it was being processed (saves memory). It can be either _TRUE_ or _FALSE_. Defaults to _TRUE_. Setting it to _FALSE_ can save you from a headache if your config does not work. -====== DiscardUnassignedPdu +===== DiscardUnassignedPdu -Whether Pdus should be deleted if they are not assigned to any Gop. It can be +Whether PDUs should be deleted if they are not assigned to any GOP. It can be either _TRUE_ or _FALSE_. Defaults to _FALSE_. Set it to _TRUE_ to save memory -if unassigned Pdus are useless. +if unassigned PDUs are useless. -====== DiscardUnassignedGop +===== DiscardUnassignedGop -Whether GoPs should be deleted if they are not assigned to any session. It can +Whether GOPs should be deleted if they are not assigned to any session. It can be either _TRUE_ or _FALSE_. Defaults to _FALSE_. Setting it to _TRUE_ saves memory. -====== ShowPduTree +===== ShowPduTree -====== ShowGopTimes +===== ShowGopTimes -===== Debugging Stuff +==== Debugging Stuff The following settings are used to debug MATE and its configuration. All levels are integers ranging from 0 (print only errors) to 9 (flood me with junk), defaulting to 0. -====== Debug declaration block header +===== Debug declaration block header ---- - Debug { - Filename "path/name"; //optional, no default value - Level [0-9]; //optional, generic debug level - Pdu Level [0-9]; //optional, specific debug level for Pdu handling - Gop Level [0-9]; //optional, specific debug level for Gop handling - Gog Level [0-9]; //optional, specific debug level for Gog handling - }; +Debug { + Filename "path/name"; //optional, no default value + Level [0-9]; //optional, generic debug level + Pdu Level [0-9]; //optional, specific debug level for Pdu handling + Gop Level [0-9]; //optional, specific debug level for Gop handling + Gog Level [0-9]; //optional, specific debug level for Gog handling +}; ---- ====== Filename clause @@ -2272,27 +2355,27 @@ ranging from 0 (print only errors) to 9 (flood me with junk). ====== Pdu Level clause -Sets the level of debugging for messages regarding Pdu creation. It is an +Sets the level of debugging for messages regarding PDU creation. It is an integer ranging from 0 (print only errors) to 9 (flood me with junk). ====== Gop Level clause -Sets the level of debugging for messages regarding Pdu analysis (that is how do -they fit into ?GoPs). It is an integer ranging from 0 (print only errors) to 9 +Sets the level of debugging for messages regarding PDU analysis (that is how do +they fit into ?GOPs). It is an integer ranging from 0 (print only errors) to 9 (flood me with junk). ====== Gog Level clause -Sets the level of debugging for messages regarding GoP analysis (that is how do -they fit into ?GoGs). It is an integer ranging from 0 (print only errors) to 9 +Sets the level of debugging for messages regarding GOP analysis (that is how do +they fit into ?GOGs). It is an integer ranging from 0 (print only errors) to 9 (flood me with junk). -====== Settings Example +===== Settings Example ---- Action=Settings; SessionExpiration=3.5; DiscardPduData=FALSE; ---- -===== Action=Include +==== Action=Include Will include a file to the configuration. @@ -2300,17 +2383,17 @@ Will include a file to the configuration. Action=Include; {Filename=filename;|Lib=libname;} ---- -====== Filename +===== Filename The filename of the file to include. If it does not begin with '/' it will look for the file in the current path. -====== Lib +===== Lib The name of the lib config to include. will look for libname.mate in wiresharks_dir/matelib. -====== Include Example +===== Include Example ---- Action=Include; Filename=rtsp.mate; ---- |