12.4. MATE Frame Analysis
Prev | Chapter 12. MATE | Next

## 12.4. MATE Frame Analysis

MATE’s analysis of a frame is performed in three phases:

- In the first phase, MATE attempts to extract a MATE PDU from the frame’s protocol tree. MATE will create a PDU if MATE’s config has a Pdu declaration whose Proto is contained in the frame.
- In the second phase, if a PDU has been extracted from the frame, MATE will try to group it to other PDUs into a GOP (Group of PDUs) by matching the key criteria given by a Gop declaration. If there is no GOP yet with the key criteria for the PDU, MATE will try to create a new GOP for it if it matches the Start criteria given in the Gop declaration.
- In the third phase, if there’s a GOP for the PDU, MATE will try to group this GOP with other GOPs into a GOG (Group of Groups) using the criteria given by the Member criteria of a Gog declaration.

Figure 12.2. MATE Analysis (PDU→GOP→GOG) flowchart

The extraction and matching logic comes from MATE’s configuration; MATE’s configuration file is specified by the mate.config preference. By default it is an empty string which means: do not configure MATE.

The config file tells MATE what to look for in frames; How to make PDUs out of it; How will PDUs be related to other similar PDUs into GOPs; And how GOPs relate into GOGs.

The MATE configuration file is a list of declarations. There are 4 types of declarations: Transform, Pdu, Gop, and Gog. A Transform block must be before any of the other block declarations that may use it.

### 12.4.1. Create PDUs (Phase 1)

MATE will look in the tree of every frame to see if there is useful data to extract, and if there is, it will create one or more PDU objects containing the useful information.

The first part of MATE’s analysis is the "PDU extraction".

#### 12.4.1.1. PDU data extraction

MATE will make a PDU for each different proto field of Proto type present in the frame. MATE will fetch from the field’s tree those fields that are defined in the Section 12.9.1, “PDU declaration block” declaration whose initial offset in the frame is within the boundaries of the current Proto and those of the given Transport and Payload statements.

    Pdu dns_pdu Proto dns Transport ip {
        Extract addr From ip.addr;
        Extract dns_id From dns.id;
        Extract dns_resp From dns.flags.response;
    };

Figure 12.3. Wireshark window - fields for PDU extraction

Once MATE has found a Proto field for which to create a PDU from the frame it will move backwards in the frame looking for the respective Transport fields. After that it will create AVPs named as each of those given in the rest of the AVPL for every instance of the fields declared as its values.

Figure 12.4. Frame fields mapped to PDU attributes

Sometimes we need information from more than one Transport protocol. In that case MATE will check the frame looking backwards to look for the various Transport protocols in the given stack. MATE will choose only the closest transport boundary per "protocol" in the frame.

This way we’ll have all PDUs for every Proto that appears in a frame match its relative transports.

    Pdu isup_pdu Proto isup Transport mtp3/ip {
        Extract addr From ip.addr;

        Extract m3pc From mtp3.dpc;
        Extract m3pc From mtp3.opc;

        Extract cic From isup.cic;
        Extract isup_msg From isup.message_type;
    };

Figure 12.5. Frame containing multiple PDUs

This allows to assign the right Transport to the PDU avoiding duplicate transport protocol entries (in case of tunneled ip over ip for example).

    Pdu ftp_pdu Proto ftp Transport tcp/ip {
        Extract addr From ip.addr;
        Extract port From tcp.port;
        Extract ftp_cmd From ftp.command;
    };

Figure 12.6. Frame with encapsulated (tunneled) fields

Other than the mandatory Transport there is also an optional Payload statement, which works pretty much as Transport but refers to elements after the Proto's range. It is useful in those cases where the payload protocol might not appear in a PDU but nevertheless the PDU belongs to the same category.

    Pdu mmse_over_http_pdu Proto http Transport tcp/ip {

        Payload mmse;

        Extract addr From ip.addr;
        Extract port From tcp.port;

        Extract content From http.content_type;
        Extract host From http.host;
        Extract http_rq From http.request;
        Extract method From http.request.method;
        Extract resp From http.response.code;

        Extract msg_type From mmse.message_type;
        Extract notify_status From mmse.status;
        Extract send_status From mmse.response_status;
        Extract trx From mmse.transaction_id;
    };

Figure 12.7. Extract from Payload fields

#### 12.4.1.2. Conditions on which to create PDUs

There might be cases in which we won’t want MATE to create a PDU unless some of its extracted attributes meet or do not meet some criteria. For that we use the Criteria statements of the Pdu declarations.

    Pdu isup_pdu Proto isup Transport mtp3/ip {
        ...

        // MATE will create isup_pdu PDUs only when there is not a point code '1234'
        Criteria Reject Strict (m3pc=1234);
    };

    Pdu ftp_pdu Proto ftp Transport tcp/ip {
        ...

        // MATE will create ftp_pdu PDUs only when they go to port 21 of our ftp_server
        Criteria Accept Strict (addr=10.10.10.10, port=21);
    };

The Criteria statement is given an action (Accept or Reject), a match type (Strict, Loose or Every) and an AVPL against which to match the currently extracted one.

#### 12.4.1.3. Transforming the attributes of a PDU

Once the fields have been extracted into the PDU’s AVPL, MATE will apply any declared Transform to it. The way transforms are applied and how they work is described later on. However, it’s useful to know that once the AVPL for the PDU is created, it may be transformed before being analyzed. That way we can massage the data to simplify the analysis.

#### 12.4.1.4. MATE’s PDU tree

Every successfully created PDU will add a MATE tree to the frame dissection. If the PDU is not related to any GOP, the tree for the PDU will contain just the PDU’s info. If it is assigned to a GOP, the tree will also contain the GOP items, and the same applies for the GOG level.

    mate dns_pdu:1
        dns_pdu: 1
            dns_pdu time: 3.750000
            dns_pdu Attributes
                dns_resp: False
                dns_id: 0x8cac
                addr: 10.194.4.11
                addr: 10.194.24.35

The PDU’s tree contains some filterable fields

- mate.dns_pdu will contain the number of the "dns_pdu" PDU
- mate.dns_pdu.RelativeTime will contain the time passed since the beginning of the capture in seconds

- the tree will contain the various attributes of the PDU as well, these will all be strings (to be used in filters as "10.0.0.1", not as 10.0.0.1)

  - mate.dns_pdu.dns_resp
  - mate.dns_pdu.dns_id
  - mate.dns_pdu.addr

### 12.4.2. Grouping PDUs together (GOP) (Phase 2)

Once MATE has created the PDUs it passes to the PDU analysis phase. During the PDU analysis phase MATE will try to group PDUs of the same type into 'Groups of PDUs' (aka GOPs) and copy some AVPs from the PDU’s AVPL to the GOP’s AVPL.

Figure 12.8. Grouping PDUs (GOP) flowchart

#### 12.4.2.1. What can belong to a GOP

Given a PDU, the first thing MATE will do is to check if there is any GOP declaration in the configuration for the given PDU type. If so, it will use its Match AVPL to match it against the PDU’s AVPL; if they don’t match, the analysis phase is done. If there is a match, the AVPL is the GOP’s candidate key which will be used to search the index of GOPs for the GOP to which to assign the current PDU. If there is no such GOP and this PDU does not match the Start criteria of a Gop declaration for the PDU type, the PDU will remain unassigned and only the analysis phase will be done.

    Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {...};
    Gop dns_req On dns_pdu Match (addr, addr, dns_id) {...};
    Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {...};

#### 12.4.2.2. Start of a GOP

If there was a match, the candidate key will be used to search the index of GOPs to see if there is already a GOP matching the GOP’s key the same way. If there is such a match in the GOPs collection, and the PDU doesn’t match the Start AVPL for its type, the PDU will be assigned to the matching GOP. If it is a Start match, MATE will check whether or not that GOP has been already stopped. If the GOP has been stopped, a new GOP will be created and will replace the old one in the index of GOPs.

    Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
        Start (ftp_cmd=USER);
    };

    Gop dns_req On dns_pdu Match (addr, addr, dns_id) {
        Start (dns_resp="True");
    };

    Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
        Start (isup_msg=1);
    };

If no Start is given for a GOP, a PDU whose AVPL matches an existing GOP’s key will act as the start of a GOP.

#### 12.4.2.3. What goes into the GOP’s AVPL

Once we know a GOP exists and the PDU has been assigned to it, MATE will copy into the GOP’s AVPL all the attributes matching the key plus any AVPs of the PDU’s AVPL matching the Extra AVPL.

    Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
        Start (ftp_cmd=USER);
        Extra (pasv_prt, pasv_addr);
    };

    Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
        Start (isup_msg=1);
        Extra (calling, called);
    };

#### 12.4.2.4. End of a GOP

Once the PDU has been assigned to the GOP, MATE will check whether or not the PDU matches the Stop, if it happens, MATE will mark the GOP as stopped. Even after stopped, a GOP may get assigned new PDUs matching its key, unless such PDU matches Start. If it does, MATE will instead create a new GOP starting with that PDU.

    Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
        Start (ftp_cmd=USER);
        Stop (ftp_cmd=QUIT); // The response to the QUIT command will be assigned to the same GOP
        Extra (pasv_prt, pasv_addr);
    };

    Gop dns_req On dns_pdu Match (addr, addr, dns_id) {
        Start (dns_resp="False");
        Stop (dns_resp="True");
    };

    Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
        Start (isup_msg=1); // IAM
        Stop (isup_msg=16); // RLC
        Extra (calling, called);
    };

If no Stop criterium is stated for a given GOP, the GOP will be stopped as soon as it is created. However, as with any other GOP, PDUs matching the GOP’s key will still be assigned to the GOP unless they match a Start condition, in which case a new GOP using the same key will be created. To group multiple PDUs that match the Start, add a bogus Stop such as

    Gop frame_ses On frame_pdu Match (frame_time) {
        Start (frame_time);
        Stop (frame_time="FOO");
    };

#### 12.4.2.5. GOP’s tree

For every frame containing a PDU that belongs to a GOP, MATE will create a tree for that GOP.

The example below represents the tree created by the dns_pdu and dns_req examples.

    ...
    MATE dns_pdu:6->dns_req:1
        dns_pdu: 6
            dns_pdu time: 2.103063
            dns_pdu time since beginning of Gop: 2.103063
            dns_pdu Attributes
                dns_resp: True
                dns_id: 0x8cac
                addr: 10.194.4.11
                addr: 10.194.24.35
        dns_req: 1
            GOP Key:  addr=10.194.4.11; addr=10.194.24.35; dns_id=0x8cac;
            dns_req Attributes
                dns_id: 0x8cac
                addr: 10.194.4.11
                addr: 10.194.24.35
            dns_req Times
                dns_req start time: 0.000000
                dns_req hold time: 2.103063
                dns_req duration: 2.103063
            dns_req number of PDUs: 2
                Start PDU: in frame 1
                Stop PDU: in frame 6 (2.103063 : 2.103063)

Other than the PDU’s tree, this one contains information regarding the relationship between the PDUs that belong to the GOP. That way we have:

- mate.dns_req which contains the id of this dns_req GOP. This will be present in frames that belong to dns_req GOPs.
- mate.dns_req.dns_id and mate.dns_req.addr which represent the values of the attributes copied into the GOP.

- the timers of the GOP

  - mate.dns_req.StartTime time (in seconds) passed since beginning of capture until GOP’s start.
  - mate.dns_req.Time time passed between the start PDU and the stop PDU assigned to this GOP (only created if a Stop criterion has been declared for the GOP and a matching PDU has arrived).
  - mate.dns_req.Duration time passed between the start PDU and the last PDU assigned to this GOP.

- mate.dns_req.NumOfPdus the number of PDUs that belong to this GOP

  - mate.dns_req.Pdu a filterable list of frame numbers of the PDUs of this GOP

#### 12.4.2.6. GOP’s timers

Note that there are two "timers" for a GOP:

- Time, which is defined only for GOPs that have been Stopped, and gives the time passed between the Start and the Stop PDUs.
- Duration, which is defined for every GOP regardless of its state, and give the time passed between its Start PDU and the last PDU that was assigned to that GOP.

So:

- we can filter for PDUs that belong to GOPs that have been Stopped with mate.xxx.Time
- we can filter for PDUs that belong to unstopped GOPs with mate.xxx && !mate.xxx.Time
- we can filter for PDUs that belong to stopped GOPs using mate.xxx.Duration
- we can filter for PDUs that belong to GOPs that have taken more (or less) time that 0.5s to complete with mate.xxx.Time > 0.5 (you can try these also as color filters to find out when response times start to grow)

### 12.4.3. Grouping GOPs together (GOG) (Phase 3)

When GOPs are created, or whenever their AVPL changes, GOPs are (re)analyzed to check if they match an existent group of groups (GOG) or can create a new one. The GOP analysis is divided into two phases. In the first phase, the still unassigned GOP is checked to verify whether it belongs to an already existing GOG or may create a new one. The second phase eventually checks the GOG and registers its keys in the index of GOGs.

Figure 12.9. Grouping GOPs (GOG) flowchart

There are several reasons for the author to believe that this feature needs to be reimplemented, so probably there will be deep changes in the way this is done in the near future. This section of the documentation reflects the version of MATE as of Wireshark 0.10.9; in future releases this will change.

#### 12.4.3.1. Declaring a Group Of Groups (GOG)

The first thing we have to do configuring a GOG is to tell MATE that it exists.

    Gog http_use {
       ...
    };

#### 12.4.3.2. Telling MATE what could be a GOG member

Then we have to tell MATE what to look for a match in the candidate GOPs.

    Gog http_use {
        Member http_ses (host);
        Member dns_req (host);
    };

#### 12.4.3.3. Getting interesting data into the GOG

Most often, also other attributes than those used for matching would be interesting. In order to copy from GOP to GOG other interesting attributes, we might use Extra like we do for GOPs.

    Gog http_use {
        ...
        Extra (cookie);
    };

#### 12.4.3.4. GOG’s tree

    mate http_pdu:4->http_req:2->http_use:1
        http_pdu: 4
            http_pdu time: 1.309847
            http_pdu time since beginning of Gop: 0.218930
            http_req: 2
                ... (the gop's tree for http_req: 2) ..
            http_use: 1
                http_use Attributes
                    host: www.example.com
                http_use Times
                    http_use start time: 0.000000
                    http_use duration: 1.309847
                number of GOPs: 3
                    dns_req: 1
                        ... (the gop's tree for dns_req: 1) ..
                    http_req: 1
                        ... (the gop's tree for http_req: 1) ..
                    http_req of current frame: 2

We can filter on:

- the timers of the GOG

  - mate.http_use.StartTime time (in seconds) passed since beginning of capture until GOG’s start.
  - mate.http_use.Duration time elapsed between the first frame of a GOG and the last one assigned to it.

- the attributes passed to the GOG

  - mate.http_use.host
- mate.http_use.NumOfGops the number of GOPs that belong to this GOG
- mate.http_use.GopStart the start frame of a GOP
- mate.http_use.GopStop the stop frame of a GOP

### 12.4.4. Adjust data (AVPL Transforms)

A Transform is a sequence of Match rules optionally completed with modification of the match result by an additional AVPL. Such modification may be an Insert (merge) or a Replace. Transforms can be used as helpers to manipulate an item’s AVPL before it is processed further. They come to be very helpful in several cases.

#### 12.4.4.1. Syntax

AVPL Transforms are declared in the following way:

    Transform name {
        Match [Strict|Every|Loose] match_avpl [Insert|Replace] modify_avpl;
        ...
    };

The name is the handle to the AVPL transform. It is used to refer to the transform when invoking it later.

The Match declarations instruct MATE what and how to match against the data AVPL and how to modify the data AVPL if the match succeeds. They will be executed in the order they appear in the config file whenever they are invoked.

The optional match type qualifier (Strict, Every, or Loose) is used to choose the Match type; Strict is the default value which may be omitted.

The optional modification mode qualifier instructs MATE how the modify AVPL should be used:

- the default value Insert (which may be omitted) causes the modify_avpl to be merged to the existing data AVPL,
- Replace causes all the matching AVPs from the data AVPL to be replaced by the modify_avpl.

The modify_avpl may be an empty one; this comes useful in some cases for both Insert and Replace modification modes.

    Transform rm_client_from_http_resp1 {
        Match (http_rq); //first match wins so the request won't get the not_rq attribute inserted
        Match Every (addr) Insert (not_rq); //this line won't be evaluated if the first one matched so not_rq won't be inserted to requests
    };

    Transform rm_client_from_http_resp2 {
        Match (not_rq, client) Replace (); //replace "client and not_rq" with nothing
    };

Examples:

    Transform insert_name_and {
        Match Strict (host=10.10.10.10, port=2345) Insert (name=JohnDoe);
    };

adds name=JohnDoe to the data AVPL if it contains host=10.10.10.10 and port=2345

    Transform insert_name_or {
        Match Loose (host=10.10.10.10, port=2345) Insert (name=JohnDoe);
    };

adds name=JohnDoe to the data AVPL if it contains host=10.10.10.10 or port=2345

    Transform replace_ip_address {
        Match (host=10.10.10.10) Replace (host=192.168.10.10);
    };

replaces the original host=10.10.10.10 by host=192.168.10.10

    Transform add_ip_address {
        Match (host=10.10.10.10) (host=192.168.10.10);
    };

adds (inserts) host=192.168.10.10 to the AVPL, keeping the original host=10.10.10.10 in it too

     Transform replace_may_be_surprising {
        Match Loose (a=aaaa, b=bbbb) Replace (c=cccc, d=dddd);
     };

gives the following results:

- (a=aaaa, b=eeee) gets transformed to (b=eeee, c=cccc, d=dddd) because a=aaaa did match so it got replaced while b=eeee did not match so it has been left intact,
- (a=aaaa, b=bbbb) gets transformed to (c=cccc, d=dddd) because both a=aaaa and b=bbbb did match.

#### 12.4.4.2. Usage

Once declared, Transforms can be added to the declarations of PDUs, GOPs or GOGs. This is done by adding the Transform name_list statement to the declaration:

    Pdu my_proto_pdu Proto my_proto Transport ip {
        Extract addr From ip.addr;
        ...
        Transform my_pdu_transform[, other_pdu_transform[, yet_another_pdu_transform]];
    };

- In case of PDU, the list of transforms is applied against the PDU’s AVPL after its creation.
- In case of GOP and GOG, the list of transforms is applied against their respective AVPLs when they are created and every time they change.

#### 12.4.4.3. Operation

Figure 12.10. Applying Transform flowchart

- A list of previously declared Transforms may be given to every Item (Pdu, Gop, or Gog), using the Transform statement.
- Every time the AVPL of an item changes, it will be operated against all the Transforms on the list given to that item. The Transforms on the list are applied left to right.
- Inside each of the Transforms, the item’s AVPL will be operated against the Transform’s Match clauses starting from the topmost one, until all have been tried or until one of them succeeds.

MATE’s Transforms can be used for many different things, like:

#### 12.4.4.4. Multiple Start/Stop conditions for a GOP

Using Transforms we can add more than one start or stop condition to a GOP.

    Transform start_cond {
        Match (attr1=aaa,attr2=bbb) (msg_type=start);
        Match (attr3=www,attr2=bbb) (msg_type=start);
        Match (attr5^a) (msg_type=stop);
        Match (attr6$z) (msg_type=stop);
    };

    Pdu pdu ... {
        ...
        Transform start_cond;
    }

    Gop gop ... {
        Start (msg_type=start);
        Stop (msg_type=stop);
        ...
    }

#### 12.4.4.5. Marking GOPs and GOGs to filter them easily

    Transform marks {
        Match (addr=10.10.10.10, user=john) (john_at_host);
        Match (addr=10.10.10.10, user=tom) (tom_at_host);
    }

    ...

    Gop my_gop ... {
        ...
        Transform marks;
    }

After that we can use a display filter mate.my_gop.john_at_host or mate.my_gop.tom_at_host

#### 12.4.4.6. Adding (Insert) direction knowledge to MATE

    Transform direction_as_text {
        Match (src=192.168.0.2, dst=192.168.0.3) Insert (direction=from_2_to_3);
        Match (src=192.168.0.3, dst=192.168.0.2) Insert (direction=from_3_to_2);
    };

    Pdu my_pdu Proto my_proto Transport tcp/ip {
        Extract src From ip.src;
        Extract dst From ip.dst;
        Extract addr From ip.addr;
        Extract port From tcp.port;
        Extract start From tcp.flags.syn;
        Extract stop From tcp.flags.fin;
        Extract stop From tcp.flags.rst;
        Transform direction_as_text;
    }

    Gop my_gop On my_pdu Match (addr,addr,port,port) {
        ...
        Extra (direction);
    }

The original example (below) would delete src and dst then add direction.

    Transform direction_as_text {
        Match (src=192.168.0.2, dst=192.168.0.3) Replace (direction=from_2_to_3);
        Match (src=192.168.0.3, dst=192.168.0.2) Replace (direction=from_3_to_2);
    };

#### 12.4.4.7. NAT

NAT can create problems when tracing, but we can easily work around it by Transforming the NATed IP address and the Ethernet address of the router into the non-NAT address:

    Transform denat {
        Match (addr=192.168.0.5, ether=01:02:03:04:05:06) Replace (addr=123.45.67.89);
        Match (addr=192.168.0.6, ether=01:02:03:04:05:06) Replace (addr=123.45.67.90);
        Match (addr=192.168.0.7, ether=01:02:03:04:05:06) Replace (addr=123.45.67.91);
    }

    Pdu my_pdu Proto my_proto transport tcp/ip/eth {
        Extract ether From eth.addr;
        Extract addr From ip.addr;
        Extract port From tcp.port;
        Transform denat;
    }

Prev | Up | Next
12.3. MATE Overview | Home | 12.5. MATE’s configuration tutorial